Sunday 24 June 2012

Wrapping... up?

I came upon POCO C++ while looking for a logging framework, and it looked a good fit for what I needed. Still, I felt a bit apprehensive about spreading Poco::Logger references throughout my code, so I decided to create a wrapper.

At a certain point, I created a class called LoggerController, that would control a container of loggers, allowing clients to create/store/retrieve loggers. And I wish I had created this class sooner. Why? Because after I started working on it, I quickly realized I was heading the wrong way.

I had created the wrong abstraction. I started with a worthwhile goal - minimize the dependency between my code and a particular logging implementation (in this case, POCO's). However, I was soon duplicating some of POCO's logger functionality, namely, the task of managing loggers.

In fact, what I actually needed was just something that encapsulated the Logger reference returned by Poco::Logger::get(). So, if I ever came upon another framework, say Moocho C++, with a logger better suited to my needs, I (hopefully) wouldn't have to change all my code, only the wrapper I created.

So, after some false starts, I reached something like this:

class Logger
{
public:
    enum Level
    { Fatal = 1, Critical, Error, Warning, Info = 6, Debug };
 
    // A SIMPLISTIC WAY TO CORRELATE ENUM VALUES AND TEXTUAL DESCRIPTION
    static std::pair< std::string, Level > LevelDescription[];

    Logger(Logger const&) = delete;
    Logger& operator=(Logger const&) = delete;
    Logger() = delete;

    Logger(std::string const& loggerName);

    // EACH OF THE FOLLOWING FUNCTIONS ACTIVATES A POCO CHANNEL. THEY'RE EXCLUSIVE,
    // A POCO LOGGER HAS A "THERE CAN BE ONLY 1" RELATIONSHIP WITH ITS CHANNEL

    // THE CONFIGURATION FOR THE LOGGER IS STORED IN A PROPERTIES FILE.
    // THE propertyPrefix TELLS US THE PROPERTY PREFIX WE NEED TO READ,
    // E.G., "com.this.and.that"
    void SimpleFileLogger(std::string const& propertyPrefix);
    //void FileLogger(std::string const& filepath); // ToDo

    void SetLoggerLevel(Level newLevel) { m_logger.setLevel(newLevel); }
    Level GetLoggerLevel() const 
        { return static_cast<Level>(m_logger.getLevel()); }
    bool OKToLog(Level l) const { return m_logger.is(l); }

    // BY CALLING THE poco_* MACROS, WE WILL BE, AT TIMES, 
    // TESTING THE LOG LEVEL TWICE, ONCE IN OUR OWN CODE, 
    // AND ONCE IN POCO'S CODE
    void LogDebug(std::string const& message) { poco_debug(m_logger, message); }
    void LogInfo(std::string const& message) 
        { poco_information(m_logger, message); }
private:
    Logger::Level ConvertLevel(std::string const& levelString) const;
    Poco::Logger& m_logger;
};

Note: The UPPERCASE comments is something I've picked up many years ago, on a Clipper 5 book, the rationale being that by emphasizing the comments, assuming these were correct and up-to-date, one could have an easier time understanding the intended functional goal of the code. However, with all the tools available to generate automatic docs from the comments, and with IDEs colored syntax (something I didn't have in those days), I'll be revising this practice.

It's actually a very small wrapper. I don't intend to create something that exposes the whole Poco::Logger functionality. That may happen, as time goes by, and my needs change. This is the way I'd rather work, when I have that option - start simple and add more functionality as needed.

So, once it's set up, it'll be relatively simple to use, you just declare it:

Logger l("TestLog");

// Do something (hopefully) useful

l.LogDebug("Insert useful info here");

And, when the scope ends, so does the wrapper.

Yes, there's the creation of an object everytime a logger is needed. For now, I'll assume that's fine. If I ever come upon a situation where it causes a bottleneck, I'll bypass my wrapper and use Poco::Logger directly. If this ever happens, I expect it'll be in a small section of code, anyway,

The actual management of TestLog (e.g., creating it, if it doesn't exist) resides in POCO C++, and I don't need (nor want) to duplicate it. I just want a handy way to get a reference to Poco::Logger without actually calling it Poco::Logger&.

It lends itself to becoming an interface, with several implementations (e.g., each implementation with its own logging framework); but, until I see a good reason to do it, I'll leave it as it is.

As I use it, I'll post here about its merits - and faults.

Tuesday 19 June 2012

Upgrading

In the process of building the libs I'm using, I've made several updates/upgrades/additions to my toolset. It didn't get out of control, but I felt the need to organize it properly, and to create a stable foundation.

So, I've been in upgrade mode. It all began with mingw, which I upgraded to gcc 4.7. Then, I've rebuilt the libs I've been using: Boost, libssh2 (and its OpenSSL and zlib dependencies), and Poco C++.

Ever since I installed a newer version of mingw (newer than the custom mingw that shipped with Qt Creator), I've found some hints that I might have trouble later on, especially where debugging is concerned. So, in order to see how well my proposed "stable foundation" worked, I created a simple C++ console program (called "Goooooooooood Morning, World"; yeah, I strive for originality), to test it.

As I hit "Debug", I got this wonderful message about how my debug process was having second thoughts about life and wasn't exactly in the mood to do its job at the moment, and could I please call back at a later time, etc, etc.

As a previous Borland tools user, the idea that command-line tools are being run when I click on a button on the IDE or select a menu item is still somewhat foreign. While this is not the case for some actions (e.g., building), because I was always aware of the command-line being there, other actions (say, debugging) have a less-than-obvious - to me - relation to the command-line. So, I've braced myself for a journey filled with the symbols of mysticism... er, I mean, debugging.

However, before I started, and with my head still full of potential incompatibility problems, I've figured if things went wrong I might have to rebuild Qt/Qt Creator with gcc 4.7; and, if that were the case, I might as well have an early start. So, while I was going through the debugger problem, I had a tail -f hapilly running in a DOS prompt on the build log, and I'm quite satisfied to say that both Qt and Qt Creator built successfully from source (which was another of the reasons why I did it - to see how it worked out).

Anyway... when I finally hit on the idea that the debugging process was actually a command-line execution, I went to my gdb-python27.exe and ran it (Qt Creator works with the python-enabled gdb). And it told me, in no uncertain terms, that it would absolutely refuse to do anything whatsoever until I got it on a date with python27.dll. Which led me to copy it to Qt Creator's debugging tools folder (which contains a copy of this DLL), and update its path in my toolchains.

Then, I had another go at debugging my "Goooooooooood Morning, World". And this time everything ran perfectly... except for the SIGILL, that is. Consulting an online dictionary, one could think the definition of SIGILL is "a seall or a signett". One would be wrong, though.

So, what was causing an illegal instruction? Google gave me no useful answers. After some time looking through Qt Creator's debugging tools docs, I found out about the debugging log, and fired another debugging session. When I got the error, I went to the log. And there was the first clue - my app was loading the wrong sdtlibc++ DLL. Why? Because the Run Settings for my project had Qt's paths before any other path (Qt's default behaviour). I wasn't entirely sure this was the cause, but I've changed it, anyway, because that was not the behaviour I wanted.

After this change, the debugging session ran perfectly. Then, I decided to up the ante, and add a QString (Qt's string class) to my example. After all, the Qt libs I'm using were built with Qt Creator's custom mingw and gcc 4.4. But, this time, everything went smoothly. No sign of the dreaded incompatibility.

So, for the time being, I'm declaring my current environment as "stable", and I'm hoping to keep it that way for a while.

Now, then... lessons learned:
  • If an IDE action corresponds to a command-line program being run, open a shell and run it yourself. You may have to do some digging to find out if your IDE is passing any arguments to this program.
  • Thoroughly investigate your IDE's debugging options and tools.
  • Run your app from the IDE and from the OS (and, on Windows, use Process Explorer to see what it's loading). This is especially useful as the ultimate test for missing DLLs.
  • If you have several toolchains (and, particularly, several versions of the same compiler), add to your build flags the option that prints version info. In my case, I added CFLAGS += -v. This way, I can look at the build output and check if the IDE is using the correct version.

Next steps:
  • I'm taking a closer look at make, to see if I need to have it include more info on the output, and how might I do it. The --debug and -w options are a good start, but I was looking for something more akin to ksh's -x or -v options.
  • I'll be taking some of Qt's examples for a building/running/debugging spin, to see if my environment is actually stable.

But, for now, back to my project.

Saturday 9 June 2012

libssh2 - Executing several commands

As I posted previously, you have to keep in mind that there won't be a context encompassing these commands, i.e., there won't be any state kept between executions.

We'll be modifying this libssh2 example. Our goal will be to refactor the channel life cycle, placing it in a function, so we can invoke it several times.

We'll start by delimiting this life cycle in the example. The first thing we want is the channel's creation:
    /* Exec non-blocking on the remove host */
    while( (channel = libssh2_channel_open_session(session)) == NULL &&
           libssh2_session_last_error(session,NULL,NULL,0) == LIBSSH2_ERROR_EAGAIN)
    {
        waitsocket(sock, session);
    }


And its destruction:
    while( (rc = libssh2_channel_close(channel)) == LIBSSH2_ERROR_EAGAIN )
        waitsocket(sock, session);
    (...)
    libssh2_channel_free(channel);
    channel = NULL;


My first idea was that maybe you could close the channel, but not free it, as freeing is mostly concerned with releasing local resources allocated during channel creation/operation. However, I found no way to open a channel other than libssh2_channel_open_session(), i.e., I found no way to reopen a channel (the ssh RFCs talk about channel reuse, but not channel reopening), so I'm assuming calling libssh2_channel_free() is required.

Having determined the code that defines the channel life cycle, we may now put it in a separate function, which I've called execcmd().

As you can see, the code is basically the same as you have on the libssh2 example. However, now that we have it in a separate function, we can easily invoke it several times.

E.g., we can change the fourth command-line argument, which is the command to run, and turn into a filename. We'll assume each line on the file is a command to run on a remote host, and each command has a max lenght of 80 chars.

Then, we add some simple code to open the file and call our execcmd() function for each line:

    cmd_file = fopen(filename, "rt");
    if (cmd_file == NULL)
    {
        perror("Error opening file");
        goto shutdown;
    }

    /* fgets will read the '\n', but that shouldn't be a problem
       HOWEVER, LINES WITH 80+ CHARS WILL BE A PROBLEM */
    while (fgets(command, 81, cmd_file) != NULL)
    {
        execcmd(command, sock, session);
    }

    fclose(cmd_file);


This code would go right after this on the example:

#if 0
    libssh2_trace(session, ~0 );
#endif


So, we follow the exact same sequence as the example, up until we're authenticated. Then, we open the file and call execcmd() for each line. Once that's done, we resume following the original example.

You can see the whole thing here. Like I've said, the actual work with libssh2 was already done in the original example, I just did a little bit of refactoring.

I've suggested this could be added to the examples in libssh2, because I've seen a fair amount of people asking for pointers on how to run several commands on a remote host, but I haven't actually got a "yes" or "no".

Anyway, I hope you find it useful.

Thursday 7 June 2012

Building POCO C++ with Mingw

I've been looking for a logging framework for C++. I'd read about several options, but hadn't really found anything I felt like trying, until a couple of days ago, when I discovered POCO C++.

I decided to give it a try, but the README seemed to be warning me I was in for some Interesting Times(TM):

Microsoft Visual Studio 7.1 (2003), 8.0 (2005), 9.0 (2008) or 10.0 (2010) is required to build the POCO C++ Libraries on Windows platforms.

BTW, the emphasis is mine.

Still, nothing new, just another OSS telling users of OSS on Windows: "You're on your own".

Before I go on, I've solved these problems on my own, but if I had searched the POCO Forum, I would've saved some time, as someone had posted solutions to all the problems I found. I did google it, but sometimes google is not enough.

I fired Msys and ran configure. I didn't follow README's advice, however, and didn't run make -s. I prefer having as much info available as possible, thank you very much, so I ran it with --debug and -w.

The first problem came in the form of an unrecognized option: -mno-cygwin. I looked in the <POCO_HOME>/build/config/MinGW file, and found these lines:

SHLIB   = $(CXX) -shared -mno-cygwin -o $@ -Wl,--out-implib=$(dir $@)$(subst cyg,lib,$(basename $(notdir $@))).a

SYSFLAGS = -mno-cygwin -D_WIN32 -DMINGW32 -DWINVER=0x500 -DPOCO_NO_FPENVIRONMENT -DPCRE_STATIC -DPOCO_THREAD_STACK_SIZE -DFoundation_Config_INCLUDED -I/usr/local/include -I/usr/include

A couple of searches, and I realized this option didn't apply to my version of gcc, so I removed it.

Then, the linker complained it couldn't find OpenSSL libs ssl and crypto. Having built OpenSSL, I had those libs, so I added another -L to this line, pointing to the OpenSSL libs (once again, in <POCO_HOME>/build/config/MinGW file):

SYSLIBS  = -L/usr/local/lib -L/usr/lib -liphlpapi -lws2_32 -lssl -lcrypto -lws2_32 -lgdi32

Another go at make, and this time I got an error with strip. After making sure I hadn't inadvertently linked in my porn collection (it is rather sizable, and tends to show up in the most embarrassing places), I've decided to google strip. And then, naturally, strip command.

From what I've learned, it's nothing I should actually worry about at this stage, although it is something interesting to look into later on. So, I've decided to remove it from the <POCO_HOME>/build/config/MinGW file, by turning this

STRIP   = strip

into this

STRIP   =

And there you go, we have build! I've built the Logger example, and I plan on giving it a try on my own project.

And now for something slightly completely different: A semi-rant...

MS is preparing to cripple the next version of VC++ Express, which the POCO C++ blog also refers in this post.

I fail to see what's the drama, actually. Maybe now all the people behind OSS can actually direct their effort at creating a better support for OSS on Windows, instead of just saying "There you go, have these VC++ files and enjoy".

Friday 1 June 2012

libssh2 and exit signals

Imagine this scenario:

1. You run ssh2_exec to invoke a command on a remote host.
2. The command takes some time to run (even if it's just a loop, with ls and sleep).
3. You open a terminal session to the host, find the process id, and kill it.
4. ssh2_exec terminates.

Did you get an exit signal?

If not, here's the first obvious tip of the day: Run the same command on another host, especially one that you know has a different OS (e.g., Linux vs. Solaris vs. HP-UX).

Second obvious tip: libssh2_trace() is your friend. Be sure to build a debug version of libssh2. In my case, this meant that win32/GNUmakefile had this:

# must be equal to DEBUG or NDEBUG
ifndef DB
    # DB    = NDEBUG
    DB    = DEBUG
endif

Back to the missing exit signal. ssh2_exec.c has this check:

    if (exitsignal)
        printf("\nGot signal: %s\n", exitsignal);
    else
        printf("\nEXIT: %d bytecount: %d\n", exitcode, bytecount);
 
       
On my first run, I got this:

Got signal:

which was not exactly as expected:

The remote command may also terminate violently due to a signal. Such a condition can be indicated by the following message.  A zero 'exit_status' usually means that the command terminated successfully.

      byte      SSH_MSG_CHANNEL_REQUEST
      uint32    recipient channel
      string    "exit-signal"
      boolean   FALSE
      string    signal name (without the "SIG" prefix)
      boolean   core dumped
      string    error message in ISO-10646 UTF-8 encoding
      string    language tag [RFC3066]
     
 
So, first thing to do was set trace on. The example has this:

#if 0
    libssh2_trace(session, ~0 );
#endif

so, I just changed it to "#if 1".

And this is what I was getting from the server:

0000: 62 00 00 00 00 00 00 00  0B 65 78 69 74 2D 73 69 : b........exit-si
0010: 67 6E 61 6C 00 00 00 00  0F 00 00 00 00 00 00 00 : gnal............
0020: 00 00                                            : ..

[libssh2] 16.161761 Conn: Channel 0 received request type exit-signal (wr 0)
[libssh2] 16.161761 Conn: Exit signal  received for channel 0/0

It became clear that the remote host wasn't sending the signal name. 0x62 is SSH_MSG_CHANNEL_REQUEST. The next 4 bytes are the recipient channel. The following 4 bytes are the length of the string "exit-signal", i.e, 0x0B. Then, we have a boolean, which the RFC says it's a byte, 00, which represents "FALSE". And another string, so we need 4 bytes for its size, which gives us 0x0F. And then... just the proverbial crickets chirping away, no sign of our string with the signal name.

Having a somewhat fertile imagination, I have a tendency to imagine all sorts of extraordinary causes for these problems, sometimes involving tiny green leprechauns misplacing bits and bytes and whatnot.

The truth, such as it is, is usually achieved by simplicity. So, I've decided to follow my first obvious tip of the day, above. I ran the same command on another server. And got:

0000: 62 00 00 00 00 00 00 00  0B 65 78 69 74 2D 73 69 : b........exit-si
0010: 67 6E 61 6C 00 00 00 00  04 54 45 52 4D 00 00 00 : gnal.....TERM...
0020: 00 00 00 00 00 00                                : ......

[libssh2] 12.655970 Conn: Channel 0 received request type exit-signal (wr 0)
[libssh2] 12.655970 Conn: Exit signal TERM received for channel 0/0

So, if your remote process is getting an abrupt termination and you're not getting an exit signal, there's a good chance the remote sshd may be at fault.