Wednesday 30 May 2012

Organizing OSS on Windows

In my old days as a developer, the development environment was a simple affair. I had an IDE, and it would take care of most of my equally simple needs.

The first time I had to deal with installing/configuring additional components/libs was with Delphi. But even that was simple. Same thing with C++ Builder. The most complex thing I remember doing back then was installing SGI STL. And even that was simple.

These days... as I said, if you're working on Windows, you either use Visual C++, or you're in for a whole lotta work. Thankfully, you're also in for a whole lotta learning.

So, after installing or building perl, OpenSSL, zlib, libssh2, Boost, mingw, msys, git, msysgit, msmtp, etc, my HD is a bit of a mess. You can't properly plan for what you don't know, and I never found (in truth, never looked for, either) a guide on how to organize these things on Windows.

So, I've decided to roll my own. It all boils down to 3 points:
  • Includes
  • Binaries
  • PATH (for additional utilities)

I'll start with two main folders
  • tool
  • lib

tool will have stuff like perl, mingw + msys, git, msysgit, or msmtp. These are not libraries, they're tools necessary for the job. Where possible, I'll have a main folder for each tool (e.g., msmtp), which will contain a folder for each different version (e.g., msmtp-1.4.27, msmtp-1.4.28). It's important to keep track of your PATH (on System, on Windows Control Panel), to make sure it's pointing to the version you want.

lib will follow the same principle. Taking libssh2 as an example, I would have a main folder, say libssh2, and sub-folders for libssh2-1.4.1 and libssh2-1.4.2. When dealing with libraries, we have to take care of the include and lib paths. Include is rather simple, but lib has several options - dynamic vs. static and debug vs. release.

gcc (and that's 3 consecutive paragraphs starting in lowercase) uses dynamic linking, by default. This means if the lib path contains both a dynamic and a static version of the same lib, it'll use the dynamic (and, naturally, it'll require it at runtime). So, where possible, I'll try to have two lib folders - one with dynamic libs (both debug and release), and another with static libs (also, both debug and release).

Another advantage of having your own main folder above the lib's/util's main folder is that it gives you a handy and - hopefully - uncluttered location for any notes you may find useful (e.g., how to configure msmtp to use your gmail account to send e-mail). Yes, I know there's a plethora of utilities to store these notes and to sync with more cloudy devices than you care to imagine. This is just another alternative.

And that pretty much covers it all for now. When I do get around to clean up this mess, I'll post the results here.

Sunday 27 May 2012

libssh2 is built. Now what?

Now I've got libssh2 built, the time has come to actually create something.

My goal is to have a reliable way to run a command (or group of commands) on a remote server, show the command's output, and allow the user to select part of that output to run another command, and so on. The commands will be predefined, so it's a kind of "specialized SSH session".

The first test was, obviously, running a command on a remote server. The example program ssh2_exec.c does it out of the box, so that's where I started. And I got success on first run.

However, I needed to run several commands. After googling it, I was overwhelmed by the lack of examples on how to do it, which led me to the conclusion it might not be such a simple task. So, I've decided to take a look at the protocol's RFCs, particularly RFC 4254, the Connection Protocol.

And I've found the info I was looking for here (my emphasis):
"Once the session has been set up, a program is started at the remote end.  The program can be a shell, an application program, or a subsystem with a host-independent name.  Only one of these requests can succeed per channel".

So, this means the life cycle for multi-command execution is: 1) open a channel; 2) execute the command; 3) close the channel. Rinse. Repeat.

This means there'll be no way to maintain state between executions. E.g., if I run "cd logs" and then run "ls -lt | head -1", the second command won't be run in the logs directory. In order to get it to work I either need to run a single command-line "cd logs; ls -lt | head -1", or put these commands on a shell script and run it.

BTW, the second option applies to anything you might want to do on the remote host. Assuming you've got somewhere to write a file, you probably can maintain state.

Thursday 17 May 2012

Finally, building libssh2

Building libssh2 turned out to be simpler than building OpenSSL. Partly, because of the lessons I learned building OpenSSL.

I went straight for the win32/GNUmakefile. I set my paths for zlib and OpenSSL, after checking how the vars OPENSSL_PATH and ZLIB_PATH were used, i.e., if I should point them to the sources or the libs I built. In case you're wondering, I pointed to the libs.

Then, I got in trouble because of _libssh2_gettimeofday(). Apparently, getting the time of day on Windows using anything other than VC++ is terribly complex (or, rather, complicated). Once again, I searched the net, but this time found no conclusive answer. So, I decided to improvise, and turned

#if defined(LIBSSH2_WIN32) && !defined(__MINGW32__)

into

#if defined(LIBSSH2_WIN32)
// && !defined(__MINGW32__)


One step forward, problem "solved" (although I can't claim to be entirely certain of the consequences).

Then, GDI decided to rear its ugly head again, courtesy of OpenSSL, so I had to add it here:
LDLIBS += $(OPENSSL_PATH)/lib/libcrypto.$(LIBEXT) $(OPENSSL_PATH)/lib/libssl.$(LIBEXT) -lgdi32

And bingo! We have build!

Then, I built an example. Looking for something simple, I went for ssh2.c.

Compiling was simple enough:
gcc -c -g -Wall -DUNICODE -DQT_LARGEFILE_SUPPORT -I"<PATH TO LIBSSH2>/include" -o ssh2.o ssh2.c

And, this time, so was linking:
gcc -Wl,-subsystem,console -mthreads -o ssh2.exe ssh2.o -L"<PATH TO OPENSSL>/lib" -L"<PATH TO ZLIB>/zlib" -L"<PATH TO LIBSSH2>/lib" -lssh2 -lzlib1 -lssl -lcrypto -lws2_32 -lgdi32

Regarding the options used, I've copied what Qt Creator uses, to maintain the similiarity between what I'm doing on the command line and what I'll be doing on the IDE.

I intend to run some tests with static linking and dynamic linking, so I tried to follow GCC's rule about libraries with dependencies being first on the list. If you're just linking dynamically, then it should probably be a non-issue.

Saturday 12 May 2012

Building zlib

Although zlib is optional for libssh2, I decided to build it. Compared to OpenSSL, it was rather uneventful.

The only problem I had was that the makefile makefile.gcc in <zlib_home>/win32 said:
# To compile, or to compile and test, type:
#
#   make -fmakefile.gcc;  make test testdll -fmakefile.gcc


which led me to cd to the win32 dir and run make. Which dutifully ran and promptly failed. After some searching on the net, I came upon a post stating that I should run make on the zlib home, like this:
make -fwin32/makefile.gcc

And, after giving it some thought, I realized it made perfect sense, since the makefile doesn't qualify the files with a path, which means it expects to find them in the current dir.

I was going to attempt building a zlib example, and settled on gun.c, an unzip example. However, I stumbled upon some problems with the use of stdunix.h and functions like chown() and unlink(). I could work this out - replacing these functions with their Win32 counterparts, where appropriate; or installing MS's Subsystem for Unix Apps. But, in the end, I decided to move on to libssh2. If I ever need to actually use zlib, I'm sure I'll be able to sort it out.

Thursday 3 May 2012

Building OpenSSL

Of course, before one builds libssh2, one must build OpenSSL and zlib. Today, I'll talk about OpenSSL. On Windows.

First, install perl (I used Strawberry perl), and MSys. I've looked for other options for building on Windows without VC++, and found none I could use. Fortunately, everything went smoothly - the build process was long, but uneventful.

Then, I decided to build a simple example from a tutorial. I built step by step - add one line, rebuild all; repeat. Right on my first attempt, I was hit by over 1.5K lines of undefined references.

I was using Qt Creator, so I decided to build it from the command-line myself. The first thing I learned was that I wasn't configuring the libs correctly for the linker.

The linking looked something like this:
gcc -Wl,-subsystem,console -mthreads -o OpenSSLTest.exe main.o <PATH TO OPENSSL>\lib\libcrypto.a <PATH TO OPENSSL>\lib\libssl.a 

I finally got it semi-working with this:
gcc -Wl,-subsystem,console -mthreads -o OpenSSLTest.exe main.o -L"<PATH TO OPENSSL>\lib" -lssl -lcrypto

"Semi-working", because I still had a few undefined references. But these were all Windows symbols, so I was on more familiar ground. And, I finally got it working:
gcc -Wl,-subsystem,console -mthreads -o OpenSSLTest.exe main.o  -L"<PATH TO OPENSSL>\lib" -lssl -lcrypto -lws2_32 -lgdi32

Yep, there at the end it's gdi32. Those, like me, unfamiliar with OpenSSL, might be asking why a "toolkit implementing SSL v2/v3 and TLS protocols with full-strength cryptography" was BitBlting and creating and deleting DCs. I know I did. Which led me to rand_win.c, and its

static void readscreen(void)

Looking it up on the web, it apparently uses screen content to generate pseudo-random numbers. I've also found some notes about this function not being used, since processes running on a server environment are not likely to have a UI (besides, it is called from RAND_screen(), which is stated as "function available for backward compatibility"). Still, being a newbie, even though I'm not likely to need it, I've decided to keep it in the build and just link with gdi32 (as we'll see in a moment, it won't matter that much, anyway).

I also confirmed that "dynamic-link-by-default" was working. I linked that simple program against gdi32, and checked it with Process Explorer, and, sure enough, there was gdi32.dll mapped into the process. Then, I commented out all OpenSSL calls, and built it without gdi32; as expected, no gdi32.dll. As a final test, I kept the code commented out, but linked against gdi32. I was pleasantly surprised to see that Process Explorer showed no gdi32.dll. Yes, I know all this is rather obvious, but I did say I was a newbie, didn't I?

I thought about toying a bit with /DELAYLOAD, but I found out that mingw-32 doesn't have that option.

As a final test, I looked for a semi-useful complete example. I got it to build on the command line, using the exact same command I used above.

Feeling more confident, I headed back to Qt Creator, and built it from there. And got a clean build right at the first attempt, with this .pro:

TEMPLATE = app
CONFIG += console
CONFIG -= qt

SOURCES += main.c

INCLUDEPATH += "<PATH TO OPENSSL>/include"

LIBS += -L"<PATH TO OPENSSL>/lib" -lssl -lcrypto -lws2_32 -lgdi32

Oh, and I found out that Windows now accepts both "\" and "/" on paths. That's a welcome change.