The project I’m currently working on involves a search engine which returns its results in XML over an HTTP connection. I decided to use an XML writer rather than make the same mistake again. After looking around for an open source solution and not finding anything suitable for my needs, I decided to roll my own and place the project on Google Code.

To use xml-writer, simply drop xml_writer.h into your project directory and #include it as necessary. It is a very simple (less than 200 LOC) class. The accompanying unit tests also serve as documentation for its use.

Briefly, you instantiate an xml::writer object bound to an std::ostream. This allows you to send the resulting XML to a string (std::stringstream), a file (std::fostream) or any ostream compatible object. In my case, I created a stream object which writes directly to an HTTP connection.

You instantiate xml::element objects to actually create the XML element tags. An xml::element object has overloaded attr() methods to write element attributes and contents() to write element contents. When an xml::element object goes out of scope, it will write the necessary end element tag.

A simple example:

#include "xml_writer.h"
#include <sstream>
#include <iostream>

using namespace std;

int main()
{
    stringstream ss;
    xml::writer xw(ss);
    {
        xml::element record("record", xw);
        record.attr("name", "fred").attr("age", 35);
        record.contents("yabba dabba doo!");
    }
    cout << ss.str() << endl;
    return 0;
}

will result in the following output:

<?xml version=”1.0″ encoding=”utf-8″?><record name=”fred” age=”35″>yabba dabba doo!</record>

NB: the extra scope created to encapsulate the record element was necessary so that the xml::element destructor is called before the resulting string is output. xml::element should always be enclosed in some kind of local scope for this to work properly.

using (var compressor = new DeflateStream(data, CompressionMode.Compress))
    compressor.Write(bytes, 0, bytes.Length);

data is a MemoryStream that I used to serialize the compressed data to disk.

zlib refused to read it. It kept throwing a “zlib error” exception.

It turns out that although they use the same compression algorithm, zlib expects a header in front of the data and an adler-32 checksum at the end. The code is now:

data.WriteByte(0x58);
data.WriteByte(0x85);
using (var compressor = new DeflateStream(data, CompressionMode.Compress, true))
    compressor.Write(bytes, 0, bytes.Length);
data.Write(BitConverter.GetBytes(IPAddress.HostToNetworkOrder(Adler32(bytes))), 0, sizeof(uint));

the adler-32 code:

// naive implementation of adler-32 checksum
int Adler32(byte[] bytes) {
    const uint a32mod = 65521;
    uint s1 = 1, s2 = 0;
    foreach (byte b in bytes) {
        s1 = (s1 + b) % a32mod;
        s2 = (s2 + s1) % a32mod;
    }
    return unchecked((int) ((s2 << 16) + s1));
}

and zlib is happy again.

If you’re wondering what the magic numbers are, take a look at RFC 1950.

Unfortunately the build failed. Bjam complained that it was missing a dependency (gzio.c). After a bit of trial and error with various boost.build options, I found the problem. I checked the zlib change log and it turns out that in version 1.2.3.9 they removed the gzio.c file from the zlib sources.  Unfortunately, this was a very recent change so boost.build has not yet been updated to reflect the current zlib.

The fix is simple. Look in the boost source tree in the /libs/iostreams/build directory. There is a file named Jamfile.v2. Remove the gzio reference from line 136 so that it reads:

crc32 deflate infback inffast inflate inftrees trees uncompr zutil :

That’s all you need to do. Boost should now build and incorporate zlib into boost.iostreams. I added a command to my boost build script to automatically make this change. It uses the Gnuwin32 sed command:

sed -i s/gzio//g "%BOOST_ROOT%\libs\iostreams\build\jamfile.v2"

Here is my complete build script. It builds Boost for Win32 using statically linked boost libraries and C runtime. It assumes that you have boost_1_42_0.zip and zlib124.zip in the same directory as the build script. It also assumes that you have bjam.exe and the following Gnuwin32 tools in your path: unzip.exe, sed.exe, and grep.exe.

setlocal

set BOOST_MAJOR=1
set BOOST_MINOR=42
set BOOST_REV=0

set ZLIB_MAJOR=1
set ZLIB_MINOR=2
set ZLIB_REV=4

set INSTALL=c:\boost

set VC_ROOT=%PROGRAMFILES%\Microsoft Visual Studio 9.0\VC
set TOOLSET=msvc-9.0
set ZLIB_ROOT=%CD%\zlib-%ZLIB_MAJOR%.%ZLIB_MINOR%.%ZLIB_REV%

set PARAM=--without-mpi toolset=%TOOLSET%
set PARAM=%PARAM% -sZLIB_SOURCE="%ZLIB_ROOT%"

set BOOST_ROOT=%CD%\boost_%BOOST_MAJOR%_%BOOST_MINOR%_%BOOST_REV%
set BUILD=%CD%\build
set STAGE=%CD%\stage

path %CD%;%PATH%

if exist "%STAGE%-x86\lib" del /q "%STAGE%-x86\lib\*"

if not exist "%ZLIB_ROOT%" unzip zlib*.zip
if not exist "%BOOST_ROOT%" unzip boost*.zip

sed -i s/gzio//g "%BOOST_ROOT%\libs\iostreams\build\jamfile.v2"

pushd "%BOOST_ROOT%"
setlocal
call "%VC_ROOT%\vcvarsall.bat" x86

set PARAM=%PARAM% variant=debug variant=release link=static runtime-link=static threading=multi
bjam %PARAM% --build-dir="%BUILD%-x86" --stagedir="%STAGE%-x86" stage

cd "%STAGE%-x86\lib"
dir /b | grep -v "\-%BOOST_MAJOR%_%BOOST_MINOR%" | sed "s/^/del /" | cmd

if exist "%INSTALL%" rd /s /q "%INSTALL%"
xcopy /s /i "%STAGE%-x86\lib" "%INSTALL%\lib-x86"
xcopy /s /i "%BOOST_ROOT%\boost" "%INSTALL%\include\boost-%BOOST_MAJOR%_%BOOST_MINOR%\boost"

endlocal
popd

endlocal

One issue I ran into, though, is that there is no good way to examine the databases for debugging purposes. Initially, I used the provided db_dump command-line tool, which was fine for small databases. Db_dump dumps the entire contents of a database, which was OK when I was dealing with databases with only a few records. But now I am working with databases with thousands and soon millions of records. Db_dump just won’t do.

I did a bit of googling but there didn’t seem to be any viewers available for Berkeley DB, so I decided to write one. I was going to do it in C# and WPF because that’s what I’m currently using. But I decided that, since BDB is cross-platform, a viewer for it should also be cross-platform. So I decided to use Qt to build the viewer.

After a couple of nights of coding, here is the result:

The interface is pretty simple. There is a button for opening a BDB database file. Once a database is opened, the “Database” combo-box let’s you pick which sub-database to view (in BDB terms a database is a key/value table and a single file can contain multiple “databases”). The list view on the left shows all the keys in that database and the view on the right shows the contents at that key. Both keys and values are formatted the same as db_dump -p -k formats them.

I put the source code for Berkeley DB Viewer (bdbvu) on Google Code. You can grab it at: http://code.google.com/p/bdbvu/.

To build it you need two things:

1. Berkeley DB, installed and built with C++ support.
2. Qt4.

Once you have those set up and you’ve downloaded the code, you will probably need to change the Qt project file to reflect your environment. I may enventually provide binary downloads, but I don’t have the time for that right now.

Finally, a few things to keep in mind:

• I’ve only tested this with DB_RECNO and DB_BTREE databases, since that’s what I’m using at this time. But, there’s no reason it shouldn’t work with DB_HASH and DB_QUEUE databases as well. (Where have I heard that before?)
• I’ve only tested it with databases with embedded databases, not with stand-alone databases. I know it won’t work with stand-alone databases because the code for that is a no-op.
• I’ve only built and run this on OS X. Theoretically, it should work on Windows and Linux as well, but you never know until you try it.
• When you open a database, it will load all it’s keys into memory. This seems to be pretty quick (a couple of seconds for a 3000+ record database on my laptop) but I may have to change it to use a more scalable method in the future.

That’s it for now. I hope you find this useful.

It turns out that Subversion keeps a copy of the unedited version of your source code in its .svn directory. That was all I needed to build a command line shortcut to OpenDiff. The original source is kept in .svn/text-base and has a suffix of .svn-base. My first attempt was to create an alias:

     $ alias od=’opendiff $1 .svn/text-base/$1.svn-base’

But, it turns out that bash aliases do not support parameter substitution. Who knew?

After a little more digging, I discovered you can define functions in bash. So the shortcut then became:

     $ function od { opendiff $1 .svn/text-base/$1.svn-base; }

This did the trick. I can now start OpenDiff with a command like:

     $ od source.cpp

Put the function definition in your .profile (or in .bashrc or in any of the many places you can put startup commands).

unsigned int var::count() const {
    struct count_visitor : public boost::static_visitor<unsigned int> {
        unsigned int operator () (null_t) const { throw exception("invalid .count() operation on $"); }
        unsigned int operator () (int_t) const { throw exception("invalid .count() operation on int"); }
        unsigned int operator () (double_t) const { throw exception("invalid .count() operation on double"); }
        unsigned int operator () (string_t s) const { return s.ps->length(); }
        unsigned int operator () (list_ptr l) const { return l->size(); }
        unsigned int operator () (array_ptr a) const { return a->size(); }
        unsigned int operator () (set_ptr s) const { return s->size(); }
        unsigned int operator () (dict_ptr d) const { return d->size(); }
    };

    return boost::apply_visitor(count_visitor(), _var);
}

This worked fine with Visual C++ 9 but g++ did not like it one bit.  After a bit of research, I discovered that g++ was right and VC++ was letting me get away with non-standard C++. You can declare a struct inside a function, of course, but it cannot have access to any objects in the function scope. Since this was a member function of the var class, it could not have access to the internals of the var class.

No problem, I thought. I’ll just move all these structs into the global scope right before the functions that used them. Except that I had forgotten that all these structs needed access to the internals of the var class, which they had when declared inside a member function. Once outside the member function, the functor code suddenly broke. So, I moved all these structs right into the var class (destroying the locality of reference which I had been so careful to maintain).

Now it compiled fine with VC++, but g++ still gave me a few type errors. Again these all seemed to be caused  by apply_visitor(),  so I decided to drop most of the apply_visitor() calls in favor of switch statements. This would eliminate the few remaining errors but would, much more importantly, put the relevant code close to where it was actually being used.

The main benefit of using apply_visitor() is that if you forget to handle a type, it will generate a compile-time error. I used switch statements whose default block throws an exception. I traded compile-time checking with run-time checking. I can live with that.

Luckily, I had plenty of unit tests in place. I tend to work in increments. Make all necessary changes to a function, compile and test. Repeat until done. There were quite a few times when my changes compiled but failed when I ran the unit test suite. Once I decided to make these changes, it took a couple of hours to make them all and test the results. Everything builds cleanly in Visual Studio and XCode now.

The PDF format consists of a series of objects, expressed in a simple syntax based on PostScript. There are primitives such as strings and numbers, and there are collections (arrays and dictionaries) which can contain both primitives and containers. You can see how things quickly become complicated when you have dictionaries containing arrays containing other complex objects.

So, I built a parser which would collect a stream of objects into a data structure for further processing. Since there was no way to know ahead of time the shape of the final structure, I used Boost’s variant library to contain each object. An object could be a primitive type or a shared pointer to an STL vector or map, depending on the underlying structure. The STL collections contained the same object class, so the resulting structure could be arbitrarily  complex.

After I got the basic parser working, I realized that I had implemented a dynamic typing system for C++ . . . poorly. Even though you could build these complex and dynamic data structures at runtime, the syntax necessary to manipulate them was a bit ugly. For  example, if you had an object (named ‘o’) that happened to be an array and you wanted to access the second element, which happened to be a string the syntax would be:

std::string s = (*o)[1];

Not exactly horrible, but it could be cleaner.

It also seemed like a good project on its own, so I tore it out of the PDF parser and rewrote it. Twice. I called the class ‘var’ so that the syntax for declaring a dynamic object would look familiar to anyone used a dynamic language. i.e. you could say:

var name = "fred";
var age = 35;

But I made a mistake in trying to make var a subclass of boost::variant<…>. I had the same issues getting the usage syntax under control. On my second attempt, I made the variant a private member of the var class and explicitly defined every supported operator. This made it simpler and more intuitive to use. The previous example became:

std::string s = o[1];

Also, because I was focused on building a stand-alone dynamic type system rather than a data structure to support another project, I put a bit more thought into making object initialization easier. For example you can do this:

var a = new_array(1)(2)(2.5)("hello")(new_dict("name", "fred")("age", 35));

which creates a five element array named ‘a’ which contains two integers (1 and 2), a double (2.5), a string (“hello”) and a dictionary with two name-value pairs. Note that in a dictionary both the name and the value can be any type.

There is also a null type, represented by a $. The default value of any uninitialized var is $, but you can set it explicitly:

var name = $;

I picked $ because it seems to be supported by many C++ compilers as a valid character in a variable name, but is not frequently used. It stands out.

There is one issue with null values that I haven’t resolved. If you try to retrieve a dictionary value whose key does not exist, it will return a null value, but it is perfectly valid to place a null value into a dictionary. e.g:

var d = new_dict("a", "xxx")("b", $);
cout << d["a"] << endl; // prints: xxx
cout << d["b"] << endl; // prints: $
cout << d["c"] << endl; // prints: $

When operator[] returns $ on a dictionary, you don’t know if it was because the item didn’t exist or it existed but was $. What’s worse is that if it didn’t exist before calling operator[], it would exist afterwards (and be set to $). This is a consequence of the fact that operator[] returns a var&. It has no idea if it’s being used as an l-value or an r-value, so it has to create the instance in case it is being used as an l-value. i.e.

d["e"] = 1;

You can down load dynamic-cpp from Google Code at: http://code.google.com/p/dynamic-cpp/

To use it, you need the following code:

#include "dynamic.h"

using namespace dynamic;

Also, you need Boost. I’ve been using 1.37.0, but older versions should work as well. You need to build the project and link against the resulting static library. The code is platform independent but so far I have only Visual Studio project files. I will try to resolve that soon.

It comes with a suite of unit tests which will give you a better idea of its capabilities.

Let me know if you find something useful to do with it.

Having spent a good portion of my career building Windows apps, I’ve had the opportunity to create UIs using the Win32 API, OWL, MFC, WTL and wxWidgets. I’ve dabbled in WPF but never did much with it since I’ve been spending most of my free time tinkering with Cocoa and Cocoa-Touch. What I really like about using XAML is that you can lay out an entire interface, including a lot of behavior without writing a single line of code.

If you really want to, you can lay out the entire interface in XAML directly. But, if you’re like me and can’t stomach working with XML except for making minor tweaks here and there then Blend is a great tool to have in your design toolkit.

Prior to this, I had no idea what Blend was. I saw it on the MSDN subscriber downloads site, but it wasn’t clear to me exactly what it was. It was obviously some kind of design tool, but the description on the site made it sound more like a PhotoShop or Illustrator competitor than a UI design tool.

Well, it is a UI design tool. A very powerful if not always intuitive one at that.  It complements Visual Studio nicely. The intent is that a designer would use Blend to design the interface and a programmer would use Visual Studio to implement the actual application. You can open the same solution (.sln) files with either product, although you need to be careful if you open the same solution with both products at once. Which version of a file is on disk depends on who saved last.

You can build and run solutions with Blend and you can also make UI changes in Visual Studio, but you will usually want to use each product for the task is more suited to do.

If you’re interested in building SilverLight or WPF apps (Blend supports those too), I would recommend attending one of these sessions. If you can’t get to one, they also have them online. Google xaml fest to find the next one.

1. Read a block of binary data.
2. Use struct.unpack() to break out individual fields.
3. Create a dictionary from those fields using the appropriate key names.

So, let’s suppose I needed to read a block which contained a name (20 characters), an age (unsigned integer) and a salary (float). The code might look something like this:

import struct

(name, age, salary) = struct.unpack("< 20s I f", src.read(28))
record = {'name' : name, 'age' : age, 'salary' : salary}

That’s not bad, but it starts getting tedious to have to calculate the size of the block and construct the dictionary manually each time, so I wrote a small function to do all that automatically:

import struct

def readStruct(src, format, names):
    s = {}
    for nv in zip(names, struct.unpack(format, src.read(struct.calcsize(format)))):
        s[nv[0]] = nv[1]
    return s

The code to read the record from the previous example then becomes:

record = readStruct(src, "< 20s I f", ('name', 'age', 'salary'))

It’s a small change, but it has made working with binary files easier and less error-prone. The next step will be to combine the format and names parameters into a single format string that has the names embedded in it. We’re going to use regular expressions to achieve that goal.

The updated code is:

import struct
import re

def readStruct(src, format):
    rex = re.compile("\{([a-zA-Z_]+)\}")
    names = rex.findall(format)
    format = rex.sub("", format)
    s = {}
    for nv in zip(names, struct.unpack(format, src.read(struct.calcsize(format)))):
        s[nv[0]] = nv[1]
    return s

and the previous example now becomes:

record = readStruct(src, "< {name} 20s {age} I {salary} f")

Which, I think is a lot easier to write and read. Note that you don’t have to interleave the formatting and name strings. You could just as easily have written the format string as “<20sIf{name}{age}{salary}” but I think interleaving them makes their use much clearer.

Sometimes, we can encode multiple return values into one. For many of us, the first C idiom we learned from K&R is processing input a character at a time:

int ch;
while ((ch = getchar()) != EOF) {
    // do character processing...
}

This works because the EOF macro was set to something outside the range of valid characters (usually -1). While this approach can work fairly well for simple cases, it quickly breaks down as the types we wish to return get more complex.

One way to return multiple values is to return one value through the function return mechanism and to pass pointers or references to objects which will receive the additional parameters as additional arguments to the function. For the sake of example, we are going to create a simple function which returns a single line of text from an input stream. This function will take one input parameter, the input stream, and return two values. The first value will be a boolean to indicate success and the second value will be the string that was read. Using the approach just described, we could code it up as:

bool readline(istream& is, string& line)
{
    if (is.eof())
        return false;
    getline(is, line);
    return true;
}

string line;
while (readline(ss, line))
    cout << line << endl;

struct line_ret
{
    bool    success;
    string  line;
};

line_ret readline(istream& is)
{
    line_ret lr = { false, "" };
    if (!is.eof()) {
        lr.success = true;
        getline(is, lr.line);
    }
    return lr;
}

for (;;) {
    line_ret lr = readline(ss);
    if (!lr.success) break;
    cout << lr.line << endl;
}

#include <boost/tuple/tuple.hpp>

using namespace boost;

tuple<bool,string> readline(istream& is)
{
    if (is.eof()) return tuple<bool,string>(false);
    string line;
    getline(is, line);
    return tuple<bool,string>(true, line);
}

for (;;) {
    tuple<bool,string> ret = readline(ss);
    if (!ret.get<0>()) break;
    cout << ret.get<1>() << endl;
}

for (;;) {
    bool success;
    string line;
    tie(success, line) = readline(ss);
    if (!success) break;
    cout << line << endl;
}