C++ Soup!

The C++ Network Library: New Release and Updates

noreply@blogger.com (Dean Michael Berris) — Wed, 21 Mar 2012 13:32:00 +0000

After a bit of development (and a little stagnation) since the 0.9.2 release, I'm proud to announce that cpp-netlib 0.9.3 is now available for general availability. This release will be supported for one year until March 30, 2013. You can download the official packages from the Github project downloads. This release was managed by Glyn Matthews and there are a number of important changes in 0.9.3 that addresses issues with 0.9.2. There are a number of bugs fixed and lots of simplifications internally.

Here's a quick run-down of what has changed between 0.9.2 and 0.9.3:

Static libraries for URI and HTTP Client connections. This dramatically reduces the compile-time costs of using the HTTP client and the URI parsers.
A new interface for building URIs. The URI builder follows the directive style that's used in the request and response objects.
Fixes to C++0x/C++11-supporting compilers (MSVC 2010 in particular).
Updates to the MIME parser implementation.

There are a few issues that are being worked on and we're planning on doing a 0.9.4 release or a patch-set for 0.9.3. If you do find issues we encourage you to file them on the official project issue tracker.

The next major release we're planning will feature the move away from and removing support for the header-only implementation. There's also a discussion on the mailing list about what the focus for the development of the library should be moving forward. If you're interested in the topic, you can checkout the discussion on the mailing list.

Major changes are going to come from my personal fork of the project. I've just recently completed the refactoring of the HTTP client implementation to make it easier to maintain and extend. In lieu of header-only template metaprogramming, I've made changes to use traditional polymorphism in places where they make sense. This allows us to maintain a stable interface against which other developers can program against to implement many new techniques that users can choose to customize the behavior of the HTTP client.

We're still in the process of planning whether we're going to release a 0.10 which contains these changes or whether we're going to straight to 1.0. C++11 support is also something the community is looking to jump to. We'd love to hear what the users think, so feel free to post comments on this post or better yet join the mailing list to get involved in the discussion.

In the meantime we're hoping to get more users and more contributors to the project. Although we are too late for applying to the Google Summer of Code effort, we are ramping up our efforts to get more contributors involved in the project. I'm personally putting in my 20% time to make sure that I can mentor new developers and make myself available to allow people who want to contribute to answer questions on the mailing list. I'm also going to be hosting regular hang-outs as well for live face-to-face interaction with users and developers in the community.

The goal of cpp-netlib has definitely changed now from something that's just a header-only modern C++ networking library to making modern C++ network programming easy for everyone. We encourage users and developers alike to raise ideas and questions to the mailing list so that we can as a community make network programming in C++ easier and more accessible to more people. It is also with this goal that we're hoping to get a proposal to the ISO C++ Committee for a high-level HTTP client interface accepted.

In behalf of the C++ Network Library group of maintainers and community members, we look forward to getting more contributors and users this year and in the coming years. We're all committed to achieving the new goals of the project and look forward to your feedback and patronage.

Cheers!

Post-Kona Mailing and More Libraries

noreply@blogger.com (Dean Michael Berris) — Tue, 06 Mar 2012 14:08:00 +0000

Beach, laptop, flip-flops, and C++. Kona, Hawai'i.

It took a while for me to get to writing up my experiences on the first ever ISO C++ Committee meeting that I attended in Kona, Hawai'i. There are around 70 attendees and I'm one of the few first timers to the meeting. What happens when you get a group of C++ developers, implementers, and language lawyers in the same venue for 5 days to talk about C++? Lots of fun and generally productive discussions. There's a lot of new things to look forward to in the upcoming versions of the C++ standard. Here's a high-level overview of what transpired in Kona and what I personally think would be nice to see in the next version(s) of C++.

I mention versions because the committee has decided that we'll be working on getting a short-term update to the standard tentatively shooting for 2017 (C++1y or C++17) and a long-term update aiming for 2022 (C++22). The Evolution Working Group (EWG), the working group whose focus is in the addition of new features, was where the Rich Pointers (N3340) paper was discussed. I spent a lot of my time attending the EWG meetings hearing about what the potential focus for C++17 and C++22 may be.

There's a lot of discussions but the big-ticket items I've listed down below:

Static If -- a means of simpler compile-time programming with a more natural and familiar syntax. One motivating factor and use-case is a replacement for SFINAE tricks with templates.
Concepts (better known as Constrained Templates) -- a means of enforcing syntactic and more importantly semantic requirements for types in templates. There's a lot of synergy and potential for integration of the work between the proponents of Static If and Constrained Templates so it would be nice to see great minds get together in finally making this happen in C++.
Modules -- not dynamic libraries (although those would be nice to have too) but more on logical grouping of symbols for replacing header files and #include. There are two different proposals on Modules for C++ and there's a Study Group formed to address this particular issue. This looks to be one of the things that might be making it into C++17.
Software Transactional Memory -- an update on the state of the art and progress makes it very encouraging. It's very exciting technology that potentially makes a lot of concurrency problems more manageable going forward. With hardware assistance apparently coming soon (this year or early next year from the major vendors) this proves to be something worth keeping an eye and an ear on.
Reflection -- this is where Rich Pointers fall under along with a proposal for static reflection. The general feedback is that a more holistic view of reflection be taken and a more thorough presentation on the matter would be appreciated in upcoming meetings.

These things are still in the proposal presentation stage where interest is gauged and potential enhancements and issues are explored. The upcoming meetings will see more fleshed out proposals and potentially more in-depth discussion of the details once the committee decides on the one big thing to include in C++17 and what two other big things to try to get into C++22.

In other potentially more important news, the C++ committee has issued a Call for Papers for more libraries to bring into the standard library specification. Here's a quote from the announcement:

The C++ standards committee is soliciting proposals for additional library components. Such proposals can range from small (addition of a single signature to an existing library) to large (something bigger than any current standard library component).

In a change from past practice, the committee is processing multiple library work items in parallel, and any resulting domain specific technical reports will ship when ready rather than waiting for completion of single large technical report.

This call for proposals in open ended; there is no cutoff date. As a practical matter, the committee is early in the post-C++11 revision cycle, and so the next year is a particularly good window of opportunity for library proposals.

The committee welcomes proposals with or without formal standard wording (often known as "standardese"). This is a change from past practice, and is intended to encourage library developers to step forward with proposals.

The motivating factor here is to dramatically increase the number of libraries that come with a C++ implementation. The committee is favoring library proposals that already intend to standardize existing practice. This means we might be able to see ICU's features made part of the C++ standard with a different (more consistent with the current standard library style) interface. We might also see Boost.Asio's interface and specification as something that comes standard to standards-compliant C++ implementations.

In a similar vein I'm going to also be working with the Networking Study Group (SG4) that will focus on getting a Technical Report (TR) out that focuses on networking libraries. The hope is that we can cover various layers of the networking stack represented well in the C++ standard.

There are now 4 study groups, and SD-3 details what these are and what the Study Groups mean to achieve. Here's the list copied verbatim from the mailing:

WG21 currently has four SGs and appointed chairs.

SG1: Concurrency and Parallelism (Hans Boehm)
SG2: Modules (Doug Gregor)
SG3: File System (Beman Dawes)
SG4: Networking (Kyle Kloepper)

An SG is not a formal project, and so ISO does not formally appoint an Editor. However, some SGs will be working on a document, and those SGs should havesomeone who acts as editor to maintain the working draft of that document, and publish each draft revision as an N-numbered paper. The SG can approve changes to its own working draft, but the understanding is that the final product will be a submission to a full WG21 meeting for further processing.

The Library Working Group (LWG) is working on multiple concurrent efforts to publish logically-grouped TRs that focus on a distinct areas. The idea is that these TRs can be published when they're ready independent of whether or how the C++17 effort is going. Of course changes to the working draft may affect the interfaces of the libraries being defined in the TRs, the agreement has been largely that any new features that go into C++17 will be incremental and non-intrusive to C++11, which means the LWG can focus on TRs that use C++11 as the core language in the libraries specified in the TR.

Call to Action

So this post is not just about a report, this is also a call to action. Let us (the C++ committee members) know what libraries you'd like to see in every C++ implementation, what's lacking, and what you want us to address in the standard library and the language. It was mentioned several times in the meetings that the committee needs more user feedback and representation from actual users, not just the experts. We'd like to hear what people think should be the direction of C++ and how we can make lives of users easier in terms of libraries and actual language features.

Thanks very much and I do hope this update helps.

Rich Pointers: Frequently Asked Questions

noreply@blogger.com (Dean Michael Berris) — Mon, 23 Jan 2012 07:34:00 +0000

Towards the end of last week the Pre-Kona ISO C++ Committee Mailing was released. This featured a few very important papers for the Evolution Working Group (EWG) regarding Concepts (N3351), Modules (N3347), and Static If (N3322). Rich Pointers (written by yours truly along with comments from Lawrence Crowl and Matt Austern) is N3340 got a few questions asked from the EWG mailing list and in Google+ (as well as through emails directly to me). Even though I have responded to these questions as thoroughly as I could, I've found it prudent to post an FAQ to make the issues raised a little more exposure.

Why a new pointer type? Why not just extend RTTI?

If you've read through the whole paper, it doesn't do much to say why a new pointer type is being proposed as a viable solution to the different motivations for the paper (see my other post about rich pointers for background). It does mention some hints that bear repeating and pointing out:

We don't want users to pay for what they don't use -- more precisely, we only want the compiler to generate the type descriptors for types that are used through rich pointers.
We don't want to break existing code that rely on normal pointer semantics that's why we didn't go about changing how pointers work. Backward compatibility is important.
We don't want to tack the implementation onto existing runtime type information (RTTI) for backward compatibility. There's code out there that has been built and linked to rely on std::type_info being a certain size and layout -- changing this on newer libraries/binaries will cause ABI compatibility issues that we don't want to introduce.

Although the proposal does hint that it can use RTTI in cases when upgrading from a normal pointer to a rich pointer, this is only going to be true if RTTI was enabled in the compilation of the module in which the type was defined. This cannot be guaranteed and this is by design as well.

Why not just a smart pointer with an assist from the compiler to generate the descriptors?

This may be feasible and as the paper points out there is a way of achieving this. Unfortunately the following cannot be guaranteed by a library-only implementation:

Deterministic runtime registration of types from libraries. There's currently no way to ensure that a rich_ptr will be able to contain the pointer to the correct type especially if the source is a normal pointer.
Deterministic invalidation of already-registered types. There's no cross-platform and standard way for registering new types in an application's runtime that also guarantees that there's only one definition of a type, and that objects of that single type can be referred to using the correct pointer type.
Clear upgrade path from one version of a type to the latest version of the type. Similar to the above point, there's no clear and cross-platform way of doing this.
Automatic and selective generation of complete types and those types it depends on in its definition. Even with a compiler assist, making this happen with a normal template class and the type for this given template is far too much work for the compiler as opposed to having the compiler treat rich pointers as a special type of pointer.

There are also many more reasons which involve the definition of a cross-platform runtime interface, which will definitely vary from platform to platform -- even though that part will definitely be a library, a library-implemented type cannot get too much special treatment that violates the basic rules of C++.

Why use the '%' character and not another character (like '^' or '@')?

There's really no strong reason to use %, but there are reasons to reject '^' and '@'. The caret character ('^') although ideal to use similar to how Pascal pointers work, unfortunately is already used by Microsoft in their proprietary C++/CLI implementation. Not chosing the caret was an obvious decision to avoid the potential clash and objections with this extension to standard C++. The at character ('@') although has been used in the Ruby programming language, isn't a special symbol (nor a valid symbol) in C++ -- making @ a valid operator character opens up potential major surgery for current compilers and the grammar. In contrast, the percent character ('%') is already a valid symbol and operator in C++ that has a binary operator counterpart -- this is symmetric with the asterisk ('*') which is both a unary and binary operator version.

I personally would also like to think that the percent symbol denotes the presence of two things on different sides of the common line -- a pointer that refers to an object (top circle) and the type descriptor (bottom circle). Of course this is a mnemonic device and symbolism that's very far from existing practice and usage of the percent operator (as a modulo operator), I'd like to think I tried to give it some meaning that's relevant. ;)

Is there an implementation available?

Short answer: Not yet.

Long answer: If the committee agrees that this is something worth exploring, I'm very much poised to go ahead and implement it as an extension to the Clang compiler front-end. At this time the proposal is meant to spark discussion as to whether it's something the members of the committee (and the users) will want to see in the language. Practically speaking, there's no point in going forward with an implementation if people don't think the original idea is worth implementing in the language anyway.

Conclusions

The Kona meeting is exactly two weeks away and I'm going to be preparing more materials for presentation among the members of the committee. I'll also be reading the other papers that are in the pre-Kona meeting to prepare for the discussions that I definitely want to be contributing to in the meetings. You can also expect that I will be writing about the meeting afterwards and will be keeping notes as the meetings go on.

I'm definitely looking forward to the meeting -- if you're going, I do hope to see you there. If you're not going and would like to send your feedback on any of the papers and aren't on the committee mailing lists, please feel free to send in your comments either through the comment form or through email.

Thanks for reading this far and I do hope this helps!

C++ Extensions: Rich Pointers

noreply@blogger.com (Dean Michael Berris) — Fri, 13 Jan 2012 14:35:00 +0000

Some time towards the end of 2011 I officially became a member of the ISO C++ Committee as part of Google's delegation. Around the same time I wrote about a few extensions to C++ that I wanted to propose or at least help champion to the next version of C++. The Kona meeting is just around the corner and I'm just about ready to make a case for a specific new extension to C++ that I wrote a paper about called rich pointers. This post is about the motivations behind the extension.

It's 2012 and I've pretty much started it off with a very ambitious project: to add a new pointer type to C++ with something called "rich pointers". This project aims to achieve the following:

Define a new pointer type that conveys semantics different from the primitive pointers inherited from C.
Associates an object's type to the pointer instead of the object.
Provide a richer reflection and runtime introspection facility for C++ types referred through rich pointers.
Attempts to standardize a runtime interface for managing types and these rich pointers.

Here are a few motivating examples for why a new pointer type would be required instead of shoehorning the reflection information into current RTTI implementations.

Suppose we want to implement serialization on any type of object that has a member function called "serialize", how would we do this from a function that's provided through a DLL?
We're implementing a distributed service and we'd like to use Google Protocol Buffers as the data interchange object but also want to render the information using the Qt GUI framework on the client side. How do we do this with as little overhead as possible and without having to adapt the type of the protocol buffer into something that the Qt GUI framework will understand?
Let's say we want a generic registry of handlers implemented as dynamically linked modules for a server application and we want to minimize the requirement for downtime of the application as we add new functionality to it. How do we ensure that the new types introduced in the dynamically linked modules are dealt with accordingly by the server application?
Suppose we're hosting a just-in-time (JIT) compiler that can compile C++ code in a long-running application. We're doing this so that we can reprogram parts of the application according to runtime constraints/inputs (imagine a query optimizer for SQL databases or for regular expressions) for efficiency concerns. How do we allow the new types defined in the runtime-compiled C++ code to be used in the host application in a type-safe manner?

These are just a number of real-world requirements that rich pointers intends to address and enable in a standardized manner. To get this done I'd have to convince the committee that:

Adding a new pointer type unique to C++ and having rich runtime type information is necessary to enable seamless solutions for these kinds of problems.
Requiring the compiler to generate type information for types referred to via rich pointers is doable and will not lead to overly bloated binaries.
Attaching type information to a pointer instead of the object makes sense.
Current RTTI mechanisms are inadequate to achieve this kind of reflection and runtime type introspection.
Implementing this feature will not require too much heavy lifting for implementors.

This paper also implies and relies on the following being true:

That C++ modules will define how dynamically and statically shared C++ libraries will be defined (maintaining backward compatibility to existing linkers).
That the committee will agree to have a standardized API for accessing details about an application's runtime (for type registration, function registration, certain system characteristics of the machine and implementation, etc.).
Runtime introspection is something we'd like to support at the language level because everyone else who needs this functionality has implemented a library that provides this feature in an inadequate and clumsy manner.

In the coming days I will be updating the post with a link to the actual document hosted in the ISO C++ Committee mailings page. I certainly look forward to shedding more details about this project and update everyone of the progress of this proposal and perhaps post some insider notes about the upcoming meeting in Kona, Hawaii.

Thank you and I do hope to be able to give more details soon.

Update: Moved!

noreply@blogger.com (Dean Michael Berris) — Sun, 04 Dec 2011 09:35:00 +0000

The Wordpress experiment has ended and I'm coming back home to Blogger. I'll be posting more here and would be encouraging others who would like to contribute to let me know so that I can add you as a contributor to the site. Here are a few more updates on what's been going on from my end.

I'm now officially a part of the ISO C++ Committee as an alternate to the Google representation! I'll be going to the Kona meeting in February to present a few papers if things go well.
The C++ Network Library project is progressing well. Check out the progress over at https://github.com/deanberris/cpp-netlib for more information.
I'll be travelling again to the Philippines for the holidays, so there would be a lull as I spend most of the holidays offline. Before the three week break I'll be posting more from here.

Thanks very much for the patience with the moves. I'll try to get more content in to make up for the disruption.

Cheers

New C++ Standard, New Focus

noreply@blogger.com (Dean Michael Berris) — Mon, 31 Oct 2011 12:18:00 +0000

Now that the new version of C++, C++11 is out in the wild as an ISO C++ Standard, there's a renewed focus on getting applications, tools, and books up to date with the new standard. The leaders of the ISO Committee have weighed in on the latest versions of the language. There's Herb Sutter writing about C++11, there's Bjarne Stroustrup's C++11 FAQ, and the upcoming C++Now! conference that's going to be held in Aspen Colorado. Clang is racing to get C++11 support into the stable release of the compiler, and GCC is humming along nicely with it as is Microsoft's Visual C++. Instead of write about any specific things, I would lend my opinion on what needs to be the new focus not for C++11 the language, but the supporting toolchains surrounding this new language -- and the version that's coming after this one.

Having been exposed to a number of programming languages and paradigms I'd like to think that I've seen quite a bit of the programming world. There's the dynamic programming languages like Python, Ruby, Lua, JavaScript, and its ilk, and there's the relatively new static languages like Go, Scala, Haskell, Erlang, among others. Systems programming and application programming have new challenges -- concurrency, consistency, safety, etc. -- that need attention from both the programming language and the libraries that come with the programming environment. There are new platforms that are designed to perform more and more demanding computations in very meager amounts of available resources. The web is ever-growing and there's a lot more things web applications need to do both at the front and back-end of the system. And there are yet a lot more applications to be thought in embedded and large-scale systems that current programming languages have yet to even think about (autonomous computing anyone?).

With these in mind, I think C++ has to evolve into a direction that makes simple things easy and hard things possible. There was a time that C++ was the language of choice because much of the infrastructure that's available now was not available then -- you kinda had to write your own tools back in the day, and C++ was the easier programming language to use for large systems. Unfortunately now, even with the new features that makes it easier to write programs in C++ using the new modern style, a big percentage of the programs still turn out to be either glue code or repetitive boilerplate. If you're writing asynchronous systems, they're made easy with lambda's, efficient containers, consistent memory model, and concurrency primitives (or at least standard libraries) but you end up with a lot of boilerplate because there aren't any sane standard HTTP implementations -- interfacing with C libraries will be a lot more painful with C++11 considering how much of the C++ standard defines things outside of the C standard's scope. C++11 is awesome if you're starting from scratch but when you get to the part where you have to tie libraries together, C++ is still an expert's language.

I'm hinting at two things here that C++ should address in my opinion going forward: a standardized way of defining modules for easier composition and higher level standard libraries with reference implementations. Let's tackle each in order here.

First, standardized modules. The reason programming languages like Go, Haskell, and Erlang are getting traction for back-end programming (in the case of Erlang, it's not really a "systems programming" language unlike Go and Haskell) is because it has a simple and standardized module mechanism. Packaging code in modules that can be programmed against (ala Pascal Units) in terms of interfaces makes modular development really easy. Right now all the module implementations in C++ are vestiges to how the linkers are implemented and implementation specific. There's a lot of details here that's worth fleshing out at a later time (there's a lot of content for the blog) but needless to say it should be simple to import modules into a compilation unit for later linkage -- and this should be made standard across implementations.

One of the problems that standardized modules will solve is that of the definition and implementation of the standard library. The STL is awesome as it is at the moment, but it can be a lot more awesome if there's a defined module layout that will be the same across all platforms and that people can implement shrink-wrapped modules that play well with many other modules in a non-conflicting manner. C++ already has namespaces and it's a mechanical manner of tying namespaces to modules. Extending a module should then be a matter of dependency and linkage. Without going much into details, C++ needs to solve the problem of coupling. Many things need to be thought out, like how the runtime environment deals with types and namespaces and how linkage and application binary interfaces can be made consistent without sacrificing performance.

The second thing that does need attention is that runtime environment. I'm not just talking about garbage collection and I'm not talking about having a virtual machine. What I'm talking about is the management of runtime elements, things like memory, I/O, concurrency, among other things. Although it is not impossible to write OS kernels in C++, the C++ runtime is mostly still the C runtime (i.e. too limited) to make things like a global optimizing allocation mechanism, thread scheduling, implementing co-routines, and magical things that makes systems programming interesting very painful to do. A standardized runtime environment can make writing stand-alone systems in C++ easier and less reliant on the C runtime. This can also finally solve things like module initialization (when dynamically loading things), static initialization, memory management, and even reflection and runtime introspection features that can be implemented in a standard manner.

The runtime should be as programmable as the Lisp environments, as minimalistic as the Lua runtime, and as performance-sensitive as the C runtime. It should also be as open-ended as possible, still allowing for implementation-specific extensions but defining a standard core that others can build upon. There are a lot of things to flesh out in this idea but it's something that needs to happen to make C++ a language that's as viable for systems programming as well as application development with all the goodness of the dynamic languages and the safety of static languages. Short of defining a virtual machine, a standard runtime implementation should definitely be considered for the next version of C++. I don't like virtual machines as much as the next person but having a well-defined runtime environment will make systems programming a lot less hack-laden as it currently is.

With these in mind I'm putting some effort into writing up papers to that effect so that hopefully these ideas get at least thought about in the next ISO C++ Standards Committee meeting in Kona Hawaii. Hopefully I can get comments and ideas and a discussion going with esteemed colleagues and users from around the world who might have an interest in joining an effort to make this a reality. Let's make C++ even better by bringing good ideas from other programming languages into C++ without sacrificing the performance and flexibility afforded by our favorite programming language.

Along with this effort I'm going to work on getting the C++ Network Library reviewed for inclusion into Boost and hopefully be a reference implementation for a high-level network library to be made part of the C++ standard. I think it's about time that C++ gets modern programming features and Web-aware libraries part of the standard to make it more attractive to developers who might want to develop high performance solutions in C++ for the Internet. I'll be supporting the standardization of the Asynchronous I/O library that's already part of Boost so that it can be the standard way of programming asynchronous I/O in C++ and made available to more people who do use C++ for network programming.

It's an exciting time ahead for C++ and I fully intend to help influence that future too.

Minus the Header-Only Dogma

noreply@blogger.com (Dean Michael Berris) — Thu, 25 Aug 2011 14:25:00 +0000

I've been working on the C++ Network Library for a few years already total but have been on-and-off with it until recently. One of the first decisions that I made in implementing the library was to try and do a header-only approach to implementing a network library implementation. At first it made perfect sense -- you get the compiler to optimize as much of the code as it can while being able to expose a very friendly and expressive interface. When it was merely a handful of header files and there was just the notion of a message and the HTTP client, all was fine. I was perfectly happy showing off really simple but powerful use-cases where you'd want a C++ application to be able to make HTTP calls and get the contents without having any knowledge of how networks worked or how to program sockets on either Windows or UNIX-like systems. Until I got ambitious and it became a real project that people wanted to use -- then reality became real.

It's not a secret that I like template metaprogramming. If you do a search on the site on 'metaprogramming' you'll find a number of posts dating back to around 2007 detailing the cool things that can be done with C++ templates. I've seen it first hand make really crazy performance improvements in production while I was still working at Friendster. Until recently the server I re-wrote with almost the whole network layer being a huge template metaprogram was humming along nicely. Unfortunately people didn't have the luxury of three things that I had at that time:

They don't have the same level of expertise with network programming. A vast majority of people who want to use cpp-netlib are people who don't want to implement the HTTP protocol from scratch or use a third-party library that was written in C. That said they didn't have the patience or experience necessarily to debug faults in cpp-netlib's implementation nor did they have the motivation to extend cpp-netlib.

They don't have time to wait for a simple HTTP client example to compile. It just took too much time to try to iterate with cpp-netlib since it was header-only and exercised the compilers too much for something that should be relatively simple.

They didn't buy into the header-only dogma. Much of the world was perfectly fine with linking to external libraries either dynamically or statically.

After getting feedback from BoostCon 2010 about the cost of header-only libraries, I then thought long and hard about how to introduce externally linked libraries into cpp-netlib. What came after that was a process of thinking about how to actually get a header-only library to a point where it can support both a header-only build and a non-header-only build. I imagine there aren't a lot of people who would encounter this problem -- if it's even a problem at all -- but I wanted to point out a couple of things that I stumbled upon while "meditating" on this issue.

Make it Optional

One of the first things I had to do was to make the header-only version of the library optional. The rationale for this is pretty simple: compiling template metaprograms costs compiler time and effort in terms of debugging, so if you want to pay this cost you explicitly say it. This makes casual users happier by not having to pay the heavy cost of the compile time every time. This makes sense if you're in development mode and doing lots of builds often enough to not want to wait too long for each build to finish.

In the C and C++ world, the way to make things optional at the source level is to make it a preprocessor macro. This is the first thing that made itself into the 0.9.0 release, the BOOST_NETWORK_NO_LIB macro which when defined makes cpp-netlib a header-only library. This made cpp-netlib a not-header-only-lib by default. This approach allows me (and other developers) to compile the heavy-compile-cost parts of the library into externally statically linked libs once and link them in at the end. The first things that went into the external lib were the Boost.Spirit based parsers. This had cut down the cost of compiling the tests dramatically -- as much as 60% reduction in compile-time.

The approach has its merits because it makes users who want to use the library in header-only mode explicitly ask for it. One of the reasons people would want to do this is to enable further inlining that the compiler can do in header-only mode when building releases. During debug or development mode, linking an external library is just fine if you have a reasonably fast linker.

Abstract Appropriately

The good thing about being able to move things out into compiled external libraries is that you can program against interfaces. This means abstract base types and later binding. This is completely opposite to the original idea of using static dispatch for most everything that I wanted to achieve when I set out to implement cpp-netlib. When I looked at it long and hard, I saw that trying to do everything through static dispatch isn't so scalable in a number of ways:

Extensibility was static. This meant that if you wanted to add an extension or alternate implementation, you'd have to hook into the library's static dispatch mechanisms. This is not trivial and only the bravest template metaprogrammers will even hazard a try to extend the library through the template dispatch mechanisms.

Horizontally scaling was much impossible. This means adding alternate implementations was severely limited by the static nature of the bindings. For example, I had an idea of being able to statically determine which implementation of a particular string processing routine would be called given a tag type. I can see now that doing things this way makes adding support for things like CString (from Microsoft), QtString (from Qt), Boehm's cord/rope, and countless other string types tragically harder than if I use the normal generic programming approach instead of using type tag dispatch.

Debugging is a pain. Not only does it take long to compile things, it takes long for the compiler to see/say where things went wrong. This makes debugging the library implementation hopelessly dependent on my patience on a given day. There were times when I just literally had no patience for compiler errors that I just left the code alone for days without touching it.

Now with the help of dynamic polymorphism I'm able to appropriately abstract some parts of the library that do need the runtime polymorphism -- things like abstracting a connection whether it's SSL or a normal TCP/IP connection, and maybe later on support other underlying protocols like SCTP, and maybe even UDP if there's a use case for HTTP over UDP. This also makes testing the library much easier than it originally was.

Doing it this way also allows me and other implementors to leverage the best of both the static and the dynamic worlds in the same library. One of the ongoing refactoring efforts that I'm undertaking is the development of a mini HTTP REST Service framework which should come out early September. This would allow for developing a service in a traditional OOP fashion while having the efficiency of a header-only network implementation supporting the HTTP server implementation. The same technique allows us to expose a Boost.Parameter based API on the HTTP Client side and hiding much (if not all) of the internal client implementation in a Pimpl idiom -- so the client implementation can remain template-heavy internally and can be compiled once, and linked to many times.

Revelations

I personally am relieved that I now don't feel like I need to fight and feel the pain of being lonely in a template metaprogramming world. Although there are lots of template metaprogrammers now thanks to the excellent books and Boost C++ Library mailing lists, there is still a lot to be done to get template metaprogramming in the mainstream. I now think that the better strategy for getting it out in the hands of more people is by using the techniques and leveraging its power not just as a power tool but also as an adjunct to more traditional object-oriented programming techniques. I am going to be the first to point out that having been exposed to both OOP and template metaprogramming along with functional programming and generic programming that the synergies between/among these paradigms in C++ is what makes it such a unique and powerful programming language.

Hopefully those who are interested in joining and contributing to an open source library that explores the power and synergies of the different programming paradigms in C++ will join the C++ Network Library project. I now host a weekly Google+ Hangout every Wednesday 11am Sydney time (GMT +11) to discuss with developers and users of the library the direction of the project, what's going on, and how the project can proceed. I also regularly post updates to the C++ Network Library mailing list.

For the next release the target is to simplify much of the internal implementations to make it easier to test and understand for new-comers to the project, and along with that define how others might be able to take on more work in the library ranging from: testing, reporting issues, issue triage, updating the website, implementing new protocols, among many of the community/administrative work that comes with an open source project. We're targeting September 2 as the release date for version 0.10 which should feature better compile times and much better testing.

You can add me on Google+ -- introduce yourself and let me know if you're interested in getting notified/invited to the regular cpp-netlib hangout.

C++0x Is Official: Unanimously

noreply@blogger.com (Dean Michael Berris) — Sat, 13 Aug 2011 08:45:00 +0000

Herb Sutter, one of the rock stars of the C++ development world has posted his thoughts and shares with the world that C++0x has been approved unanimously as an International Standard. From his post:

The final ISO ballot on C++0x closed on Wednesday, and we just received the results: Unanimous approval.

The next revision of C++ that we’ve been calling “C++0x” is now an International Standard! Geneva will take several months to publish it, but we hope it will be published well within the year, and then we’ll be able to call it “C++11.”

I want to extend my thanks again to Bjarne Stroustrup for sharing his work with the world and continuing to help move it forward, and to all of the participants whose hard work went into achieving this important milestone in the history of a great language. Thanks!

-- Herb Sutter on Sutter's Mill

Now the real work begins in moving the world from C++98/C++03 to the more sane and much more wonderful C++0x.

Finally us C++ programmers can start looking forward to modern C++ features that makes programming with C++ just that more pleasurable and powerful.

Here's to a brighter future with C++0x!

cpp-netlib 0.9.1 Released!

noreply@blogger.com (Dean Michael Berris) — Thu, 04 Aug 2011 04:36:00 +0000

Today marks another important day in my calendar as I'm proud to announce the general availability of cpp-netlib 0.9.1. You can now download the source packages from the Github download page. If you want to see what's new you can head on to the online documentation page for what's new in 0.9.1. This release fixes a number of issues with 0.9.0 so if you're still using 0.9.0 you might want to upgrade to 0.9.1. That's not the only reason 0.9.1 is good though, and the rest of this post is about giving a little bit more insight into what's new and what's coming in the coming months for the C++ Network Library project.

For all intents and purposes, 0.9.1 should really be version 1.0 -- but there are a few things holding it back from becoming ready for prime-time as a 1.0 version. Here's a list of things that are in there that makes the release extra special:

Support for streaming body callback handler in HTTP client requests. This means you can perform HTTP client requests and pass a callback function that gets called every time a chunk of data is available from the connection. This allows you to handle larger response bodies without having to fit all the data in memory. This is important for implementing downloaders that need to save data straight to a file or in general when you just want to not have to wait for all the data to be put in memory before processing it. You can find out more about this by reading the updated HTTP Client reference documentation.

Default HTTP client is now asynchronous. This release switches the default HTTP client implementation defined at boost::network::http::client to use the asynchronous features. There are no changes required to the original code that used the default HTTP client implementation, but linking to the new version may require you to link in Boost.Thread. You can easily change the default tag by overriding the BOOST_NETWORK_HTT_DEFAULT_TAG macro at compile time or before you include any cpp-netlib header.

Tons of bug fixes from 0.9.0. You can find the list of issues that have been closed on the GitHub issues page. Needless to say the amount of changes that went into the library between 0.9.0 and 0.9.1 are pretty substantial that would have merited a major number advance, but there are more good things in store to wait for to get into 1.0.

The list above is pretty packed but I'd also like to update everyone on some good news regarding the cpp-netlib project. Needless to say they should be treated as great news but I'm not one for exaggeration. Here are a few updates:

There's now a git mirror up on Google Code Project Hosting. At some point in the future -- depending on how large the community grows and/or whether there's popular support for it -- the Google Code Project Hosting site may just become the de-facto development repository of the project. You can find the Google Code Project here.

I'm now officially working on cpp-netlib as my 20% project in Google! This means I can now devote a measurable amount of time and effort into making sure the project succeeds. This doesn't change the project goals and ideals but hopefully it does change the pace of development as I get more effort into it.

I'm looking to expand the developer pool by offering to mentor interested developers who want to learn the techniques and be part of the project. Since cpp-netlib is not my 100% time project in Google I'm not going to be able to do all the fun work that still needs to be done to make cpp-netlib better. Good news is that I'm making myself available to help mentor developers to get up to speed on the project and have more people eventually able to contribute to the effort.

I've found a willing review manager for cpp-netlib's eventual review for inclusion in the Boost C++ Library. It's going to be just a matter of time now before the review is scheduled and I'm able to set time aside to get the review done and over with. The project will continue to evolve and develop even outside of Boost following the Boost.Spirit and Boost.Asio model and hopefully get more people contributing to Boost through the cpp-netlib project.

If you're interested in the mentoring, please join the cpp-netlib Google group and introduce yourself. Let everyone on the list know what you want to learn and what part of the library or project you want to be involved in. We're currently looking for people who are willing to:

Test the library on the major development platforms. Right now we're only concentrating on Linux and Mac with GCC and Clang, and we're looking for people willing to test on Windows and/or other platforms.

Update the look and layout of the website. This doesn't require much C++ knowledge but should be proficient with HTML, CSS, and has better design aesthetics than I do.

Read/update the documentation. Feedback on the documentation and what's lacking are very important.

Implement new features and/or refactor the internals of the library. This part will be involved and is where I would be doing most of the mentoring and hand-holding as much as necessary.

Thanks for reading so far down and I do hope you'll help spread the word about the new release.

Cheers from down under!

What You've Missed With C++: 2011 So Far

noreply@blogger.com (Dean Michael Berris) — Tue, 05 Jul 2011 13:18:00 +0000

The past few months have been a jam-packed with a few developments in terms of C++. Not only did C++0x get the almost unanimous nod of the industry to be good enough to be called C++11 but there's also a few stories that have come out that need some attention too. There was the Google findings which show that C++ is by far the best performing programming language (if you have enough programmers that know the language well enough to wield it proficiently) and C++ AMP which aims to bring massively parallel C++ programming closer to the mainstream by Microsoft. Also, last May BoostCon 2011 was held with a pretty good lineup of talks that Boost users would be most appreciative of. It seems like 2011 is the year that C++ dusts off its boots and starts marching on again to the top of the programming language hill. Or is it?

For a programming language that started in the 80's and is still being used by and large successfully by the tech companies that matter (I'm talking Microsoft, Google, Apple, Adobe, Facebook) you'd think C++ would enjoy a revered spot in the programming language hierarchy. Unfortunately if you look around and ask programmers what they think about C++ more or less you're going to find people who will have a number of bad things to say about it. Why does a programming language so hated be so effective and still be in use in industry?

First of all I think it all comes from a fundamental fact that good programmers develop good solutions. If you put a good tool (in this case a programming language) in the hands of a good craftsman (in this case a programmer) then the task you're set to accomplish would be accomplished with good enough results. You can still get a good outcome if you have bad tools and a great craftsman or if you have great tools and a bad craftsman. But the best results come from good tools in the hands of great craftsmen. Cathedrals, airplanes, ships, bridges, and all the infrastructure that last are built by craftsmen that had access to the best tools they had at the time they were built. I would argue that the software that stands the test of time (those running in Google or Facebook's servers, Microsoft's software, Adobe's software, Apple's products) are those written by great craftsmen regardless with good tools.

Second reason I can think of is the flexibility of the programming language. There are not a lot of programming languages where you can use it to write device drivers, operating systems, web application systems, realtime monitoring systems, embedded software, desktop applications, imaging applications, and even mobile applications by basing on just a single standard. There's a good reason why the flexible "general purpose" programming languages enjoy the most usage among the craftsmen that know their craft. It just so happens that C++ has enough flexibility to allow pretty much anything to be done with it with features that support the development of easily scalable software -- and I mean scale as in the capacity to grow the software as far as complexity and design is concerned. With a limited programming language that wasn't focused on "usability" in its current version (C++03) it's managed to be one of the most effective tools used by a lot of programmers in industry. Now with most of the shortcomings addressed in the upcoming version of the language, the flexible programming language just got a lot more flexible and usable.

The last thing that I think makes C++ a very compelling programming language to use today and going forward is: for any interesting problem to be solved at scale, you're going to need a programming language that gives you performance, flexibility, and scalability without sacrificing any of these three. I have yet to come across a programming language that comes near the performance profile of C++, supports as many programming paradigms that C++ does, and allows software construction and design to scale favorably as much as C++ does. There are some interesting programming languages that have come out recently but I have still yet to see one that actually can seriously replace C++ as the programming language for "serious software".

The reason C++ is still significant today is because the problems worth solving require as much of the machine as possible while being efficient in doing so. Sure you can use any programming language to write a web application, but if you're trying to handle billions of queries per minute and would want to get the most out of the machines you have at your disposal you're going to use one that allows you to scale your development effort and your application without sacrificing performance. The problems that are worth solving now require massive amounts of computing power and it's actually an advantage to have your application as efficient as you can make it. This is where C++ as a programming language shines.

With C++ in the hands of experienced and good craftsmen, solutions to problems worth solving are efficient, scalable, and maintainable. With a new version coming up I'd say this decade is the one where C++ is more relevant than ever.

cpp-netlib 0.9.0 Released!

noreply@blogger.com (Dean Michael Berris) — Mon, 18 Apr 2011 13:19:00 +0000

After the one-week beta period for the 0.9.0 release, I'm happy to announce that the official 0.9.0 release is now available for general availability at the Github project downloads page. This 0.9.0 release is the first version of the library to be submitted for review to the Boost C++ Library project. Most changes that happened from the beta to the official release involve adding debugging information to critical parts of the server and client implementations as well as documentation and Boost.Build system updates.

This release series will be supported until April 2012, which means all new development for the 1.0 line will happen on a different branch and will continue to contain new implementations and interface-breaking changes over the 0.9.x line. Currently the roadmap for the library is still in flux, but if you're interested in putting in feature requests you can file them through the Github project issues page. Soon enough a wiki page containing the roadmap for the 1.0 release with updated milestones will be released for public availability.

Help Wanted

At this time I would also like to announce that I am currently looking for help in various aspects of the project namely:

Website

Documentation

Testing

Development

This is mostly due to the fact that I've just recently started work at Google Australia, and may not be able to work on the library on a full-time basis. I'll transition more into a maintainer role and will still be doing most of the development on high-impact items.

New Protocols

The 1.0 line will get the HTTP implementation into a state of stability, which should allow other protocols to enter the library. Those that are highest on the list of priority protocols are:

SMTP

XMPP

The focus for 1.0 will be to implement other protocols so that the generic parts of the implementation can be lifted. Also, better installation and integration support for users that might not like the header-only nature of the library along with better customization points for tweaking implementation details would have to be made available as well.

The 1.0 line will be dealing with making the library more flexible in the facilities it uses to achieve the designs it uses. There are a number of other performance and usability points which will have to be addressed to make cpp-netlib adoption in more projects easier.

The Road Ahead

Along with a stability push, the goal for the cpp-netlib project is to attempt to build a community which mimics the Boost C++ Library process, with a more open development/contribution model. The C++ Network Library Project will be started officially with moves to make collaboration and further development of the code a more inviting and more open process. Once these preparations are in place, I'll be announcing what the plan for the foreseeable future of the library and the project will be like.

At the moment the library is looking to adopt other projects that would be useful in the context of a modern C++ library that deals with network-related programming. In the end the goal is to ultimately allow for developing both server and client services that talk in different (and maybe custom) protocols easily with the C++ Network Library. Along with this is an attempt to get the library into Boost as well as maybe part of the Standard C++ Libraries.

It's good times ahead -- I hope to be able to deliver more good news with cpp-netlib development from here down under.

Memcache++ 0.12.2 Out, Last Release by Me

noreply@blogger.com (Dean Michael Berris) — Mon, 11 Apr 2011 06:26:00 +0000

I have just finished tagging and uploading the 0.12.2 release of memcache++ which is a maintenance release that includes one diff away from 0.12.1 -- you can download the library here. This release is the final release that I will be managing and starting now I am on the lookout for others who would be interested in continuing the development of the library. I will still host the project on Github and I will still be available to answer questions but I will no longer be actively developing and adding new features to the library. If you or someone you know might be interested in taking over the project please let me know and join the user group as well. I had fun developing the library when I was still at Friendster and even though it's a little hard to acknowledge that a project is being retired, but my focus now has been put elsewhere -- and more recently the C++ Network Library is really what I'd like to focus on. Thank you to everyone who's checked the code out and those that are using it in production even up to this day.

cpp-netlib 0.9-beta Out Now!

noreply@blogger.com (Dean Michael Berris) — Sat, 09 Apr 2011 09:42:00 +0000

After months and months of delays, finally the 0.9-beta version of The C++ Network Library project is out now. The beta period will take a full week before an official release is made available for general availability. The 0.9 release will be the version that gets submitted for review to the Boost C++ Library project. Thanks to the wonderful users and contributors to the cpp-netlib project for getting the project up to this point in its development. I shall post another update when the beta period is over and when the 0.9 release is made available for general availability. In the meantime, you can download the 0.9-beta from the Github project downloads page.

What's New in 0.9:

IMPORTANT BREAKING CHANGE: By default all compile-time heavy parser implementations are now compiled to external static libraries. In order to use cpp-netlib in header-only mode, users must define the preprocessor macroBOOST_NETWORK_NO_LIB before including any cpp-netlib header. This breaks code that relied on the version 0.8.x line where the library is strictly header-only.

More consistent message API for client and server messages (request and response objects).

Refactoring of internal implementations to allow better separation of concerns and more manageable coding/documentation.

Client and Server constructors that support Boost.Parameter named parameters.

Client and Server constructors now take in an optional reference to a Boost.Asioio_service to use internally.

Documentation updates to reflect new APIs.

Feedback would be most appreciated through the C++ Network Library Developers Mailing List or through the Github cpp-netlib Issues Page.

C++0x will be C++ 2011

noreply@blogger.com (Dean Michael Berris) — Mon, 28 Mar 2011 12:47:00 +0000

As you might already have heard from other places and read from other sources, the ISO C++ committee has now finally voted to publish the current Final Draft International Standard (FDIS) for the language referred to affectionately as C++0x. This means that the FDIS will pretty much be the version of the C++ standard that the international representatives will vote upon for ratification in the upcoming votes happening in August this year. Hopefully this means that the world can start writing code in the new version of the C++ language that is already being supported by numerous compilers. You can find Herb Sutter's post here which is pretty much the de facto post which explains what has happened in the past committee meeting in Madrid. So what does this mean for ye old C++ programmers?

In the interim, the reality of the matter is not much will change with C++. C++03 code will still compile with C++0x compilers, although there may be some subtle deviations in some corner cases of the language. I'll get into more detail about what these things are at a later stage when I get my hands on a copy of the FDIS, but it's safe to say that C++03 code (with the exception of those using deprecated libraries, like std::auto_ptr) will pretty much build in any C++0x compiler.

That is the good news. The better news is that the C++0x -- or should I say C++11 -- version of the language will introduce nice things like rvalue references, move semantics, lambda's as first-class elements of the language, automatic type deduction (with auto), and standardized "typeof" in decltype. There's also atomics, a defined memory model, and new keywords and functionality (final).

There isn't much bad news that comes with the new C++ standard, but it does spell bad news for the compiler vendors and open source C++ compiler projects. This latest version of the C++ language is pretty extensive and the specifications are thorough. It's not going to be easy to have language disputes resolved now that there are too many things to deal with -- exception safety guarantees, threading, the memory model, are just a few issues that have to be visited when code generation time is concerned. I think though we're going to see a race and a real competition among the compiler vendors as far as quality of the implementation and standards compliance is concerned. In the end I think this is also a good thing.

That said though it should be a good time right about now to be a C++ developer and/or someone looking to learn C++ as a programming language for the future. Especially now that machines will be much more powerful and the parallel/concurrent computing practices will infiltrate not only the server-based applications but also the handhelds, C++11 is poised to be the language that delivers on the promise of mainstream concurrent/parallel programming.

Here's to the decade ahead that will be all about parallel/concurrent programming with C++.

PS. on a side-note, I now have a spiffy new Macbook Pro 15" with the quad Intel i7 2.2Ghz -- now I can write more and do more programming, so hopefully I can write more updates as the days go by and I get more familiar with this machine.

Broken C++: Old Compilers

noreply@blogger.com (Dean Michael Berris) — Fri, 25 Feb 2011 09:04:00 +0000

Up until recently I've only had to deal with standard-compliant C++. I've pretty much enjoyed writing C++ code that abides by an accepted and well-defined standard. This means I have been for the most part using most of the standard features of the Standard Template Library. That changed most recently when I had to deal older versions of compilers which did not support standard C++, and thus I felt the pain that most others have had to deal with a while back. This post explores the issues as to why C++ has been given a bad name by mostly spineless compiler vendors with deep hooks into various parts of industry.

Do not get me wrong, there are a lot of good compilers out there -- much more than bad compiler vendors. C++ is a good programming language if you abide by the standards -- but if you're using tools that don't support the standard 100% then you will get bitten and you will see the ugly side effect of history. Here I lay out some reasons why the baggage of history is sometimes too much of a burden for progress.

History of Vendors

Some compiler vendors for C++ have been around since the beginning -- around the 80's when C++ has been unveiled and released. However the international standard for C++ didn't get ratified until 1998. Some compiler vendors built up empires around server systems where customers were already spending tons of money for the stability and support offered by these companies. You'd get huge contracts selling machines and tools that allowed developers to write applications on these platforms that were guaranteed to not change from under them while support for these platforms were active -- usually the bigger firms wanted support contracts that span multiple years, renewable at the end, and usually cost a metric ton of money. It's all good business for the server vendors (and particularly, UNIX server vendors) who made money by making sure that the systems were stable and that any changes to the systems would be isolated into newer versions of the software and are not backward-compatible to older versions (mainly because they'd like to sell more licenses to the new ones, and upgrades are a big business too).

There are industries that have built around this ecosystem of server vendors who relied on the intricacies of the mix of hardware, software, and support. The business dollars were mostly poured into the hardware and support parts, and the software was mainly something that had to be maintained. If the software didn't have to change, then that was more manageable for these server companies because, well, they can just keep selling hardware and support and it would all be good. Until the software guys caught wind of the strategy and figured out that there's big business to be made in selling upgrades to software -- and this is when the hell starts.

Backward Compatibility and Interface Stability

The biggest headache for software releases is that of backward compatibility. Hardware manufacturers never had to deal with this because, well, they can just ignore the old hardware that was already there and they never had to make patches to stuff that's already out there. There are two platforms that had issues with backward compatibility and huge deployment bases: Microsoft and Sun Microsystems. On the server-side of things, Sun particularly has a big problem with breaking backward compatibility -- mostly because they are deeply entrenched in industries where change was frowned upon: the telecom industry, and the financial industry. Almost every telco in the world has a sprawling Sun server farm and 10-year-old installations of Solaris and Sun OS -- it would have been suicide if Sun forced them to migrate their C++ applications that built with the (broken) Sun-provided "standard" library, which they fought very hard to have to change the Application Binary Interface of. Other vendors in the same situation (IBM with the AIX server, DEC with Alpha, etc.) had to deal with the same issue: how do they upgrade their systems to support the newly ratified standard in 1998 of C++, and again in 2003.

On the flip side of things, Microsoft had a different strategy: they made backward-compatible changes a minimum priority, but instead they maintained multiple versions of the compiler and accompanying infrastructure. There were different platform SDK's (for Win XP, Windows Server 2003, 2005, 2007, ...) and then different compiler releases (Visual C++ 6, 7, 7.1, 8, 9, ...). This allowed developers to stick with the compilers they were comfortable with and not have to worry about later releases until they wanted to upgrade. Because these were separate products and developers were involved in the determination of the features and bugs that were released in every new version of the platform SDK and the compilers, you'll still see Windows Server 2000 installations using the Visual C++ 6 compiler and they would still be running 11 years later.

The big problem with the UNIX world is this insistence of maintaining ABI backward compatibility -- mostly because usually the standard library would be "frozen" interface wise, while the underlying implementation would change to accommodate bug fixes while maintaining that it looked the same from the outside. On Windows you rarely had to worry about this because you just had to say which version of the API you were dealing with -- see DirectX 6,7,8,... -- while the standard C runtime would even be versioned -- vc6, vc7, vc8, etc. -- so you could have multiple "standard" libraries installed on the system. Now mixing versions in the same application would be a problem, but that was beyond the compiler/system vendor's concern.

You see this with the GNU Compiler Collection where the 3.x line would not have binary compatibility with the 4.x line, as well as the Sun compiler where Studio 11 compiler outputs are not binary-compatible with Studio 12 compiler outputs. Sun Studio 11 is notoriously still standards non-compliant because, well, they chose to keep ABI compatibility with older releases of the standard library that comes with the compiler. Because it would be suicide for them to force customers who were stuck with an older version of the Sun standard C++ library to upgrade to a newer version, well, they got killed in other ways.

Another good model for dealing with ABI compatibility is what the Mac guys have in the universal binary format. I think this is a good way of going about it, and if LLVM gets supported on more platforms then we might have a better way of distributing applications in LLVM bytecode instead of native platform executables. That's another article for another time though.

Multiple Implementations

One of the biggest problems that C++ faced (and continues to face) is that there are a lot of different implementations of the same language. Even with a C++ standard, different vendors have different interpretations of the standard, and have their own extensions that encourage customers to write non-standard C++. As opposed to other languages like Python, Ruby, or Java, C++ has no "de facto" implementation that everybody can base other implementations against. Somehow this is the same problem that Perl 6 is trying to address, and I think that Lisp has suffered from the same problem that's caused it to stagnate in the face of other more dynamic languages. There are some merits to having multiple implementations, but having a reference implementation is also a good thing.

Because I have been developing code mostly on Linux with an older compiler (GCC 3.4) and having to deal with making sure the code also builds with the Sun Studio 11 compiler and libstdC library that Sun provides, having to deal with just these two different implementations of C++ is really painful. This greatly constrains what kind of C++ I can write and what features of the standard library I can rely on. This I think is one of the reasons why some cross-platform projects constrain the features of C++ they allow in the code. Having dealt with it first-hand has given me an eye-opening look at how bad some of the developers stuck with older versions of the compilers and non-standard compilers.

Bad Policies and Bad Decisions

History written by those who succeed and unfortunately for the case of C++ the history is written by the compiler and platform vendors. There were many chances for these vendors to rectify the situation, by improving the adoption of newer and standard C++ features for a long time already. Sun, while it was still a company on its own had a decision to make to overhaul their compilers and support the C++03 standard since the C++03 standard came out. They should have followed Microsoft's lead when they made Visual Studio 7.1 start supporting more C++03 features. Developers on the Sun platform deserved more than having a notoriously non-standard C++ implementation on which to build their applications upon. They should have been a lot more aggressive in pushing the standards as they have a lot to gain in having more applications build and run on their platforms.

What standards allow you to do is develop solutions that you have a reasonable level of confidence will apply to a broader base. Unfortunately because Sun has been late to the standards C++ party, applications like Google Chrome which build fine in standard C++03 mode in later versions of GCC won't build on the Sun Studio compilers and not run on the Sun workstations. Things like KDE which would be a good CDE replacement won't build on Sun compilers (but will build with GCC). On Sun, when library vendors still develop or release against the broken standard library, application developers are stuck with a broken implementation that does not help the community.

A Plea

Because C++ is a programming language that is crucial to the implementation of various system services and applications where performance matters, compiler vendors, please give us better tools to be able to develop the next great application with 100% of the features available to the programming language. Please be the drivers of innovation instead of the ones holding progress back. Please be more aggressive in pushing for change and making up-to-date implementations of the language for the benefit of all the C++ developers in the world.

If you're a paying customer to these compiler vendors, please write them and tell them that you deserve more because everybody knows that standards are good and standard-based implementations allow you to do standards-based solutions. If you're a vendor yourself of applications that still work with older platforms of compilers please be the drivers to push your customers towards upgrading their systems or take the step towards using a more sane implementation of compilers for the systems you will support.

For the sanity and sake of the generations of C++ developers who are yet to learn C++, let's stop using the non-standard compilers *today*. There is a saying that says:

There are always excuses if you don't want to do it; there are always reasons if you do want to do it.

-- Anonymous

So if you're like me and you want to further the use of standard C++, find the reasons to support it and upgrade *today*. The only reason these broken compilers are still being used is because we still keep using them.

Share your stories of your C++ nightmares with broken compilers and maybe we can get the vendors to finally stop supporting these broken compilers.

Software Engineering and C++

noreply@blogger.com (Dean Michael Berris) — Tue, 25 Jan 2011 17:07:00 +0000

There is a largely unspoken understanding between/among C++ developers that goes a little something like this: "any sufficiently complex problem can be solved by working with the appropriate abstractions". This is not a hard/fast rule but it's something that the senior C++ guys understand and internalize from an engineering perspective -- it's not about language features, the tools, or the techniques. Notice that I say engineering instead of development. What's largely missing from the body of C++ knowledge out there is the incorporation of engineering practices. This is understandable because most of the C++ books are written to shown how to use C++ language features and not how to engineer proper abstractions and algorithms. Is there hope yet for the people who miss the point when they say "C++ is over because language X will kill it"?

What makes great software teams and products work is largely not the technology used behind it. It's a combination of factors that determine the likelihood of success and of staying power:

Opportunity -- is the product positioned and developed to take advantage of an opportunity for profit/adoption? This is largely a function of timing and of positioning more than it is a matter of technology. Languages like Javascript and software like web browsers are largely a function of the opportunity to develop a new medium/market through which much of the innovation would happen. Had Javascript been developed and shipped outside of a browser it wouldn't have grown to be a popular language by any standard. There was an opportunity and those that execute well enough to grab that opportunity usually succeed.

Execution -- a poorly executed product is not going to have staying power. There are a ton of examples of this in many different spaces that I don't think it's worth mentioning any of them.

Iteration -- being able to add features or evolve the product in a consistent and predictable manner makes for great staying power in the market. Lots of examples here too so I'd rather not mention any.

Quality -- it's largely impossible to describe what quality is, but you know it when you see it. It's that general "feel" you get that this thing you're looking at or using is of a high quality. It's almost impossible to quantify quality which is why it's so important as a differentiator and as a factor for success of any endeavor not just software.

With these in mind you'd think that C++ would score really low on these points. Let me list down a few reasons why I think so:

C++ was developed largely at a time when C was already "sufficient" for all intents and purposes. Other programming languages already featured object-oriented programming and Lisp even had a good model for functional programming and imperative programming idioms to allow anybody to write great software with it. There was not an opportunity to create a new programming language that built on top of C because other programming languages were already available for doing things as far as programming was concerned.

C++ wasn't executed in a manner that you would call "well". It was meant to be bolted on to C as a means of "formalizing" the object-oriented paradigm and introducing it in "familiar" C terms. The first implementations were largely bloated and poorly done that it was thought of as an exercise in futility.

The first C++ version debuted in 1983. This was largely a research project and not really a commercial product. The time between the first version and the international standard was a good 15 years -- the first official ISO C++ Standard was published in 1998. The first addendum to the standard was in 2003 and the next version is supposed to come out this year. This is hardly iterating in a consistent manner -- unless you think 15 years is a good time frame for iteration.

C++ has drawn a lot of flak for being a "too complex" or "too rich" a programming language. It's far from a simple programming language. And even if there is an international standard out there on what the standard C++ language is, it's not until recently that the majority of the compilers available to programmers in general actually follow the standard to the letter. For this the quality of C++ implementations are actually suspect.

If you read the above points you'd think I've changed my mind and that I've turned my back on C++. It would seem like I'm throwing stones at an easy target here. However though it is important to point out that the reason why C++ hasn't died yet and why it would take a while for it to get killed -- if it will ever be killed which I am skeptical about -- has something to do with the external factors other than the technology. I'm talking about the software engineering practices that C++ above all the other programming languages enables: developing rich abstractions and implementing succinct algorithms that operate on these abstractions.

Precisely because C++ is a huge and largely permissive programming language, it was designed to enable people to model solutions in a richer manner than how C would have you constrained in doing. It assumed that programmers would know what they're doing instead of being dumb kids that need to be prevented from shooting their feet. It's a tool that takes time and effort to be able to wield effectively but when you get to that point of knowing how you can leverage the features to model your abstraction and enable your implementation, it's pretty powerful. When C programmers were fine with just structs and pointers, C++ gave programmers the ability to model dynamic polymorphism, generics, and more intuitive modeling with the use of namespaces, references, value semantics, and operator overloading. The difference between C and C++ is like having a toolbox for fixing cars and having a toolbox for fixing jets: for all intents and purposes, the toolbox that has more tools would allow you to do more things easily than the one that has less tools.

The point on execution is largely a good thing: the first implementation didn't need to be perfect, but it needed to be believable and an effective proof of concept. If someone came up with a programming language implementation that is already largely perfect to begin with (think Common Lisp, C, and more recently Python) then the room for improvement is a lot smaller than if a good idea was implemented with a proof of concept that needed improvement. This was how Linux started: a simple attempt at a PC-based UNIX-like operating system which was not more than a simple kernel when 0.1 was released. It's the same with DOS. Although there wasn't an obvious opportunity for greatness, there was enough opportunity to build traction and a following while the room to grow is still huge. Because the opportunity was really manufactured by having a poor initial attempt at an implementation of a good idea, it breeds involvement and bigger opportunities for a larger win in the long-term.

Having an internationally recognized standard is one thing but I think if C++ was rushed into standardization too early it could have nearly killed its potential. Consider the version of C++ that was debuted in 1983 -- imagine if there was an international standard before templates were introduced and properly implemented. That would have made C++ no better than Smalltalk in terms of OOP features, and Smalltalk would be arguably a better OOP implementation. Because the first standard was developed when there were enough people around the world investing in and building upon the (admittedly small) ecosystem of tools and platforms available at that time, there was greater incentive to pull the standardization effort off. And because the standard was built on industry feedback and actual user requirements, it better reflected the state of the art than had the standard been developed in isolation. The same is true for the C++03 addendum and the C++0x current final committee draft. There are some things not worth rushing and the standard having a good amount of time to be exercised and used in the wild is actually a good thing.

For a rich programming language C++ is getting a lot richer -- and that's a good thing. I'll go back to that analogy of a car maintenance toolbox and a fighter jet maintenance toolbox: if you needed to build a bigger/better/stronger fighter jet, I doubt it would be wise to build a new toolbox off the car maintenance toolbox. And this is where the evolution factor comes in handy: as in evolution as defined by Darwin, you need to have enough of a population of samples to be able to generate better offspring and have a better chance of surviving in the next set of generations. C++ is evolving at a time when multi-core programming is slowly (if not already) the norm to properly define the semantics of threading. Because of the cross-pollination of ideas with/from functional programming languages C++0x is getting inline function object definitions (lambda's/closures). When template metaprogramming and a richer type system is making itself a more integral part of the C++ programming experience, things like 'auto', 'rvalue references', move semantics, and goodies like decltype and template typedef aliases just makes C++ that much more powerful.

What's been largely untouched yet in this post is the fact that programming with C++ is more an engineering endeavor more than it is a programming endeavor. If you think about it a little, other programming languages makes gluing things together and calling it a day easier. Your solution might not be robust in the face of strict requirements on inputs and/or predictable in the face of exceptional instances (for the case of dynamic programming languages), nor be efficient in the use of available resources (in the case of garbage-collected and/or scripting languages, but for some cases that might just be what you need. People still write shell scripts and python scripts to do the odd little thing that would benefit from scripting -- but anytime they need to be controlling the resource usage or facing performance issues, they reach for the tools that enable them to model their solution cleanly and have pretty much the most powerful machine code they can generate.

I think what sets apart the senior C++ programmer from other programmers is that because C++ is a language that forces you to engineer your solution in an intuitive manner rather than just clobbering together a set of tools already available at your disposal. In some cases duct-tape solutions work until you need to start replacing the ducts themselves -- and for that matter duct tape is the worst thing for joining ducts together. Looking at most of the successful C++ based applications out there in the wild (and even hidden behind monster enterprises) you will find that they succeed mostly because:

They take advantage of an opportunity that is either obvious or one that is being manufactured by them.

They are executed as engineering projects that take time to think about the abstractions and algorithms dealing with these abstractions.

They are implemented to be iterated upon and maximizes the investments as far as time/resources are concerned in the building of that piece of software.

The quality of the implementation is held in high regard not only in terms of internal quality but also as far as external quality.

It takes a certain mix of pragmatism and discipline to be able to wield C++ in an effective manner. But when you get a group of these proficient C++ developers together that hold the engineering practices in high regard you almost always just get great software in return.

It's not C++ that makes the software great -- it's the great people who make C++ software work.

Software craftmanship matters.

Hiatus, Coming Back!

noreply@blogger.com (Dean Michael Berris) — Thu, 13 Jan 2011 19:28:00 +0000

If you follow the blog you will know that I typically try my best to write something meaningful either at the end or at the start of a new year. If you haven't been following the blog that long I think you will find that it's typical for blogs to do a year-end or year-to-come post. Unfortunately, I've been on a pretty extended hiatus no thanks to travel before and during the ending holidays of 2010. I've been meaning to write something substantial ever since I came back to the home office but there's been quite a lot of things happening that I haven't been able to write something I would be happy publishing... until now. :)

Just to be up-front about it this post is not about to be a 2010 review nor a 2011 preview. Consider this an update on what I've been up to lately and what you might expect from me. Hopefully though I can keep this one short as I'm typing this up while waiting for a build to finish -- so I only have a few minutes to do it! :D

So this is how this post is going to be structured like: I have three headings, one for "work", one for "open source", and one for "C++". Each one will be self-contained and could very well be posts of their own, but I rather put them together in the context of being an update more than anything else. So without further delay, here are some quick updates:

Work

I've started doing full-time consulting/development work with a company in the UK building a product that's meant to be used in pretty demanding environments. This is (finally) a project that allows me to work on C++ in a considerably large application -- complete with internally developed libraries and idioms. It's a good challenge to jump into a large application and starting to contribute -- I like hitting the ground running and learning as I go. This is actually the first time I'm actually working with a non-US or non-Philippine based client and I'm just starting to get the hang of the English. ;)

I'm not sure if I mentioned it last year, but I did go to Australia for a quick visit to interview for a company that shall remain nameless at this time. I don't want to give hints as to jinx my chances although the recruitment process is pretty strenuous -- that much I can say. I was just really happy making it to the final stages of the process and getting an offer from that company would be the gravy to my first taste of Sydney.

Just recently though, I also got invited to go to Norway to interview for another company -- which shall also remain unnamed at this time -- in a really interesting role. I look forward to going there around February and hopefully that trip becomes fruitful too. I'll get more details of my travel plans when I have more clarity on that. As for now I'm just going to start the visa application process which hopefully doesn't take too long. :)

There are other things going on in the professional front that I want to share, but I'll do that at a later time. So far 2011 looks to be a very promising year as far as work is concerned for me. Hopefully I'll be able to write more about C++ in the work related to what I do wherever I end up being this year -- of course, subject to social media policies and what not. ;)

Open Source

I did write a couple of posts already on my other "personal" blog about open source development, open source and the Philippines, and why Linux is important. I linked to those posts for a number of reasons:

This year I hope to do a lot more open source work. It won't necessarily be programming work though as I find that I really like the community aspect of the open source projects as well. Depending on where I end up in the short-term, I look to continue to keep contributing to the open source projects that I feel indebted and attached to (Boost, Ubuntu maybe, and definitely cpp-netlib).

I believe this decade is going to be about how the open source model moves into the mainstream. It's not just going to be software but it will also get into other facets of society. Open source won't just be a software thing anymore and I think 2011 is the first year where it will start happening. If 2001-2010 was about the long tail, 2011-2020 will be about the open source effect on society.

The economies and countries that will benefit most from open source will not be the US, but rather I think it will be BRIC (Brazil, Russia, India, and China) along with other developing economies around the Asian region. Because of the attractiveness of building startups now along with increased activity in the open source world, open source won't just be a political issue but also a democratizing factor in the next big wave of companies and projects in other parts of the world.

These are the reasons why I personally want to be involved in the open source communities that matter to me, because I want to be able to contribute in these three things I mention. Even if I just inspire readers of the blog to start contributing to open source projects no matter how little -- just posting issues and helping out on mailing lists or thanking the people working on the projects goes a really long way -- I think that's something for me to be thankful for.

Hopefully I can start getting into more meet-ups locally and in other parts of the world centered around open source development. Maybe I should start considering saving up for OSCon or another BoostCon. ;)

C++

2010 wasn't too big a C++ year for me especially in the early part of it. I got distracted with a startup, doing python development, and systems architecture consulting. This year is going to be a C++ year for me no matter which company (or prestigious consulting group) I end up joining full-time. C++ has been good to me and I plan to be good to C++ in return.

2011 is also the year when C++0x seems to be stable enough to be the year when it finally happens. I definitely look forward to learning more about the features that are really going to make it to the final version of the standard -- and I promise to post something about it more often than I used to. Really, C++0x is absolutely huge and there's something in there for everyone. Definitely a topic that will merit a few thousand words worth of blog posts from me this year alone! :D

I might not be able to keep up the weekly round-up, but I do plan to get contributions from readers of the blog as well. This blog is hosted on WordPress.com and I just might accept contributor posts if you ask nicely -- and if you have something really cool to share. I'll try guesting some big names or maybe doing interviews as well. If it's not me, maybe others who would like to contribute their interviews of C++ personalities to C++ Soup. I just might consider making C++ Soup a more regular thing if it turns out I can make a profit out of it somehow but 2011 might not be that year for me yet. Loyal readers would be the first to know of course if there's any changes that's going to happen in the short and mid-term as far as C++ Soup goes.

Onward 2011!

So that short post actually turned out to be a long one -- and I was even trying to cut it short! :D If you've read up to this part thank you very much -- your feedback and contributions in terms of comments, tweets, and just forwarding the link around would be most definitely appreciated.

PS. I have a personal goal of getting C++ Soup's RSS subscriber numbers up to 3000 -- if I get to that point, I just might make it a "real business". If you'd like to see C++ Soup expand and be a more regular content source for anything C++, please help me make it to 3000 subscribers! :D

2010 in review

noreply@blogger.com (Dean Michael Berris) — Sun, 02 Jan 2011 10:13:00 +0000

The stats helper monkeys at WordPress.com mulled over how this blog did in 2010, and here's a high level summary of its overall blog health:

The Blog-Health-o-Meter™ reads Wow.

Crunchy numbers

A helper monkey made this abstract painting, inspired by your stats.

The Louvre Museum has 8.5 million visitors per year. This blog was viewed about 110,000 times in 2010. If it were an exhibit at The Louvre Museum, it would take 5 days for that many people to see it.

In 2010, there were 33 new posts, growing the total archive of this blog to 181 posts. There were 5 pictures uploaded, taking up a total of 59kb.

The busiest day of the year was October 6th with 12 views. The most popular post that day was Clang > GCC.

Where did they come from?

The top referring sites in 2010 were reddit.com, news.ycombinator.com, Google Reader, stackoverflow.com, and dzone.com.

Some visitors came searching, mostly for c++ soup, c++ blog, boost asio, c++ lambda, and boost spirit.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

Clang > GCC October 2010
33 comments

C++ Hating November 2010
129 comments

Life After OOP November 2010
47 comments

Boost.Asio and Patterns December 2006
6 comments

Advantages of C over C++?! January 2010
12 comments

Agile C++ Development

noreply@blogger.com (Dean Michael Berris) — Tue, 14 Dec 2010 14:46:00 +0000

I haven't been writing much lately and that's mostly because I've been ramping up work with a new client. That said, I've always wanted to write about how to do Agile Software Development with C++, but I didn't have enough time or context to be able to do it. This current project has let me look at how us C++ developers who work on different size projects do our work in an Agile manner. Here are a few points I'd like to share as I lay down some insights into how I've been doing Agile C++ Development for the past few years.

Before I dive into the points, let me just give a quick backgrounder on what Agile Software Development is. Notice I didn't mention any specific methodology, and this is because being Agile is more a state of mind than a rigid set of practices -- if anybody tells you otherwise, they might be misunderstanding the Agile Manifesto.

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.

-- Agile Manifesto

So what are the hallmarks of Agile Software Development? For one thing, the premium on:

Iterative Development -- building software in a way that allows for quickly changing implementations over a period of time, and making sure that something gets delivered and iterated on.

Testing -- having tests to ensure that components of the system (and the system as a whole) works as specified.

Welcoming Changes -- instead of having lengthy planning, the solution is built to assume that anything at any given point later on can (and probably will) change.

In the world of dynamic languages and object-oriented systems, is there a place for Agile development with C++? Well, of course the answer is yes!

Because Agile Software Development is a process or a "way of building software", the tools you use aren't largely affected -- but the practices you follow might be affected. For example, instead of writing lengthy design documents, you'd rather write code and generate the design document out of the code; instead of writing specification documents, you actually write unit/integration/system-level/acceptance tests; instead of having planning sessions you have quick 5-minute meetings and more collaborative modes of development (pair-programming, for example). Now other languages have tons of tools that help with making these practices more in-grained in the development process -- things like one-click unit-test runners and continuous testing environments (that build and run the tests as soon as changes are being made to the code for immediate feedback).

I for one have been following different practices in the various Agile C++ projects I've been involved in.

Test Driven Development -- In any software I write now, I feel like fish out of water if I don't write the tests first to define what I require from the component I'm writing. Having tests to verify both your specifications and implementation are actually sufficient saves you from the guesswork you otherwise have to go through when developing software. Having a deterministic and honest -- even automated -- way of making sure you're understanding the specifications and implementing something acceptable is definitely the way to go in case you intend to make changes along the way.

Active Code Review -- In all the previous software projects I've been involved in, there is no sacred code and everything is subject to review (and improvement). This makes people care about the code they write and the design decisions they're making. It also ensures that everybody is made involved in the process of developing and learning from the code that's being developed.

Pair Programming -- I swear by the productivity improvement of pairing developers up to work on the same part of the system. The additional support and guidance you get from an additional set of eyes really affects the way I write code and the way I approach problems. The best part about pair programming is not the part where you're writing code, but the part when you're talking out the approach and the design decisions before writing -- or while writing -- any code.

Brutal Retrospective -- It's important to stay honest when coding to see where the faults and issues are and be able to say what these are and how they should be addressed. There's no point in sugar-coating the state of affairs especially when the success of a project is at stake. In other words, call it as you see it, and learn from the experience by making changes along the way.

Measure Progress and Productivity -- It's great to be able to measure the actual productivity of a team and be able to adjust as you go along in the project. There are a lot of ways in which progress and productivity are measured, but the details don't matter as much as the fact that measuring it matters. Do it however you like, but just make sure you're measuring progress and productivity.

Overall I've been in various roles in the Agile projects I've worked with in the past. Having a clear understanding of what your software needs to do and what the customer wants is key to the Agile process. The clarity yields happier customers and better software in the long run, and even improves developer productivity by being able to focus and iterate on the solution rather than trying to design it right at the start.

So have you been part of an Agile Software Development project lately with C++? How did you find it? Did I miss anything obvious above?

Weekly Roundup for C++, Dec. 5, 2010

noreply@blogger.com (Dean Michael Berris) — Thu, 09 Dec 2010 03:36:00 +0000

It took a while, but I now have enough time to sit down and concentrate on writing this article. It's been a busy week for me so far, ramping up work with a new client and getting things done in real life, and today is the first day I've been able to concentrate on writing this blog post. I hope you find the past week's interesting, bizarre, controversial, and link love posts worth looking into. I'm working to make this a more regular thing so hopefully you'll bear with me as I make some transitions as far as time/work management is concerned. Without much further fanfare, here are the interesting, bizarre, controversial, and some posts that I think deserve the link love for the week ending in Dec. 5, 2010.

The Interesting

Discovering Virtual Tables -- This isn't the title of the blog post, but the writer just discovered how virtual tables are generated and work with C++'s OOP system. This is not precisely how it's done, but basically the mechanics are the same, and is how the compiler does the binding of virtual member functions. Definitely interesting.

C-201x -- This is a good read about some reasons why C's next version isn't making much fanfare as much as C++'s next version is making.

Kinect Hack Makes User Invisible -- You may already have known that there were people who worked on getting a Kinect open source driver out, but this guy used that and another C++ framework to make himself look invisible from the video. Definitely interesting use of C++.

The post-Bavaria Meeting Mailings -- If you have enough time and wanted to get up to speed with what's been happening with C++0x, you will want to read the papers here. Unfortunately, it takes a while to get context, and you'll need to have been following the discussion to make sense of what's been going on. Interesting still.

The Bizarre

A Virtual Worlds Viewer -- I'm not a big fan of virtual worlds and I think it's a pretty bizarre concept in the first place, which is the only reason I am actually linking this sole entry in the Bizarre section. I'm not even entirely sure, but it looks like they wrote this viewer in C++. Go figure, I can't understand the article that much anyway. Weird.

The Controversial

Open Languages are Not Required -- This one is not about C++, but the argument being made by the writer of the article for Java is really bizarre. I'm not sure if Java developers are in denial but the writing on the wall is that Oracle will eventually kill Java, and that's the precise reason why people should move away from it. I just really want to see C++0x kick Java's behind when it comes to writing applications that are fast, efficient, and easier to reason about.

Giving Link Love

November 2010 C++ Committee Mailings -- This is a great summary by Anthony Williams (a member of the UK representation to the ISO committee for C++ Standardization) about what went on in the latest meeting about the on-going C++0x standardization effort. You'd want to read this if you want to get an update about C++0x.

Yet Another Web Framework for C++ -- This is a great write-up on the on-going open source project to provide an HTTP Server implementation built on top of the Qt framework. If you're working on an application that's using Qt and you wanted to embed an HTTP Server that only ever relied on Qt, this is something you will want to look into. Spreading the link love over to Marek.

I'll get back into a more regular writing schedule as soon as I figure out what time of the day it's good for me to write. Thanks for reading and I hope you enjoy this little recap!

Offline Weekend

noreply@blogger.com (Dean Michael Berris) — Mon, 06 Dec 2010 14:07:00 +0000

This is just a quick one to say that I spent an offline weekend, and would like to apologize to those waiting for the weekly roundup. I'll get to that tomorrow, having just dealt with the backlog and started work on a project with a new client. I'll get back into the writing mode and sharing the link love with the noteworthy things that happened with C++ around the web the past week soon enough. If you'd also like me to write about a couple other things that I haven't been touching on, now would be a good time to let me know what you'd like to read on C++ Soup. As a sort of "plug", here's something I wrote last week about what I really wanted to do with C++ Soup, why I'm doing it, and a hope that I can keep doing it. Until tomorrow!

cpp-netlib: Moving Away from Header-Only

noreply@blogger.com (Dean Michael Berris) — Fri, 03 Dec 2010 14:55:00 +0000

Today I pushed the initial step for cpp-netlib to move away from having a header-only design. Yes, the project has come to a point where a decision had to be made: lower compile-time requirements and wider acceptance, or high compile-time requirements and limited acceptance. Until earlier today, builds of the cpp-netlib test suite had already been taking orders of minutes to build all the tests. I'm not talking about two minutes, I'm talking about multiples of 10 minutes on a reasonably powered machine. The whole test suite is an exhaustive suite of tests meant to make sure that the implementation is doing what it's supposed to do. This limited my personal productivity as a developer, and the productivity of the users of the library developing other things on top of cpp-netlib. Something had to give and today I made the necessary changes to make that happen in 0.9.

Earlier on in the project, I really stuck to the "header-only" nature of the library. This was because I was convinced that the gains were really worth the costs. This is still largely true but there are a few compromises that had to be made. In this case the sacrifice was the added step of linking in an external static library for anything that is using cpp-netlib in their project.

The Change

First things first, not all the parts that I want to move out to static libraries have been moved out yet. The thing that requires the most compile time were the parsers -- specifically the URI parser. I made very little changes to make the transition of the parser from a header-only parser implementation into an external lib implementation. You can find the relevant change set and the commit message through this link.

Just doing this has cut down the build time of the whole test suite by 60%. It used to take 10 minutes, and now the whole test suite just builds in around 4 minutes. I'll publish exact numbers at a later time but generally this improvement in compile time would be welcomed by those using the HTTP clients. Because the URI parsing is a really important part of the library and because that in itself makes cpp-netlib unique, savings on the compile-times can really mean big gains for users of the library all-around. This addresses most of the concerns that users (both current and future) have of the library.

Balance

The library will at some point keep most of the implementation header-only while the heavier parts that take a lot of compile time would generally be moved out to externally built libraries. This means, almost all of the parts of the library that use Boost.Spirit will be moved out to external libraries because these are the parts that cost the most compile time (so far, that I've identified). This allows developers working on the library (there are a handful of us doing this) can keep some of the implementations header-only -- when the costs become unbearably high on the compilation side, moving them to an externally linked library is always an option.

I personally think this is a good direction to go and I'm thankful to those who have advised me to go this route. Some have been a little more vocal than others but generally I've listened and I think this really does improve the library's usability.

Future Direction

There are still some parts of the implementation that do cost quite a bit of compile time -- not all the Boost.Spirit based parsers have been moved out just yet, and the parsers that deal with HTTP headers may very well be a candidate for "externalizing" in the coming days. All the other goodies that come with library development -- tracking down bugs, improving the build system, creating distribution packages for various platforms, and the good stuff -- would require the help of more developers. The website needs some help too, and the team would really appreciate help in coming up with layouts, designs, and copy for the website.

Testers on various platforms and compilers would very much be welcome too, and I'm looking to start investing and setting up that company to ensure that cpp-netlib keeps getting developed as an open source project. That company will provide paid support licenses to those looking to get commercial-grade support for the library in their enterprise and or the applications they are building. I'm looking to set up a (paid) support forum for those wanting to get commercial-grade support for cpp-netlib -- hopefully I can get that out of the way and set up soon, but in the meantime if you're looking to get support, you can get in touch with me through this form. ;)

As for content of the blog, don't worry -- the cpp-netlib updates will pretty much stop. I'll be getting back to writing about generic programming in the coming week. I'll just be taking a break this weekend and I'll probably be writing a couple of lighter articles for light weekend reading. Look out for the weekly C++ roundup on Sunday (Philippine time).

cpp-netlib: Demolition Day

noreply@blogger.com (Dean Michael Berris) — Thu, 02 Dec 2010 16:19:00 +0000

Yesterday I wrote about some of the things I'm trying to do to reduce the compile times of cpp-netlib -- at least in the tests. Today was mostly demolition day for me. This means today I spent most of my time tearing big things (tests) down into smaller pieces. This also allows me to isolate problems with individual areas of the implementation so hopefully I can get the re-arranging of things going. Here's a quick report on what I've already done.

But before I get into that, I just want to say that I will be talking about some of the deep internals of cpp-netlib on this post. If you don't have the patience for that feel free to just read the bullet points. If you're interested in the internals then please feel free to indulge me in my discourse below.

1. One Test, One File

As I mentioned already yesterday I'm trying to reduce the compile times when building cpp-netlib and the fastest way I thought about doing that is by breaking down the huge test files into smaller ones so that I don't have to do too much template metaprogramming on a single test. Using the HTTP client still does pull in quite a bit of code, and I'm really thinking about simplifying the dispatch of the client implementations by introducing different templates and include headers for the asynchronous client. That's for another day; today I went through the redundancy in the HTTP 1.0 and 1.1 tests, and decided I should move those out into different tests.

The way it's being done at the moment is that the HTTP 1.0 tests instantiate the HTTP basic_client template with all the supported tags that come with cpp-netlib. Boost.Test supports writing template tests so that you can provide an MPL list/vector for it to iterate through at compile time to create the different tests. That's fine to a certain extent and if you have 10 tests in the same compilation unit and have 6 tags to instantiate with, then you can see that it quickly starts becoming a template instantiation party just for compiling a single test. Reducing the number of tests to 2 greatly reduces the compile time requirements in that respect.

I'd still like to bring that number down and I already have a strategy for that tomorrow, so hopefully I can bring the compile times down lower using that strategy -- and you can guess I'll write about it then too. ;)

2. Identify the Pain Points and Attack!

So one of the real pain points for me was the fact that I jump through hoops in the underlying implementation of the basic_client template just to insulate the user from making some of the decisions that I as an implementor would eventually give users a chance to choose through different means. The static tag dispatch for "choosing your parent" is clever, works very well, but unfortunately is immensely costly on the compilation side. I'd stick to the "choosing your parent" approach if the cost of doing so doesn't start to become prohibitive -- in the case of cpp-netlib, it worked earlier on, but now the choice of whether you want a synchronous or asynchronous client isn't entirely obvious and 90% of the time people will just use the default implementation.

So today I thought hard about offering alternative "lighter" means of letting users choose which implementation of the client they would want. After all, there is very little value to providing a single template type especially if I'm dealing with a generic interface. Starting tomorrow I will have a go at implementing different types for the different implementations and interfaces of HTTP clients. The next release will see different HTTP client types and a simpler extension mechanism that uses the tried and tested -- and scalable -- policy-based design prescribed by Andrei Alexandrescu in Modern C++ Design.

The tag dispatch mechanism used by cpp-netlib, although clever, is unfortunately too demanding on compilers. Until the compilers change and suddenly support template metafunctions natively, that mechanism will only be a good academic exercise at best.

3. Small Steps Sometimes Have Big Consequences

With my decision to provide a static library implementation, I'm now seeing that the way cpp-netlib is implemented in the moment doesn't make it easy to accomplish. This means the small step of providing that externally built library may have big consequences in the way cpp-netlib will look like in 0.9. The innards of cpp-netlib, even down to the implementation of the directives, wrappers, and modifiers for the messages will change -- hopefully for the better. This means that's a lot of work to be done, but it also means that the upside for improvement is definitely higher.

I'm hoping tomorrow that I can really hunker down and get down to business with minimal distractions so that I can focus on getting a solid few hours of work into cpp-netlib so that I can make the static lib option happen. Hopefully by the end of the day tomorrow, this, and the above two points I'm able to address.

4. Spirit is Cool, Beware the Spirit

Tomorrow, I'm going to move out the parser an implementation that is linked in externally (through a static lib) so that the parser doesn't get re-compiled every time the HTTP client include is brought into the project. I think this is the single biggest change I can make to lower the compile time cost of using cpp-netlib, and as much as I would like to hope that compilers will improve in terms of supporting template metaprogramming in C++, I'm going to have to face the music and see that the state of the art right now is not favorable to extreme template metaprogramming calisthenics.

I absolutely love Spirit, but unfortunately I can't keep imposing the cost of Spirit on users of cpp-netlib. Maybe when Clang or GCC get good enough to handle all the template metaprogramming tricks that Spirit/Proto/Fusion use without barfing or thinking really hard about it, then I'll switch back to just going header-only. For now though, I'm going to have to give up that fight and move the parser implementations to an external lib. This way I don't have to trim the number of rules so I keep the expressive and very well detailed and declarative implementation of the URI parser that's already in there.

What To Look Forward To

Tomorrow I'm looking at stabilizing the implementation and lowering the compile times effectively, so that I can start adding the features that I really want to be able to ship in 0.9. Hopefully the universe cooperates and I actually get more things done tomorrow than what I was able to accomplish today.

Until then, I'm getting some rest.

cpp-netlib: Reducing Compilation Times

noreply@blogger.com (Dean Michael Berris) — Wed, 01 Dec 2010 16:35:00 +0000

Earlier today I spent some time to get paperwork done for my application to get an Australian Visa. I'll write more about the reason for that at a later time, but right now I want to write about a glaring issue (the white elephant in the room I like to call it) with cpp-netlib which is really the compile time requirements. I've already shared some of my thoughts about the issue in the cpp-netlib developer mailing list and I'd like to share some of my thoughts about the compile-time issue not only with cpp-netlib but with C++ programming overall. I'll touch a bit about GP and the costs of doing it here just to keep the theme of my writing about GP the past few days.

Before I go into the issue at hand, I'd like to give a little backgrounder on the motivations for starting the cpp-netlib project and a little about the history of the project. I promise this should be really short.

The motivation really started with a small project that involved writing an HTTP client application in C++ that was supposed to be a straight-forward endeavor. I wanted to write an application that can make an HTTP call to a web service. How hard could it be right? Well apparently it turns out, it's really hard to do that even with the abundance of libraries already out there. I can name a few that I tried but suffice it to say there wasn't a generic HTTP client library out there.

I looked around some more and found C libraries that did it, but they were ugly in terms of interface and most of the code I was going to write was really meant to adapt my solution to the library, rather than having the library just be really simple and easy to use. None of the frameworks allowed me to do the same thing in a simple manner.

Frustrated, I did what any self-respecting software engineer would do: quit. Well, no, not quite quit, quit looking outside and just start writing one. I took this on as a personal mission to develop a generic HTTP client library inspired by how the STL was developed. I also made it a point to make it part of the Boost C++ library. So that fateful day in 2007, on my own time I wrote the first simple HTTP client that did what it was supposed to do in an easy manner with no external linkage requirements -- and it felt good. That for me was the point of no return and fast-forward to 2010, after putting very little time over the years to getting the library to a point that would make it suitable for a wide array of situations, here I am adding features and keeping the library source-compatible to previous releases -- there have been 8 releases that started from 0.1 which is now in 0.8, and 0.9 is under active development too.

Because of the additional baggage that has come on to the library, the amount of template metaprogramming going on has put a big toll on using the library. Here are some of the areas of improvement where I intend to focus on as the development of 0.9 progresses.

1. Isolated Testing

In my daily development cycle, I follow the test driven development model. This means I write a test first for some feature I want to implement then do what's necessary to make that test pass. I do this because I work from the outside in -- looking at it from a user's perspective first, then worrying about the implementation later. This allows me to keep my focus on making the interface simple, intuitive, and powerful.

However, one bad habit I've taken from my earlier Java TDD programming was having a huge test suite in a single file. I hadn't been refactoring test code out into smaller tests, which I should have learned back then, but I'm applying now. Because cpp-netlib relies heavily on template metaprogramming, and since cpp-netlib is a header-only library, changing a small part of it causes a re-build of a huge test suite. To mitigate the costs of the builds, I'm separating the tests into smaller units, so that building the smaller tests would reduce the amount of resource required during builds of tests overall.

So far this technique has been helping me identify which usage patterns yield the worst performance results and it looks like this approach is going to have a noticeable impact in my development cycle.

2. Reducing Spirit Qi Rule Assignments

The library relies heavily on HTTP RFC-compliant URI parsing -- thanks to Jeroen Habraken for the contribution -- but the number of rule assignments is causing the build times to really blow up. Many might not know that Spirit v2 is now using Boost.Proto to implement the domain specific language of Spirit's rendition of Extended-Backus-Naur Form (EBNF) for generating parsers. This means there's a full-on compiler/optimizer implemented using C++ template metaprogramming -- think about it as meta-compilation, because with Proto, your compiler is running a compiler while it's compiling. You can read more about Proto here.

That said, when a parser expression is assigned to a rule in Spirit, it invokes Proto's evaluator that actually compiles the expression into an actual parser that gets run at some point. All this type programming makes sure that the code produced at runtime is optimized by the compiler by not hiding anything from it during compilation. This also means that for each rule assignment, you're essentially running a compiler. That's not good for compile times.

One of the things the team will be exploring is reducing the number of rules so that there's less invocations of Proto's evaluator. Hopefully that yields favorable results so I'm keeping my fingers crossed.

3. A Static Lib Option

One of the things I'm exploring is making a static lib option available to those who don't want to use cpp-netlib as a header-only library. This is a little more involved, which means exposing a non-templated face of cpp-netlib to users who just want to use the HTTP client for example. Pre-configured and pre-compiled HTTP client types and interfaces can be linked to in cases where an HTTP client is all that's required. For the HTTP server case, a simple service framework for doing dynamic registration and reconfiguration of handlers would be welcome as what I'm hearing from other users.

This means that cpp-netlib will still be header-only, but a non-header-only view of the library will be made available. This isn't a terrible amount of work and it might even be welcomed by those who are used to having static libraries linked into their applications (or auto-linked in Windows) and those who would like to be able to use cpp-netlib maybe in a DLL. Hopefully this experiment bears fruit and would make cpp-netlib more accessible to more people.

Future Talk

All these things above are a preview of things to come. In the next few days I might wind down the writing and move into lighter articles as I try to concentrate more on the development of cpp-netlib ramping up to a version 1.0 snapshot for submission to Boost -- hopefully to be reviewed early next year if I can get a review manager. :D

Comments, suggestions, and definitely encouragement would be most appreciated. Let me know what you think and I'll share them with the developer's mailing list if you don't beat me to it there.

You can download cpp-netlib's 0.8 release and previous releases here.

Generic Programming Barriers to Entry

noreply@blogger.com (Dean Michael Berris) — Tue, 30 Nov 2010 15:42:00 +0000

I thought about the subject a little more and tried to look into myself and what was stopping me from learning generic programming earlier in my C++ programming experience. One recent interview I had still sold C++ as an Object Oriented Programming language, which was surprising to me because it was only half-true. I knew better than trying to correct the interviewer, but that's a different thing. After a little introspection, here are a few reasons why I think GP has had a hard time getting into the mainstream.

This list is not meant to be exhaustive -- I have to keep it to a sane list however, so I'm listing down the top 5 reasons why I think GP is having a hard time becoming a dominant programming paradigm. Drum roll please...

5. Lack of Personality

Just like with any movement, you need a charismatic leader who's screaming and pushing for the practice to be adopted by a larger mass of the unwashed. Although Dr. Stepanov is the godfather of the movement, I don't think a genius needs to be the one leading the charge. Look at DHH -- although I have respect for him, I hate his guts and pretty much disagree with a lot of what he says about business and when it comes to Rails. But, he's an effective figure head to lead the Kool Aid selling of Rails. There's also Steve Jobs -- I don't buy the Apple hype because I think their machines are over-priced -- but man if there was anybody who can sell Kool Aid, Steve is the man.

GP needs a brash outspoken hippie or "down in the trenches" person to lead the charge. Someone who's willing to say that "OOP for everything is BULLSH*T". Someone who also has a lot of experience under his belt and has had successes with GP would be a good candidate. Of course that person shouldn't mind being in the spotlight and still be the figure head of an ideology -- new Kool Aid -- but still be effective in what he/she does. I'm not this person yet because I don't qualify -- nor do I want to be this person, I can only stand so much FUD being spread around me. ;)

4. Lack of Direction

Think of it as a lack of propaganda, organization, and vision. Although the "GP Monks" that I've alluded to at an earlier post -- I shall not name names -- have had it good, much like the Buddhist Monasteries that I know, it's pretty much "every man for himself". You don't see conferences, websites, books, or even Google Tech Talks about the subject. And those who understand seem to be content in just understanding -- there aren't a lot of fervor around it even though I believe it's the best thing since sliced bread. And I'm not just talking about GP in C++, because GP can be applied in a lot of other programming languages -- it's the paradigm that hasn't been getting much direction/promotion.

It's like "Buddha has left the building" in the Monasteries, and the Monks have no idea what to do next except to live by the code and just be content in their life. GP needs a champion or a league of champions -- like what happened to Agile Software Development. We need the leaders of the cause to come together, come up with some sort of manifesto, and sell it like Kool Aid for it to be anything but the best kept secret of successful library developers.

We need more libraries written to be generic so that everybody wins. I get so tired with libraries that are so rigid and cumbersome to adapt that they're pretty much useless except in the context in which they were developed. The world needs more generic libraries and GP needs to get a direction -- and that direction should be towards mainstream.

3. Lack of Information

Who in the world knows about GP except those who bother to read about the C++ Standard Template Library? (There's a great book about that too, STL Tutorial and Reference Guide: C++ Programming with the Standard Template Library (2nd Edition) (C++ in Depth Series)). And you just get a taste of GP with the Elements of Programming book (again written by the venerable Dr. Stepanov) -- unfortunately that book is a workbook for those wanting to sharpen their generic programming chops, assuming pretty much that they already know GP from the start.

If only everybody learned programming from the start and heard about this great thing called Generic Programming, then perhaps the OOP languages start looking like they were designed decades ago -- except that they really are -- and that there were programming languages way older than these (C++ and Common Lisp) which support GP already. I want people to start asking why the world ever tried to live with just OOP when GP can do what OOP provides *and still* provide more things that OOP can't. Then again people generally don't get that Common Lisp is powerful despite how many people are able to do with it.

2. Lack of Support

This is a business issue, but because GP didn't become a buzzword unlike OOP, nobody supports it in the industry. Every other programming language that comes out now either has to look like Java because that's what the college kids learn, or they come out like Ruby because that's what the snobby new-age web 2.0 hackers like. Very few people bother creating powerful programming languages like Common Lisp or C++ because they're scared of what they don't understand.

Like people who write novels for the masses, they try to appeal to the broadest audience possible and in the meantime losing the essence of being a great and powerful tool. Perl is a programming language that I have respect for, but because it has cultivated its own niche of programmers and kinds of programming, it's really hard to be hacker-level in Perl. The technology behind it (recently I listened to FLOSS Weekly about Rakudo Perl 6) is incredible and I really wish it becomes more mainstream and beat out the likes of Ruby -- I also wish it would be JIT'ted to Machine Code just because I like it that way (it may already happen, I don't know much about Perl 6 to be honest, just wishful thinking).

Even D might have great support for GP but then it's not being touted as one of the good things with the language. I recently looked at the language again (version 2) and it looks promising -- things like not requiring '->' when dealing with pointers is such a good idea, why doesn't C++ just adopt it and get rid of the C legacy for crying out loud. The good parts of D are really good, while the good parts of C++ are missing (like templates). I'll write about D at a later time and what I would like to see happen in C++ 2.0.

And the #1 reason I think for GP is...

1. Apparently It's Not Obvious

It's hard to argue about this one. I know when I "got it", it wasn't immediately obvious. GP is so much on the "meta" side of programming, that it's not easy to tell the normal programmer that "look, just deal with the abstraction, leave the details for later". Largely because it's apparently not how 99% of the people think. It's not obvious that there are parts of this implementation that can be lifted out and made generic -- sorry, even I sometimes need to squint a little and think hard about which parts of the solution I have in my face can be made generic, and I've been doing this the past 10 years! (of course that says more about me than the paradigm, but I digress).

There isn't a programming language yet out there where GP is obvious. Maybe in Common Lisp it might be more obvious, but even there the only abstraction you have that really matters is whether something is a list or an atom -- that's powerful abstraction, but not nearly high level enough which is why people unfortunately came up with CLOS to make it a little more manageable, and also because of that the OOP bug bites you because of the rigidity, coupling, etc. Other languages like Haskell have promise with the Type Classes (acting like Concepts) and its close lineage with Math -- maybe I should start writing more in Haskell and have the C++ I write take on a more Haskelly feel, but I'm afraid that's not nearly as natural as just doing GP in C++ for me.

One of the things I'll give to OOP is that you can teach anyone how to think in terms of objects because nearly every human language in the world addresses the issue of classification to some extent. Because OOP thrives on taxonomy and the idea of putting things under hierarchies (a Dog is a Mammal is an Animal is an ... Object?), getting people to think this way is almost natural. However, GP works in the domain of Math and any Mathematician will tell you that Math is much more beautiful than the real world -- and sometimes it's sad that way.

Somehow if GP is going to thrive, there has got to be a higher premium on thinking like a Mathematician, in terms not of hierarchy or taxonomy but rather dealing with abstractions -- concepts -- instead of types of "things". And that's the sad part, not enough people doing programming know their Math well enough to be able to understand and properly use GP in their daily programming lives.

Conclusion

There's a lot of work cut out for those (like me) who love GP and want to see it applied in more places. I think these barriers to entry to the mainstream can largely be broken down with enough determination and with a vocal enough group of followers. I don't claim to have that much of a pull, but I guess the Internet is a huge place and people just might hear the frustration of someone who's convinced "there's a better way" of programming and either agree or at least think about what this one guy is yapping about.

Help spread the word about GP even though you don't understand it fully or just get to use it through the STL in your daily life. You might not understand it, but those who understand might hear you and might take up the cause and tell the world about this beautiful little paradigm called Generic Programming. Maybe they'll think the world will be a better place with more Generic Libraries out there and so the may be inspired to either keep doing what they're doing, or convince more people to do the GP thing too.

Feedback most appreciated.