C++ Soup!

The C++ Network Library 0.10.0 is Released

noreply@blogger.com (Dean Berris) — Fri, 28 Jun 2013 10:29:00 +0000

Finally, a new release for the cpp-netlib project has been published -- and we're making a few changes to the project in the process! To get the packages, please head on to the official project website:

http://cpp-netlib.org/

This 0.10.0 release is an incremental release on top of 0.9.4 which brings a couple of important changes. The first one is a change in how options are set when constructing the server and client objects, shedding the dependency on Boost.Parameter. The other change is better handling of chunked transfer encoding on the client side.

There are a number of bugs that had been fixed so if you encountered some issues with 0.9.4, you might want to give 0.10.0 a shot.

The reason we upped the minor version number is because of an interface change. We're using a different pattern for setting options which is a little more extensible and less magical than how Boost.Parameter works. To find out what's changed, please see the online documentation for the project:

http://cpp-netlib.org/0.10.0/index.html

With this release the project will now follow a three-month release cycle. Any changes on top of 0.10.0 will be released at the end of September, 2013. As with the release of 0.9.4, we will be supporting the 0.10.x line for two years.

Another change is that the project will be moving forward to push changes to release a 1.0 version by the end of 2013. That release will be fully reliant on C++11 features and will minimize dependency on Boost -- preferring standard libraries where they make most sense.

If you'd like to be part of the effort please join us on the mailing list. As always development happens at the official repository using a GitHub workflow.

Feedback, bug reports, and as with most open source projects, contributions are most welcome!

Hiatus and Reflections

noreply@blogger.com (Dean Berris) — Fri, 03 May 2013 04:30:00 +0000

So I've left one hanging promise at least fall off my radar -- a manual vtable based implementation of the concept-based polymorphism implementation. As you all have known for a while now I've been working at Google Australia for two years now and part of the work involves some really interesting C++ development. I can't talk much about the things I've been doing in Google, but I can talk about some learnings I've picked up along the way as far as large-scale C++ development is concerned. If you're not interested in "meta" or "personal reflection" discussions, you can safely skip this post and you wouldn't have missed much.

Overview

In the past three months I've been fortunate to get my hands dirty with working on evolving a widely used piece of C++ code from one form to another. This piece of code is a core library and the amount of code depending on it is non-trivial (in the order of hundreds of thousands of libraries/tests) and my task was relatively trivial: mark a member function from a class deprecated, migrate all users to two alternatives, and finally remove the member function from the said class. This post is about the things I've learned in the process that I can share with the outside world.

Challenges

There's a number of challenges that I've faced in this effort. Without going into specifics (and the how I've made the changes) there are more pertinent human issues that I've had to deal with.

When you have thousands of engineers who have to learn to use your API, having a smaller surface area is generally better. Cutting down the API later on is going to be a problem -- it's much easier to add than to remove and if you can avoid supporting complex APIs then you should avoid it.
Social issues are much harder to solve than technical issues. When people are already used to doing things a certain way, change is harder to effect. In general getting people to agree that a change is a good idea is usually harder than making the actual change.
Teaching people is hard. Novices aren't as hard to teach as people who know enough to be dangerous -- I'm happy that teaching is one of my passions, and that I enjoy teaching. I like teaching and sharing makes the endeavour a little easier, but in general is still hard.
Change should not be for change's sake. Having a goal, a plan, and a good rationale that's clearly explained and widely understood makes large scale changes easier. If the costs are worth paying for to get to an end state that is measurably (or qualitatively at least) better than the current state, communicate and make it clear.

The larger theme for me in the past few months (and two years that has passed) is the realisation of how scale totally changes how you think about getting things done.

Scale

When you're responsible for a result that spans more than just one project, the amount of impact you're able to make is both significant and insignificant at the same time. There's a funny thing about large numbers: in relation to the whole, a big number may be a lot -- but many little things also add up. If the change you're making is an effective "no-op" (or a change to keep things the same) then no matter how many changes you make the effect in aggregate is insignificant. In comparison if you're improving the performance of a specific function that gets used everywhere, even if your change is small the effect is big.

Dealing with a large codebase (in the order of a hundred million lines of code) changes the scale conversation significantly. Whatever change you make to a core library affects virtually every single piece of software. Without thinking about the details -- build systems, applying the changes, running the tests, etc. -- what's important to note is how important being conservative is. The amount of planning involved in making sure that changes on core libraries being used will not result in wide breakages and performance regressions is non-trivial and should not be taken for granted. It's not a simple problem and the key to executing well is good planning as well as a healthy dose of patience.

At this scale, language features and local style become very distracting. In a codebase that large what counts more than intellectual purity and idiomatic expression is consistency. At Google, the style guide and a small group of arbiters become the de facto sources to refer to when the question of style comes up. This is for good reason -- everybody has to be able to read, understand, and maintain the code you write and the style you use should be consistent with everyone else. Surprisingly, when you're tasked with reading thousands of call-sites and surrounding code context, having a consistent style saves you tons of time and effort. Anything that keeps you from having to think about "magic" or "details" is a net win for productivity at scale.

I've heard arguments for and against the Google style guide (it is publicly available) and as someone who's had to read code that other people have written, I welcome having a style guide that is actually followed.

Local Optima

Having said all that, using scale as a single hammer to treat every style decision is not the best way going forward. Usually there's a tension between what's best for a specific situation -- in design and in implementation -- and what's best for the good of all. Outside of Google, I've encountered this discussion multiple times and it has always been a judgment call.

Because judgment calls, exemptions, special concessions specifically do not scale they are frowned upon when you reach a certain size. This makes the alternatives more dominant thereby making the cost of allowing something that had previously been outlawed (or non-existent) compounded over time. In this case scale becomes a problem.

What I have learned though is that when you get a bunch of really smart people having a conversation about merits and plans where everyone's interest is served, you get results. I'm proud to be associated with Google engineers who I believe are top notch in terms of skill and attitude, who can have a civil conversation about what the right thing to do is and how we can forge a way forward to progress. Being among smart people who are willing to listen and make a change for the better is one of the reasons I'm very thankful I work in Google.

Process

Part of the problem introduced by scale is process: too much of it and you grind productivity to a halt, too little of it and you have chaos leading to very little productivity. Either way is bad so finding the sweet spot becomes crucial and very important. I won't be sharing the process involved in making a change, but I will share the process of getting to the process. I would think it applies to many different engineering organisations where process forming is not a well-defined process.

I've condensed my learnings into a few bullet points:

Experiment. Do this with rigor though, not just "try something and see what sticks". Be smart about what you're going to try, how you're going to define what actually works, and how you're going to analyze the results of the experiment. Treat it like a research project where you have hypotheses and treatments and proper analysis of results.
Iterate. Once you've found one or two processes that seem to work, make incremental changes to improve productivity. It's crucial that you get your goals straight and your iteration incremental. Making changes doesn't mean throwing the whole process-building process out. Get feedback, tweak things, remove steps, add steps where necessary.
Measure. Every step of the way it's important that you're keeping track of metrics. Rely on data not on just qualitative observations. If you can make measurements based on actual actions and results then do that -- there's nothing better than being able to quantify your progress or efforts then making changes based on actual data.
Optimise. Once you have measurements from some iterations and enough observations and results, then you go back and make changes that will further improve your productivity. It doesn't matter whether you get to the optimal state as long as you're getting a decent amount of productivity.

If you're building processes any other way you may be missing out on some opportunities for order of magnitude improvements in productivity.

C++

So far all I've been saying has more to do about software engineering and not really much about C++. There's a few things I've noted and observed as I was reading and writing lots of code in the past two years. These are observations about C++ itself and in general, writing software with it at Google's scale. I won't be revealing anything official from Google, as these are my personal observations.

Avoiding chaos is easy, avoiding the chaos of order is hard. When you have multiple thousands of developers writing and changing code that have complex dependencies and sufficiently complex designs there is a premium on avoiding chaos. This is where design up-front or at least thinking about the design of the system early pays off. What becomes trouble is the chaos of order that happens when you have multiple things going on at the same time and multiple available components to use. It's a good problem to have but still something very hard to manage -- C++ as a programming language requires more discipline than other languages where dependencies can typically be broken up into proper modules. There are well-known ways of managing this but not all the engineers know how to do this. I'm positive there are changes in C++ coming down the pike (modules) and already available (inline namespaces) for managing change and dependencies.
Warts are features and workarounds are idioms. I understand that C++ has its warts and for whatever it's worth the more you know about them the more things you can do with them. The good thing about the workarounds are they can be taught as idioms -- and they seem to carry well to industrial practice. Teaching how to use C++ has given me a better appreciation of how important idioms and communicating the deficiencies are. I would argue this is a good thing about C++, that it's not burdened by elegance or by nice and shiny bells and whistles. What you get with using C++ is the raw performance that your tools can afford you and the scale of the systems you can build with it -- warts included.
Experts scale better with C++. When you have experts in the language and the standard writing and maintaining the core libraries and the tools made available to everyone else, you get David standing on the shoulders of Goliath. You get the users of the libraries to a state of productivity leveraging what the experts build. Having the experts available one email away is also an invaluable resource to the organisation. Because the experts can keep banging out the amazing libraries and have others learn from the library designs you immediately raise the level of understanding of the language. I'm thankful that in Google we have C++ experts on tap from whom I have already learned a lot from.
Pound per pound, C++ is the best programming language for systems programming. Especially when you're measuring things in the millisecond scale, every microsecond counts. There's yet to be another programming language that lets programmers control every aspect of the solution, enable the scalable design and implementation of sufficiently complex systems, and be as efficient as possible with resource usage. Being able to control exactly how many allocations you're going to make, avoid copies of data, exploit concurrency at the lowest possible level, and in general only pay for what you actually use is golden. It translates to effective resource utilisation, the avenues for optimisation are wider, and the return on effort more measurable.

It is no secret that Google's success is built on great systems programming, amazing infrastructure, and brilliant engineers. It so happens that the company has made big bets on C++ and it has paid off a lot.

Conclusion

In the end I've been learning a lot about large scale library and systems development. Making changes to something that's core to the system and being able to execute these changes has exposed me to very interesting learning opportunities. Being exposed to and working beside some amazing engineers has taught me a lot not just about how to technically get things done but also how to take the learning experience and build on it. Having a better insight into how large scale development works first hand is a very valuable experience -- something I'm very thankful for that I'm able to earn in Google.

Here's to looking forward to more insights and more learnings from inside the GOOG.

cpp-netlib.org is here!

noreply@blogger.com (Dean Berris) — Fri, 25 Jan 2013 04:22:00 +0000

This is a short post just to direct your attention to the new cpp-netlib project website! The project's new home is now at http://cpp-netlib.org/ (links to the old site should automatically redirect to this domain). We're looking at scaling the effort of the project and enforcing a few processes for consistency and for velocity. A few highlight to the website content:

A style guide. We are now getting to the point where we have a lot of code and more people sending in pull requests. In this light we've started a very minimalist style guide. We will discuss and update this as soon as we have more data to make decisions.
A development policy document. We've attempted to codify what we do and how we do it.
A proposal process. If you're considering getting involved or helping get your library developed or included in cpp-netlib, there's now a process for that.

We're also now ramping up our migration to use Google Test and getting ready to use Clang Format to use the Google style-guide formatting (just for the formatting) to make the code more consistent. I'm driving that process and getting the codebase into a more maintainable state. I've learned in these past few years that readable code makes my life easier, and contributors lives easier. Having said this we're not going to compromise on features of C++11 that we're going to use. I believe that now is the time to use C++11.

Personally I'm also driving the HTTP server re-write and hoping others will be interested in pitching into that effort (as well as the HTTP client re-write). Glyn has been driving the standardisation of the URI library and we're generally pleased with the feedback we've been receiving. My dream in the long term is that cpp-netlib can play a similar role that Boost has played for getting libraries widely used and tested focused on the network programming domain.

To that end I'm happy that Google has officially acknowledged this effort, and starting really soon most of the work I will be doing will be going through the Google fork of cpp-netlib.

Thanks for reading and I hope to be able to share more news about cpp-netlib soon!

Postscript: I will go back to the topic of value semantics and virtual tables without using the virtual keyword at a later date. I'm taking my time with that but I should definitely have something ready soon. Stay tuned!

Different Forms of Polymorphism

noreply@blogger.com (Dean Berris) — Fri, 09 Nov 2012 13:54:00 +0000

In the previous post I teased a little about vtables, how they work, and whether people were familiar with them. Before I go into detail into that I'd like to rehash the notion of polymorphism first. It may be obvious already to OOP programmers or at least those who have been educated about OOP even without using it deeply. Then I go into how vtables work, why they're there, and what good vtables allow programmers to do in C++. Even if you already know all these things, I'd be interested to find out what you think about the content of this post -- comments and discussion would be most appreciated!

First things first. Let's deal with the object-oriented notion of Polymorphism. Polymorphism is achieved in OOP by treating multiple (related) types as if they were the same. Polymorphism is almost literally translated to "many forms". Here's a simple example of the OOP implementation of Polymorphism in C++11 using inheritance and virtual methods:

View in GitHub

This is very simple, elementary OOP. We have classes Book and Door and they inherit from and override virtual methods in Object. We treat Book and Door like they were normal Objects. We have a function called "print" that lives alone and can print a reference to an Object.

This is what is called the "substitutability" principle, where we can substitute any type that's derived from Object and treat it as we would an Object. There's more nuances with pure abstract types and even final types (for which no other type can be substituted) but you get the idea.

There are a few questions that arise in this design though. Let's explore some of these questions:

Can I treat instances of Object and instances of types derived from Object as I would an int? (Regular Types)
Can I put objects of types derived from Object in a container of Object's? (Value Semantics)
If I have a type that "looks" like an Object, can I also treat it as an Object? If I can't how can I work around that? (Concepts)

For OOP, these questions get really hard to answer.

Is there a different way?

The other kind of polymorphism is Conceptual polymorphism. This is you might be familiar with if you've done anything with templates and concepts. The idea is to define requirements supported by types instead of looking at relationships between types, to determine whether an algorithm/function can be applied to instances of a given type. That's a lot of words, but code works best.

View in GitHub

We take the Book and Door types and we define the concept of an Object. This definition goes like this:

A type is an Object if it can be default constructed, can be copied, and accepts the no-argument print message (has a member function named print that doesn't take arguments).

Now let's see how we can implement this in C++11:

So, not much changed -- but if you squint you'll see a few things. First, the inheritance is gone! We just turned the print function into a template. We see that this works quite well for unrelated types, and we've now got a "generic" print function. Let's then try answering the questions we raised before in the OOP section above:

Can we treat instances of Object (or O in this case) as we do int's? Yes, as long as O is a regular type.
We have types that "look" like an Object (or more directly, types that model the Object concept), can I treat each one as I would an Object? Yes!
Can we put objects of types that model the Object concept in the same container? No (at least not yet).

Oh darn -- we're missing that part of putting things of the same concept in the same container. This is a little embarrassing, but then is there a way to do this? Let's look at our options.

First option is we can use "type erasure" to do the trick. However this still almost requires some sort of inheritance-based polymorphism. This doesn't sound so value-semantic and polymorphic (we'll look at this again later).

Second option is we can stick with the inheritance-based polymorphism and use std::unique_ptr in the container. In C++11 this is a completely valid option but it doesn't allow you to treat the objects referred to as values -- you're now relying on reference semantics to maintain identity. In most situations and in purely OOP design, you can stop here and just take the hit.

Is there a third option? Can we mix the two ideas together and achieve what we want in the end, which is objects of different types being stored in the same container?

Effectively we're going to have to be a little more creative. We can actually use both approaches to achieve what we want in the end: dealing with values, supporting polymorphism, removing the explicit inheritance, and then getting to the juice.

Before we go there, we make a little detour into vtables.

In essence, the way dynamic polymorphism (polymorphism via inheritance) works is through virtual functions, and under the hood they're represented as a table of function pointers. Each instance of a type that has virtual functions in its bases and its declaration will typically have a part of the object's storage in memory dedicated to a pointer to a table of function pointers mapping a function's signature to a function pointer. Compiler vendors may implement virtual tables differently and the C++ standard doesn't define the layout of virtual tables.

You can find out more about virtual tables (or vtables in short) if you reach for your favorite search engine and look up [vtable]. There are a lot of good articles on the web about it, but the gist is that's how virtual function calls determine which actual function to invoke at runtime via inheritance-based polymorphism.

Contrast this to how templates work and how the polymorphism is really happening at compile-time. There's no type erasure going on and there's no dynamic lookup at runtime of which function to actually call, because the compiler knows at compile time exactly which function it needs to execute at the point where the function is invoked. There's no need for a vtable in the concepts-based world.

Or is there?

Let's explore the possibility of putting instances of Object, Book, and Door in the same container by combining type erasure (inheritance-based polymorphism) and static dispatch (concept-based polymorphism). We're also going to be creative with how we define the container holding the values all together and get creative with how we implement some auxiliary "helper" functions for each type.

Intermission: this borrows a lot from Sean Parent's talk at C++Now, where he goes about it in greater detail. If you haven't yet, watch his talk on YouTube.

We change the previous listing and remove the template function "print" and instead we add the following:

View in GitHub

Now we've gotten two things working together: type erasure (we see object_concept_t, embodied as an abstract type, and an opaque object_t type that can hold different kinds of objects that can be printed) and static dispatch -- the compiler knows exactly which "print" to call at the top level. Let's re-examine our earlier questions here:

Can we treat instances of object_t as we do int's? Yes!
We have types that model the Object concept, can I treat each one as I would an Object? Yes!
Can we put objects of types that model the Object concept in the same container? Yes!

The good part about this approach is that when you have a new type that has a "print" function, you can suddenly add instances of the same type (so long as it's copy+move constructible+assignable) to the container above. This is powerful because now you're dealing with values exclusively -- instances of object_t have value semantics.

If you look closely though we're still relying on the compiler's vtable implementation because we still have virtual functions in object_concept_t. Can we get rid of those?

That's the subject of the next post. Hopefully you've gotten enough in this post to chew on and discuss going forward. I invite you to consider thinking about whether this can be useful in your codebase so that you can better leverage value semantics and get the benefits of type erasure as well as concept-based design. Do you think it would be nice if you never had to do inheritance-based polymorphism when you're designing your classes? How much easier would it be to extend your programs using this approach?

I've love to know what you think in the comments.

Thanks for reading!

Value Semantics and Polymorphism

noreply@blogger.com (Dean Berris) — Tue, 30 Oct 2012 14:03:00 +0000

Lately I've been interested in formalising the notion of value semantics for one reason or another. I've been in email conversations with other people who have been interested in making value semantics and value-oriented programming the norm in C++. There's a great debate to be had on whether object oriented programming is really the right way of building secure, scalable, and correct software solutions. This post is not about that debate but it is about how a different way of thinking may get more people understanding (and liking) how it is to think about values. In the previous post I wrote about design as computation. In this one I think about one way of designing computation.

I've already raved about Elements of Programming by Alex Stepanov before but if there's any one chapter you should read in that book is the first one. The succeeding chapters build on the first but what's really important in the first chapter is the fact that once you understand how to think about values, you can start understanding how to really use the STL and how you can start getting more and more into the practice of generic programming. Even though the book is not really about generic programming, it is a glimpse into a fundamentally different way of thinking.

Let's take something dear to the Object Oriented Programming model and look at it in a different perspective. If you have heard the term polymorphism before you pretty much think about base classes and virtual functions in C++. You're probably already thinking that the only way to do polymorphism in C++ would be through virtual functions -- and you may be forgiven in thinking that runtime polymorphism is best modeled through virtual functions and inheritance. This is however just one way of looking at the world.

How else you ask would you think about polymorphism? I've lately been glad to watch Sean Parent's talk at C++Now via YouTube where he explores a different kind of polymorphic typing using concepts. If you have two hours to spare learning about how to implement something as complex as Photoshop's history functionality using vectors and values, I suggest you stop reading now and watch that video first.

The whole idea that you can do a lot with representing things as just values is sometimes mind-boggling. I personally have been comfortable with value semantics but I've always found myself needing inheritance based polymorphism at some point. There were times that I was able to get away with not using runtime polymorphism but it always seemed to be forced or unnatural. The interesting thing is that once you see one example of how it can be done, then the gears start turning in your head and you find that some problems you've encountered before you can actually see in a different light.

What I found fascinating are the following points which I want to discuss in further detail going forward in the next few articles:

Design with copies in mind. Once you can reason about values then the simpler your programming model becomes.
Know when to move. Copies and optimisations on copies is one thing, it's another to know when it's better to move. Understanding how moves work is crucial in getting this right.
Avoid shared state. This is easier said than done, but if you design with copies and values in mind then shared state becomes a code smell and something you should start avoiding if you want to be able to reason about your code locally.
Find the abstraction and leverage it ruthlessly. Sometimes the simplest abstractions are the best ones to build more complex abstractions on, while complex abstractions become harder to conceptualise when you keep it simple.

I'm going to enjoy writing about the above four points in detail in the coming articles. In the meantime I leave you with a question:

Do you know what vtables are? Do you know how they work? If you answer yes to both, can you simulate vtables and use templates to adapt a set of types onto a concept? If you don't know what vtables are, can you make an adapter type that adheres to a non-virtual interface but still allows you to have different implementations at runtime?

Musings: Design and Computation

noreply@blogger.com (Dean Berris) — Tue, 02 Oct 2012 12:46:00 +0000

I have a personal theory that the fundamental source of bugs in most software is the impedance mismatch between the notion of design and computation. We as software engineers are torn pulled in usually two seemingly different dimensions -- that of elegance in design (of systems, of interfaces, and of architectures) and that of computational efficiency (in implementations, algorithms, and practical realities of the computers we deal with). There's been countless books written about design: you have the design patterns movement which paints the world in terms of structure and interaction between objects, pattern languages dealing with different families of systems, etc. I know of a few books written about the practice of algorithmic problem solving: generic programming books, functional programming books, "Elements of Programming", and Knuth's The Art of Computer Programming. Focusing on either usually leads to gaps in a complete understanding of the problem space and therefore the result is badly designed and implemented solutions. Is it something we can do anything about? This post is my attempt at exploring the intuition that says "we can marry these two modes of thinking if only the tools we use allow it".

Before anything else, let me further qualify my personal theory by elaborating a little more than a single sentence.

There are two modes of thinking when solving problems using a computer program: design and computation. When in the design mode of thinking, we think about structure and organization of code, of interactions and relationships, and data flow. When in the computation mode of thinking, we think about mutation, transformation, state, transition, invariants, side-effects, and order. When in one mode we're blind from another, yet when we switch modes we influence the solutions in either.

When reading patterns and pattern language books I notice that the writers usually take a structural form of code and address different forces like extensibility, isolation, encapsulation, and polymorphism. It's heavily reliant on fixed interfaces and uses interfaces to convey relationships almost exclusively as opposed to invariants. Because invariants are only implicitly (or externally) conveyed in the interface, they aren't subject to the same rigor as when in the algorithmic frame of mind -- invariants, complexity, and side-effects aren't as obvious when thinking about design.

On the other hand when I'm reading algorithmic books like "Elements of Programming" and "The Art of Computer Programming" it seems that design is an artifact as opposed to the goal or even "the subject". It's hard to reconcile the design-based mode of thinking when algorithms are meant to operate in a hermetic fashion -- you have inputs, transitions, mappings, iterations, and traversals of data structures producing output. Most of these books read like an exercise in mathematics and it just so happens the programming language being used to implement the solutions require structured code to come up with the solution. Sometimes though the notion of a mathematical function is lost when things like mutation and state come in, and only because there are very few programming languages where math can be properly encoded efficiently to implement algorithms (Haskell comes to mind, but there's still the reality of state and I/O which requires Monads that remarkably not very "math-like").

I see a few ways of attacking this impedance mismatch and I'll just throw these out there.

Design is Computation. There aren't a lot of programming languages that allow this -- Lisp has a macro system that allows programmers to treat code as data, and since code is an embodiment of design then design and computation come hand-in hand. However if you've seen any serious Lisp system where macros are used to great effect, you will see that there are intermingled masses of design and computation in pockets of domains all being glued together by the syntax tree. There's probably a reason why Lisp hasn't been dominant even with the great power that it affords programmers that use it.
Computation is Design. There are languages where the only way you do any meaningful computation is with design. Languages like Ruby encourage and support writing domain specific languages where you use composition and design to organize and effectively perform computation. Another language that allows this is Java, where the only meaningful way of doing any sort of computation is by designing your solutions into classes and functions. OOP languages encourage computation by design more than other languages.
There is only solutions. Languages that allow writing solutions and providing features for implementing these solutions do not tend to weigh to either side of the continuum. This is where I think C++ is capable of excelling where you get design as a means of organizing computations. This I think is the "sweet spot" but it's lacking in enabling the previous two modes of thinking.

There are valuable lessons to be learnt from the first two points. In the "design is computation" mode of thinking, being able to perform calculations based on properties of the computation is very powerful -- making decisions based on the structure of the code as opposed to just inputs and outputs yields far superior automated ways of "yielding" design. More to the first point, being able to assert certain expectations on design as a computation is a very important tool for enforcing invariants and expectations on design. This enables solving one of the issues with the design-only mindset.

In the "computation is design" mode of thinking, enabling thoughtful and rigorous computation through composition is a hallmark of scalable (and re-usable) design. It's no secret that C++ has already been home to multiple domain specific languages but the key to this is being able to perform the computation only as a by-product of design but in the process of design. Generative programming relies on being able to use the design of programs and performing computations on the design to yield new types and therefore new behavior. Extracting the generative parts of DSEL building is already being done by Boost.Proto but only to the extent that it only has partial information available compared to the compiler -- one approach used by Haskell is to allow defining extensions and optimizations in the language itself, dealing with certain constructs to enable transformations that make sense in a given context.

In the world of "there is only solutions" we're currently in the middle -- C++ has been a very wide programming language offering both OOP techniques and to some extent some functional programming techniques. With C++11 we've expanded the functional programming aspect with lambdas. We have more facilities for generic programming in auto and decltype. Templates got more love with variadic templates. Template functions got the capability of defaulted template parameters enabling better use of SFINAE techniques.

So are we stuck with trying to reconcile these two modes of thinking? I don't think so and there are a few anecdotes that points to the attempts at reconciling them though attacking this from two different directions.

First off, there are a number of attempts at making computation at compile-time more powerful than with the current state of constexpr. There's ongoing exploration of the possibilities for expanding the scope of what can be done in this space is really exciting as it widens the potential of being able to do something like Lisp macros at compile-time for generative programming. Even though we have some of these techniques implemented as template metaprograms, some other techniques we can't just achieve just yet with just template metaprogramming.

Secondly, this ties in really well with being able to write C++ that works on types for real generative programming. This thought process is just in the beginning stages for me and I've a number of ideas on how I think introspection and reflection will allow more generative programming techniques. I'm still at the phase though of trying out a few forms which are consistent with "normal C++" and I'm nowhere near ready to show any of these preliminary attempts. Maybe further down the line I'll be able to share more with a wider audience.

Lastly there are a number of other ways to achieve the result of married design and computation. I'll explore those in upcoming articles as I look deeper at what it means to do design-driven development and computation-driven development with C++. In the end I would hope to construct a middle-ground to get both design and computation at a good mix to leverage the power of C++ from both sides.

Discussion on the topic is most welcome, please feel free to comment below. If you have thoughts you'd like to share as an article on C++ Soup as well, please contact dean.berris@cplusplus-soup.com so we can get your guest post queued up.

Rich Pointers (Update) and Reflection

noreply@blogger.com (Dean Berris) — Fri, 28 Sep 2012 16:00:00 +0000

The Portland meeting is just a few weeks away and even though I'm not going I along with my co-authors have prepared an update to the rich pointers paper. It took a while for us to update the paper because contributing to the C++ standard isn't my 100% commitment at Google, but I am happy to be able to take some time to commit to updating it, as well as having my co-authors review and make changes as well. The result is a greatly simplified proposal which turns the proposed new pointer type and turns it into a library interface. The paper also gained a static metadata proposal that aims to make type introspection a first-class operation.

The motivation for getting introspection into the language as a first-class entity is backed by a lot of previous and well known pain-points in three main areas:

Serialization -- being able to save the state of an object as a sequence of bytes and reconstituting a new instance of the same type at some later time. This is important for all sorts of applications, one of the most compelling is distributed computing.
Generative Programming -- being able to generate and compute with types to reduce repetitive code. Automating repetitive patterns like setter/getter functions and adding value semantics to types via generated proxy types at compile time are very powerful techniques.
JIT and Embedded Runtimes -- allowing generation of types in embedded runtimes to be inspected and used appropriately by the hosting application. Dynamic languages have this capability because precisely of this feature (think about how "eval" and similar functions work).

C++ has generally been viewed as a bare-metal power-tool for heavy duty performance-critical development, so why bring in introspection features into the language now? There's a number of reasons but better than reasons there's also experience.

Personally I've had to deal with the issue of serialization at one point or another in my career. I've been happy with Boost.Serialization which I've used in production successfully. I know the power of being able to send data from one place to another via the network and being able to scale systems horizontally this way. There are countless numbers of distributed programming systems that rely on serialization. Precisely because there's no automated way to make this happen is the reason why multiple man-hours and millions of dollars worth of bugs in C++ have been wasted.

One of the better solutions I've run into before and while at Google is this wonderful thing called Protocol Buffers. Protobufs have this property that you can extend the definition and still have older messages be valid and even backward compatible. It does this by having a means of associating metadata with the actual message on the C++ side, so that it can interpret messages in a consistent fashion. The idea is to use a type descriptor on the C++ implementation to be able to know how to deal with messages.

The unfortunate part of protocol buffers is that messages have to be defined in a different configuration language. This means if you want to get the extensibility and performance that protocol buffers afford you when serializing and deserializing messages you need to define protocol buffers using a different language. The reason for this is because C++ doesn't have a way of allowing code inspect a type definition (struct or class) to figure out what the data members are. With a static reflection mechanism, we can use a struct's definition as a protocol buffer definition and generate efficient serialization/deserialization routines. With dynamic reflection we can look at objects with runtime polymorphic types and determine what to do at runtime what to do.

I share a passion with my co-authors and other people who want to see C++ be a language that allows heavy-duty general programming solutions to be both powerful and awesome. We have a great facility in templates and the new C++11 features like constexpr that may enable at a later time more generative programming techniques. I hope static metadata along with dynamic introspection would allow standard library implementers to provide a serialization library taking a cue from protocol buffers -- or something else, hopefully as efficient if not more.

The full paper submitted to the ISO committee is N3410 for the Portland meeting. You can find the sources that generated them in the N3340 repository at Github.

I would like to thank Matt Austern, Lawrence Crowl, and Lally Singh for their co-authoring the paper with me. I'd also like to thank the ISO C++ Committee members who gave constructive feedback in the Kona meeting who have helped shape this revision to the paper.

cpp-netlib: HTTP API Plans Update; Part III - Details

noreply@blogger.com (Dean Berris) — Thu, 27 Sep 2012 16:00:00 +0000

This is the final part of the series on the HTTP API Plans. The previous two posts shed light on the history and the current issues with cpp-netlib's implementation and why we'd like to make changes to the client and server API now. In this post we look at the details of the design document which outlines this new API. Our eventual goal is to get to a point where the API is ready for proposal to the C++ committee as a reference implementation. If you haven't read the document yet, please do that now before proceeding any further -- or keep it open in a separate tab/window so you can refer to the sections appropriate in the discussions below.
In the document we list out a few goals. Let me list those down here too just for convenience:

Support configurability.
Support fully asynchronous operation.
Streaming should be a first-class concept.
Basic session and authentication support should be built-in.
Use standard C++11 libraries and features.

Let me go through each of these goals in a little more detail.

Configurability

A lot of HTTP is about options on how the client behaves and how requests are processed. Artificially limiting the configurability of a client is bad design not only for users but also for implementers. The current latest version of the library doesn't support many options largely because the API has not been designed to be configurable in the beginning. The mistake here was thinking that configuration would be complicated so we rather provide "sane" defaults -- unfortunately the defaults though are still hard to change and that's not a good user experience. For something that's meant to be a full-featured HTTP client or server, configurability is an essential feature. We aim to change that by introducing a configuration idiom that is extensible and familiar.



namespace http = network::http;

http::client_options options;

options.follow_redirects(true)

       .accept_encoding({"*/*"})

       .version(http::version::http_1_1)

       .proxy("localhost:8080")

       .idle_timeout(30);

http::client client{options};

If you look closely at the above code, the configuration is as declarative as possible and in case we do support new options later on adding those in would be a matter of adding new function calls. This lets us, the implementers, support configuration options incrementally without having to jump through too many hoops. A similar approach is done for setting per-request options.


namespace http = network::http;

http::request_options ro;

ro.timeout(std::chrono::duration(30))

  .client_cert("/tmp/my_cert.pem")

  .accept_transfer_encoding({http::transfer_encodings::gzip, http::transfer_encodings::plain});

auto response = client.get("https://www.example.com/foo", ro);

Now in this case we're setting the options both on a request level and a client level. We're allowing this so that we can override client-specific options on a per-request basis. This gives the library a fully configurable interface in the case where we do want configuration. The basics will still be choosing defaults, which makes this a very attractive approach.


namespace http = network::http;

http::client client;

auto response = client.get("https://www.example.com/foo");

The main direction the library is going is that of "making simple things easy and making complex things possible". We achieve this in the client following the above pattern. For the server implementation though we allow configuring the server dispatch instead of making users roll their own dispatch mechanism, and make configuration the default way of using the server implementation.


namespace http = network::http;

http::server server_;

server_.register_handler(

    "/hello", [](session& s, std::shared_ptr<connection> c) {

      c->response().write(s["user"].empty()? string("world!") : s["user"]);

    });

server_.register_handler(

    "/echo", [](session& s, std::shared_ptr<connection> c) {

      if (c->method() != http::server::POST) {

        c->response().write("echo!");

        return;

      }

      c->data().read(getpagesize(), [c](const std::string& data) {

        if (!data.empty())

          c->response().write(data);

      });

    });

s.run();

The intent is to configure a server more than to implement one. There are more details on the document about changes to the HTTP server API so please have a read of that to get more of the elaboration.

Fully Asynchronous Operation

The library has traditionally been mixing both synchronous and asynchronous APIs in a very clumsy way. Thankfully though with C++11 we have a de facto way of doing asynchronous APIs in the form of std::future. This handy utility along with a standardized means of performing asynchronous operations with std::async makes it a very attractive option for simplifying asynchronous programming with C++.

In the previous code snippets using http::client you'll find a response object whose type was deduced using auto. Changes to the API will make calls to the http::client functions return std::future objects. The intent of this is to convey directly through the API that all the calls made to the client are asynchronous. There are many standard ways of dealing with future objects, which allows users to easily schedule multiple concurrent requests.


namespace http = network::http;

std::future<http::response> response_future =

    client.get("http://www.google.com/");  // returns immediately

http::response actual_response =

    response_future.get();  // will move instead of copy

As an aside, in the future it's highly likely that std::future will gain a "then" member function that allows you to define a completion handler when the future is ready. Until then, explicitly waiting on a future is best supported.

In the server implementation all the function calls associated with the response are all asynchronous. There are a lot of details when it comes to the server implementation that's covered better in the document.

First-Class Streaming

The current implementation of body streaming is very clunky and is a big source of confusion on how to best handle based on the interface. This stems from the problem that the API was not designed to meet the streaming use-case from day one. The plan is to change this and support streaming as a first-class (and in fact, the preferred) operating mode.

In the Client API, we have two chances for supporting streaming: when requesting and when handling responses. Given that we already have a means of waiting for future responses, we also define a clear way of providing streamed data to requests. It used to be that the payload for HTTP requests would need to fit in the client's memory (in a std::string at that) but now we've changed this to allow streaming from an arbitrary byte source.


namespace http = network::http;

namespace util = network::util;

http::client client;

auto fr = client.get(

  {"https://www.myservice.com/",

   util::gzip_encode(new file_byte_source{"/tmp/data"})}

  );  // we're gzip encoding the request body

auto r = fr.get();

std::string data = r.read(getpagesize()).get();

while (!data.empty()) {

  std::cout << data;

  data = r.read(getpagesize()).get();

}  // will print data as it comes

When reading the response body in chunks notice that we are also using futures, this time std::future. We're relying on move assignment and move construction for this interface so that we are consistent with all the APIs where we support asynchronous operations.

In the Server API we're dropping support for the synchronous implementation. Going forward operations on the connection API imply streaming (and chunked, depending on client request properties like HTTP version supported and whether chunked transfer encoding is supported) body responses.

Basic Session and Authentication Support

This goal applies to client and server code. One of the biggest feature requests has been to support cookie processing for both client request/responses as well as for server-side processing (setting/retrieving cookies, etc.). Because precisely everyone is eventually going to need this, the library will support this from configuration options on the client and the server-side. Only basic support will be added and hopefully when the infrastructure has been laid down others will be able to contribute different kinds of session and/or authentication managers.

On the server side session parsing will be first class but will depend on a SessionManager policy. This policy can be controlled both statically or can be plugged in dynamically. An API for the SessionManager will be provided when the server is actually implemented.

On the client side session management will be plugged-in dynamically and supported on both the client and per-request options. Essentially cookie parsing and variable management will be hooked into the request and response handling. The basic cookie management facility will be provided, but in the future different strategies like encrypted cookies and OAuth2-style digesting among other things will be implemented.

C++11 Features and Libraries

Needless to say (and as should be obvious in the examples) the full power of C++11 will be used in both features and libraries. This will make cpp-netlib less reliant on Boost for things like function<>, move(...), shared_ptr<>, and even Asio (since stand-alone Asio already has support for C++11 features). I've already mentioned std::future, std::thread, and std::async and it's worth noting the welcome language feature called lambda's will make all this really fun to code.

The over-arching goal is really two-fold: implement a modern C++ networking library for wide availability and contribute to the C++ standards whenever possible. We've taken the URI proposal to the committee and hopefully some time in 2013 we can bring cpp-netlib's HTTP interface and implementation experience to the committee as well.

Conclusion

This concludes the three-part series on cpp-netlib's history, current state, and future direction. There's been a lot of lessons learned with the implementation of cpp-netlib over the past five years and I'm personally thankful for all the effort people have already put into it all these years. The feedback we get from users and contributions we get from like-minded developers makes cpp-netlib a project worth continuing to work on for me. Even though goals have changed and we're reaching higher and getting more ambitious, I personally stay committed to making sure the project and the community grows to become something that users, contributors, and observers can be proud of.

Thanks for reading and I'll hope to see you in the mailing list discussions!

-- Dean Michael Berris, Founder and Maintainer of The C++ Network Library Project

If you liked the direction of the HTTP Client/Server API in cpp-netlib please let us know at the cpp-netlib mailing list. Contributions in the form of pull requests for code/documentation and early-adopter feedback would be greatly appreciated -- you can find the cpp-netlib project on Github here.

cpp-netlib: HTTP API Plans Update; Part II - Current Issues

noreply@blogger.com (Dean Berris) — Wed, 26 Sep 2012 16:00:00 +0000

Yesterday's article was an abbreviated history of the cpp-netlib project. Before I go into greater detail on the planned HTTP API updates (see the document here) I'll write about what the current challenges are with the library and getting changes in and a little about what we're doing to address these short-comings. This is part two of a three-part series that aims to highlight the work that's gone into the library and how it's going to change moving forward. The next part will cover the details of future work on the project as far as HTTP is concerned among other initiatives as well.

Given the history of the project, it's been evolving from simple to a little more complex and back to simple again. The strains encountered in the internal design have mostly not manifested in the API -- thankfully we've been able to keep the API of the client implementation stable across three releases. I think we've been very lucky to be able to do that because we've kept the API as simple and straight-forward as possible.

Unfortunately though the requirements on modern C++ applications have since changed from 2007. More and more resources are moving to HTTP-based APIs for good reason: HTTP is a ubiquitously supported in most modern programming languages either as primitives (or close to primitives, as in JavaScript) or as part of a standard library. It's also traditionally (and continues to be) the most allowed protocol through multiple layers of security. It is simple and applies very well to a whole host of programming patterns involving remote resources and sometimes data interchange (you may want to look up REST APIs to get a feel for what I mean).

Another thing that happened since 2007 was the focus on multi-threaded programming and the acknowledgement that the future is increasingly concurrent and parallel in C++. The C++11 standard defines a memory model and standard libraries for threading in C++ which makes concurrent programs the majority of programs written going forward. A whole host of libraries came out like Intel's Threading Building Blocks which focuses on getting foundational algorithms and data structures that leverage the new multi-core architectures and multi-threaded programming paradigms.

The combination of a want to support fully asynchronous I/O models along with the pressures of supporting threads and synchronization primitives in a standard manner has meant that the current API for HTTP clients and servers in cpp-netlib are no longer "modern". For a library claiming to want to provide modern C++ usage semantics, it's not doing a good job of exposing an API that lives up to the modern standard.

Other issues being encountered is still the inherent and apparent disarray of the implementation since it has evolved from a purely blocking (read: non-asynchronous or synchronous) model to one that's mixed the concept of sync and async APIs together. Because of the desire to support both sync and async modes the APIs become neither -- and thus supporting more features means a lot more work than necessary. The poor organization and documentation of internals also makes it really hard for someone new to the project to get up and hacking on the internals.

Even though most of the template metaprogramming has gone, the types in cpp-netlib are still templates. However though it's been really hard to keep the API stable with the move from header-only to non-header-only as some interfaces needed to become non-templates. Parts that used to be exclusively static that didn't need to be purely static have been migrated to becoming dynamic interfaces. Things like determining the behavior of a connection based on the URI -- should be using an encrypted socket (SSL) or not and whether the response should be handled as HTTP 1.0 or HTTP 1.1 just to give a couple of examples.

Initial attempts at providing an API for handling the body response chunk-by-chunk but not as part of the response object was a very bad design decision. It broke the simplicity of the API and it introduced a point of complexity where it shouldn't have had to be there in the first place.

This is the client side only though -- the server side had its fair share of problems design-wise.

The purely synchronous (but thread-aware) version of the server implementation suffered the same problem -- because it defined the handler interface as a generic but static concept-based approach, most of the required details of modern HTTP 1.1 implementations couldn't be introduced easily. Because the server was designed to be very generic, it couldn't get most of the usual details right. I'm talking about things like session management, supporting request queue limits, supporting HTTP/1.1 pipelining, among other things. Even though I've implemented these things before in a proprietary implementation, I didn't have to think about external usage scenarios.

The server also suffered from the sync/async split. Because precisely of the generic API that simplified (too much to a fault, using the wrong abstractions) handler implementations, things like sessions couldn't be made part of the library implementation right away. There also was not a way to get request handlers to write chunks out to the connection, since the API was fully synchronous. The implementation then of the synchronous and asynchronous servers diverged greatly and the maintenance burden of supporting one feature in one implementation while not supporting it in another has literally doubled.

These factors have contributed to the acceptance and recognition that the current interface and implementation are insufficient -- that they cannot really be just evolved incrementally from the current state to a future state. Supporting the synchronous implementation no longer makes sense in this world of C++11 where concurrency is a first-class citizen and that asynchronous operation is the norm. The API is seriously hampered flexibility-wise because C++03 didn't support enum classes -- for example we made a choice to just support GET, PUT, POST, DELETE, and HEAD as member functions, instead of providing a request function that took type-safe enums that determined statically which request implementation to pick.

A wanting to leverage the dynamic nature of HTTP request/response handling also influenced the direction of both the client API as well as the server API. C++11 has lambdas and they are powerful for defining not only completion handlers (or callbacks) to support real streaming, but it also provides a more localized and simpler handler implementation. With the help of real modern C++ design idioms (policy classes, dependency injection, configuration patterns, and scalable error handling, etc.) I'm personally looking forward to rationalize and greatly improve the facilities available when doing HTTP server-side C++ development.

In the next part of the series I dive into the API changes and the explanation to some idioms and patterns that recur in the new design. I also tackle the direction that the project is taking and how you can help the team achieve even more ambitious goals.

cpp-netlib: HTTP API Plan Updates; Part I - History

noreply@blogger.com (Dean Berris) — Tue, 25 Sep 2012 16:00:00 +0000

With Glyn's work on the URI proposal the C++ Network Library project has been moving forward onto bigger goals. One of these goals is the overhaul and update of the library to use more C++11 features and to update the HTTP client and server implementations based on past implementation and user experience. A lot of the feature requests and issues encountered with the older API are being addressed by the API overhaul. This post is about what direction the library is taking with background information on the recently published HTTP Client/Server API Design Doc (link). This is Part 1 of a 3-part series on the history, current issues, and future direction of the cpp-netlib project.

First, a little bit of history on how the C++ Network Library came to being. This is important to frame the following discussion on what's happened since then and what will be happening going forward.

I was working on a project that required making HTTP client requests to a third-party service from a client application in 2007. Up to that point I had mostly been working on HTTP on the server side -- implementing servers was something I was used to doing and I much enjoyed. However though I found myself unimpressed with any of the interfaces of HTTP clients that I cared to look at -- at the time there were mostly C libraries (CURL is a popular one) that did it and on Microsoft Windows there were the .NET managed C++ bindings for the HTTP implementation (can't remember if it's in the ATL).

I thought to myself: this couldn't be right, there should really be a simple HTTP client implementation in C++ out there. At the time Boost.Asio was only getting into Boost (1.34 was out then, but I was already using Boost.Trunk because of Boost.Asio) and the example HTTP client implementation seemed simple enough to use. However I ran into issues with Windows having wchar_t as default so I can't easily just stream bytes to the server, because HTTP is an ASCII/UTF-8 protocol.

This gave me a choice: either I use the managed C++ bindings and the Microsoft ATL or start an open source project that implemented an HTTP client using Boost.Asio. As any self-respecting open source enthusiast with an extremely positive attitude and an itch to scratch would do, I chose to start an open source project to tackle that problem. I first set out to get Boost developers involved in the effort too with the hopes that I would be able to spend 6 months worth of unpaid work time to hack something up quickly and get it reviewed and maybe accepted into Boost.

My experience with implementing an HTTP server in C++ got positive results in doing a lot of things with template metaprogramming and little to no dynamic polymorphism. This gave me an idea to do the client implementation the same. I am personally very comfortable with template metaprogramming and had been very successful with it on the server side that I thought it would translate well into the client side.

Boy was that a mistake -- not the starting the project part, but in the under-estimating the amount of work that goes into an HTTP implementation using template metaprogramming techniques for extensibility. What I came up with initially worked well, was efficient enough, but was single-threaded and didn't use resources effectively (all the data is in std::string's and I still couldn't support std::wstring's). The early releases of the project worked its way to a point where the string type was a template parameter and that the only implementation that was actually tested was the std::string -- literals were just bare "const char*" instead of being something wrapped in a macro.

Initial feedback on the library's client interface has been positive. It's really simple to use and you approach the HTTP client like it was an object that did all the HTTP work for you -- you pass in a request, you get a response. Easy, simple, intuitive.

Then requests came in to get an HTTP server implemented. People liked the client implementation and some had asked whether we can get a server into the library as well because that's where C++ would really make sense. Not a lot of people wanted to write C++ code that actually made calls to HTTP resources, as there are a lot of other dynamic languages that would make this task really easy (besides there's already a lot of client libraries out there, so this is a very big mountain to climb). I hesitated first to implement an HTTP server because even though I had done this while I was at Friendster, that implementation remained proprietary despite my pleas to make that implementation open-source.

When I left Friendster in 2009 though I found myself able -- and wanting -- to implement an HTTP server template that would be simpler to use and was more generic. Unfortunately I couldn't bring the techniques I used while I was in Friendster because the build times for the server were actually pretty awful -- a full build took me, with -O3 optimizations, using GCC 3.4 at the time 20 minutes on multiple concurrent threads on a quad-core 2007-class Opteron. The resulting binary was tight and the bottleneck had become the memory-cache transfers (computations were mostly memory I/O bound, not CPU bound) but you can't ask everyone using the implementation to wait 20 minutes before they can start using the resulting binary.

I hacked up a quick server side implementation and got it mostly working and put it into the project. There were lots of requests for features but I had found myself involved with other endeavors (and real life -- I got married and became a father in 2009 and had started on a new role with greater responsibilities as well). The server implementation had been useful to some people already and is one of the popular topics on the mailing list right after the release where it was included.

Then another mistake was thiking that the tag-based trait system -- which is still the most flexible way of designing static type-families (I should really write a paper about this technique independent on cpp-netlib) -- would scale well. By the time BoostCon 2010 came up, I had an HTTP client and server implementation that only had one template parameter: a tag. All the traits required to make decisions in the HTTP client's and server's implementation was hinged on this single type -- there were a dozen trait metafunctions that determined what strategy to use for resolving domain names, what type of string to use, whether the implementation was fully async, etc. I also provided a way of creating tags by composing multiple "foundational" tags.

This approach proved effective -- if you think about Haskell's type classes, the tag would have been a type constructor where you had functions defined in the form of trait metafunctions.

That was the good part -- the bad part is that building anything that used the client or the server implementation meant 15 minutes of CPU time and >2GB of RAM compiling the object file. Some users had complained that MSVC 2009 couldn't handle it in 32-bit mode. That clearly wasn't going to work.

I trudged along though and still maintained that implementation. The problem came then when I wanted to allow streaming support to the client API. All the template metaprogramming that made things efficient and effective also made it really hard to read, even for me. I would spend hours building the mental model of what I wanted to achieve and trying to rationalize the solution and fit it into the ever growing tangled mess of templates and intermediary types. One day a typo I introduced that turned a reference into a bare copy parameter caused a spew longer than 2000 lines -- and I didn't want to touch my own implementation for a couple of weeks.

I became disgusted with the implementation and I found that the only way forward was to abandon the header-only approach and remove most of the template metaprogramming. This allowed me to take a good look at the API and cut down a lot of the mental gymnastics just to implement a feature. I still love the power that template metaprogramming affords me when I need it. Unfortunately though I cannot tell anybody with a straight face that template metaprogramming is something I still love doing.

With this realization, more recent work became moving most of the implementation from being reliant on template metaprogramming to leveraging more traditional library design. I embraced the notion that I can model things better using OOP and implement the generic parts as templates. The current stable release (0.9.4) already defaults to a non-header-only approach which cuts down the build times for users significantly and makes it easier to integrate the library in more traditional applications where development cycles rely on quick builds.

In the next part of this series I write about what the current state of affairs is, what the issues we're currently encountering in supporting new features and addressing deficiencies in the implementation are, and see why these issues exist. The final part of the series will delve into greater detail on what the reasoning is for the API refresh and what the plan of attack is for further work on the library.

C++ Standard Proposals: A Standard URI Type

noreply@blogger.com (Dean Berris) — Mon, 24 Sep 2012 16:00:00 +0000

Much has been going on with The C++ Network Library project even though I've not personally been productive this past few months. Glyn Matthews (follow @glynos and on Google+) has been hard at work in getting the URI library implementation in cpp-netlib in good shape and he's written a proposal to get the URI library interface adopted in the next version of the standard. If you'd like to get the details of the proposal, you can get to it from his Github page. This post is about a little background as to why we're doing this and what we intend to achieve as a larger goal.

It's not a secret that the C++ standard library is full of great things already, especially in C++11. However the committee believes that there's a lot of great ideas outside of the committee and that we'd like to get more useful libraries into the standard library. There's a call for proposals for the library working group and members of the committee are looking forward to hear the great ideas coming from the community on what should be included in the next standard.

One of the areas which is sorely lacking especially in this day and age is a standard network programming library in C++. There are a number of foundational libraries (Boost.Asio is one, ACE is more a framework, while Berkeley Sockets are very low-level building blocks) which deserve mention and attention, but nothing yet has become a "de facto" standard. The committee recognizes that network programming is a crucial area for modern application development -- after all we are living in the age of cloud computing.

One of the goals then is to get a coherent and comprehensive network programming library into the standard. Glyn and I have been working together for a while now on the C++ Network Library project since 2009 and we both have been doing this on a part-time best-effort basis. However we believe that our experience with implementing some of the hairy parts of HTTP (I still have a dream of getting other protocols included in the library) especially the URI implementation would be useful in getting a conversation started about how it should go.

We've worked together with Niklas Gustafsson from Microsoft and Kyle Kloepper from Riverbed (head of the Networking Study Group) in forming this proposal. Niklas has also been in contact with people working on the Casablanca project -- which, by the way I'm very excited about as it's a validation of the clear need for a modern C++ library for network programming -- and has given great feedback on the early drafts of the paper. Glyn though did all of the heavy lifting of actually writing the proposal and now we've got a conversation starter for the Portland meeting.

This idea of getting a network library into the standard is a big undertaking, one that will require lots of hard thinking and implementation experience to get right. The last thing we want to have is to put something into the standard that cannot be implemented in a sufficiently efficient and effective manner and has an interface that does not achieve its goals. On the other hand we'd really want to have a networking library as part of the standard to some extent, even if they're just foundational -- although it would be awesome if we can get at least an HTTP client in there.

So for Portland, we have the URI implementation. If more people can help out in getting the implementation up to snuff for the next meetings as a reference implementation, that would be a good thing. Hopefully by this time next year the HTTP implementation of cpp-netlib will have also evolved to use C++11 fully, so that we can have HTTP servers and clients proposed for the standard library.

This is an exciting time to be part of this project and I'm thoroughly thankful for Glyn's efforts in doing the work to get this proposal to the committee. Now we'd appreciate feedback and contributions in making the goals of cpp-netlib a reality, and hopefully all C++ programmers get to do network programming in a standardized manner across all platforms where C++ is supported.

Stay tuned as we try to achieve more with cpp-netlib in the coming months and years.

If you'd like to get involved in the project, please subscribe to the cpp-netlib mailing list and follow/fork the cpp-netlib project on Github. We'd love all sorts of contributions. The goal has slightly changed from being a project that's supposed to get into Boost, to one that's a self-sufficient and active project with a community built around it. What hasn't changed though is the development and maintenance of a production-quality modern open source C++ library for higher level network programming. Template metaprogramming experience no longer required to get involved either.

Portland Meeting: Oct. 15-19, 2012

noreply@blogger.com (Dean Berris) — Wed, 19 Sep 2012 12:00:00 +0000

It's just a few weeks away and the ISO C++ Committee will be meeting again this time in Portland, Oregon. If you're interested in the agenda, you can find it from the official Post-Kona Mailing (N3373). Just like the previous meeting the committee will continue to discuss defect reports, new library proposals, and language extension proposals. This meeting should be interesting because some new and exciting things will be coming in through the pipeline. The papers haven't been finalized yet but judging from what I've been hearing it's going to be a filled with cool updates and new proposals taking a lot of user (and expert) comments into consideration.

Unfortunately though I won't be able to make it to the meeting as I have a scheduling conflict with other events. That being said though I will be sending an update to the Rich Pointers (N3340) paper with updates and comments from the previous meeting. I'm just working out some final kinks -- if you're interested in seeing the progress you can head on over to the N3340 github project (the paper has been converted to LaTeX instead of a Google Doc).

There are a couple other things that may be coming up which I will be discussing on the blog when the pre-Portland mailing gets finalized and published. If you're interested in this process, stay tuned!

In the meantime if you're looking to get involved with the process, I can shed a little more light on how it works in this Google Tech Talk I gave earlier in the year. The upshot of this is to point out which libraries you think should be made part of the standard (or at least what libraries you'd like to see part of the standard) and let us (those who go to the meetings) know so that we can start working on the proposals. If you would like to get involved in the proposal writing process too then that would be awesome -- the committee always appreciates all the help it can get especially outside the meetings.

I've taken the liberty to embed the video here so you don't have to watch it on YouTube.

As always, feedback is very much appreciated.

C++11 -- The Time is Now

noreply@blogger.com (Dean Berris) — Fri, 14 Sep 2012 11:38:00 +0000

All the compiler vendors are racing to get C++11 features supported. There's currently a C++ "arms race" that's only benefitting users and the community at large. I don't currently have the time to survey the various language features and library implementations available out there so I'll leave that out for later. In the meantime I'd like to encourage everyone who's been waiting to use C++11 features and can get them from their compilers to go ahead and do it. In this post I list down some of the reasons why it makes sense to depend on C++11 now more than ever.

The Landscape

Currently we have three major compilers out there getting attention and fast iteration releases supporting more and more of C++11: MSVC, Clang, and GCC. I haven't kept abreast of the features that each one of these compilers support but it's interesting to see that more and more of C++11 is becoming available to more people.

Much of C++11 features that get the most usage I would imagine are the ones that don't require too much changes to current code and mental models. Things like lambda's are really powerful and as far as I know all three compilers already support them in their latest two releases. This makes using the STL algorithms a lot easier. Another is range-based for loops. To those already doing template metaprogramming things like function template default arguments, auto, decltype, and variadic templates along with all sorts of new SFINAE tricks are going to make life a lot easier.

We already have experts writing about the new hotness that is move semantics. This will take a little more time to become well understood by a lot of C++ programmers. Just the differentiation between rvalue references and lvalue references might not make it easily understandable -- as I don't think there's any other programming language out there that has these concepts. Variadic macros I hear are really powerful but I've never really fully understood the preprocessor so I tend to stay away from it. The seeming duality of constexpr functions -- does it execute at compile time or does it execute at runtime? -- makes it all the more exciting.

Then there's concurrency which ushers the new era of a number of different programming styles aiming to take advantage of the boom in making sure you're using all the cores available on your machine. Instead of just counting the cycles on a single code now you're thinking about cache-line alignment of objects and atomic operations as well as which memory model you're going to leverage. Gone are the days where you just think about the processor pipeline but now you also have to think about the NUMA channels, the L1/L2/L3 caches, and even the OS VM implementation.

Uncharted Territory

In my personal opinion I think we are at the edge of the known and understood parts of C++ programming. Up to now there's been good understanding of the features we have available to us but then we get to a position where we're trying to push the boundaries. The people who designed lambdas and the concurrency model understand that we can execute the lambda's in user-created and managed thread-pools. We still don't know though whether someone comes up with clever lambda programming that will simulate a LISP machine that's actually concurrency-aware (yes, I'm just making that up).

We still don't understand how constexpr will actually change the way people wrote functions in C++ -- will everyone then try to isolate their non-mutable computations that may be statically optimized away by the compiler into constexpr functions? We don't know what the effect of that on compilation times might be, or whether someone will actually just come up with an embedded compiler engine that knows how to do constexpr evaluation on the fly (yes I'm just making that up). Also we don't know whether we can get purely compile-time programs that don't look like template metaprograms but that look like "regular C++" that just so happens to be executed at compile-time.

Then there's domain-specific embedded languages. I can already see lambda's being named, composed, and with the help of constexpr maybe even evaluated/expanded at compile-time -- not sure how feasible that is though. Now that it's relatively easier to do perfect forwarding in C++11 libraries like Boost.Proto (which I maintain is a work of pure genius) and Boost.Spirit (which is still the most awesome library I've ever seen implemented) I'm excited to see better error messages and even more efficient implementations going forward. I would love to see a C++11 aware DSEL that works with user-defined literals instead of just operator overloading.

The future of C++ sounds uncomfortably exciting and I'm looking forward to the next generation of programmers who will grow up and use C++11 as the only C++ they ever knew and hope they come up with even more awesome things that the current experts already have come up with. I'm willing to be a spectator and perhaps even a participant in this future.

Challenges of Reality

It's fun to dream and hope for better tools and better libraries. Unfortunately there's the problem of reality: how to we get to that point?

My suggestion is three-fold:

Use the language. Find out whether you're noticing that there's certain new idioms that you're discovering. See whether using the new features to solve old problems makes the old problem easier to solve. Migrate your code to use the new features in places where it makes sense (or even where they don't just to have a feel for what works and what doesn't).
Share your findings. Let the world know how you find the features. It's not always going to be good -- there are still some WTF's with certain changes to the language around some things. If you make your problems known to more people the more chances other people will help you out. I'm impressed with the communities that have been built around Ruby, Python, and JavaScript -- they aren't afraid to share their experiences and findings with the whole world. C++11 is so awesome that it deserves a community around it too in my opinion.
Get involved in the process. You say you would like a networking library? Let your friendly neighborhood C++ committee member know. We're here to listen and take suggestions. If you point us to the direction where you think C++ should go, we're all ears. I for one would like first-class reflection/introspection both statically and at runtime in standard C++ -- it's not too hard to get your proposal heard and even championed by people that are on the committee. Let's all work together to make the next version of C++ even more awesome than the current one.

I hope you'll join me in getting the word out that C++11 is worth the effort and is ultimately a better C++ for everyone. I'd gladly love to hear what you think.

This is the first post since March 22, 2012 -- I'm sorry for being too busy to share my thoughts with the larger community. I can't promise much but I definitely will strive to write more about C++.

The C++ Network Library: New Release and Updates

noreply@blogger.com (Dean Berris) — Wed, 21 Mar 2012 13:32:00 +0000

After a bit of development (and a little stagnation) since the 0.9.2 release, I'm proud to announce that cpp-netlib 0.9.3 is now available for general availability. This release will be supported for one year until March 30, 2013. You can download the official packages from the Github project downloads. This release was managed by Glyn Matthews and there are a number of important changes in 0.9.3 that addresses issues with 0.9.2. There are a number of bugs fixed and lots of simplifications internally.

Here's a quick run-down of what has changed between 0.9.2 and 0.9.3:

Static libraries for URI and HTTP Client connections. This dramatically reduces the compile-time costs of using the HTTP client and the URI parsers.
A new interface for building URIs. The URI builder follows the directive style that's used in the request and response objects.
Fixes to C++0x/C++11-supporting compilers (MSVC 2010 in particular).
Updates to the MIME parser implementation.

There are a few issues that are being worked on and we're planning on doing a 0.9.4 release or a patch-set for 0.9.3. If you do find issues we encourage you to file them on the official project issue tracker.

The next major release we're planning will feature the move away from and removing support for the header-only implementation. There's also a discussion on the mailing list about what the focus for the development of the library should be moving forward. If you're interested in the topic, you can checkout the discussion on the mailing list.

Major changes are going to come from my personal fork of the project. I've just recently completed the refactoring of the HTTP client implementation to make it easier to maintain and extend. In lieu of header-only template metaprogramming, I've made changes to use traditional polymorphism in places where they make sense. This allows us to maintain a stable interface against which other developers can program against to implement many new techniques that users can choose to customize the behavior of the HTTP client.

We're still in the process of planning whether we're going to release a 0.10 which contains these changes or whether we're going to straight to 1.0. C++11 support is also something the community is looking to jump to. We'd love to hear what the users think, so feel free to post comments on this post or better yet join the mailing list to get involved in the discussion.

In the meantime we're hoping to get more users and more contributors to the project. Although we are too late for applying to the Google Summer of Code effort, we are ramping up our efforts to get more contributors involved in the project. I'm personally putting in my 20% time to make sure that I can mentor new developers and make myself available to allow people who want to contribute to answer questions on the mailing list. I'm also going to be hosting regular hang-outs as well for live face-to-face interaction with users and developers in the community.

The goal of cpp-netlib has definitely changed now from something that's just a header-only modern C++ networking library to making modern C++ network programming easy for everyone. We encourage users and developers alike to raise ideas and questions to the mailing list so that we can as a community make network programming in C++ easier and more accessible to more people. It is also with this goal that we're hoping to get a proposal to the ISO C++ Committee for a high-level HTTP client interface accepted.

In behalf of the C++ Network Library group of maintainers and community members, we look forward to getting more contributors and users this year and in the coming years. We're all committed to achieving the new goals of the project and look forward to your feedback and patronage.

Cheers!

Post-Kona Mailing and More Libraries

noreply@blogger.com (Dean Berris) — Tue, 06 Mar 2012 14:08:00 +0000

Beach, laptop, flip-flops, and C++. Kona, Hawai'i.

It took a while for me to get to writing up my experiences on the first ever ISO C++ Committee meeting that I attended in Kona, Hawai'i. There are around 70 attendees and I'm one of the few first timers to the meeting. What happens when you get a group of C++ developers, implementers, and language lawyers in the same venue for 5 days to talk about C++? Lots of fun and generally productive discussions. There's a lot of new things to look forward to in the upcoming versions of the C++ standard. Here's a high-level overview of what transpired in Kona and what I personally think would be nice to see in the next version(s) of C++.

I mention versions because the committee has decided that we'll be working on getting a short-term update to the standard tentatively shooting for 2017 (C++1y or C++17) and a long-term update aiming for 2022 (C++22). The Evolution Working Group (EWG), the working group whose focus is in the addition of new features, was where the Rich Pointers (N3340) paper was discussed. I spent a lot of my time attending the EWG meetings hearing about what the potential focus for C++17 and C++22 may be.

There's a lot of discussions but the big-ticket items I've listed down below:

Static If -- a means of simpler compile-time programming with a more natural and familiar syntax. One motivating factor and use-case is a replacement for SFINAE tricks with templates.
Concepts (better known as Constrained Templates) -- a means of enforcing syntactic and more importantly semantic requirements for types in templates. There's a lot of synergy and potential for integration of the work between the proponents of Static If and Constrained Templates so it would be nice to see great minds get together in finally making this happen in C++.
Modules -- not dynamic libraries (although those would be nice to have too) but more on logical grouping of symbols for replacing header files and #include. There are two different proposals on Modules for C++ and there's a Study Group formed to address this particular issue. This looks to be one of the things that might be making it into C++17.
Software Transactional Memory -- an update on the state of the art and progress makes it very encouraging. It's very exciting technology that potentially makes a lot of concurrency problems more manageable going forward. With hardware assistance apparently coming soon (this year or early next year from the major vendors) this proves to be something worth keeping an eye and an ear on.
Reflection -- this is where Rich Pointers fall under along with a proposal for static reflection. The general feedback is that a more holistic view of reflection be taken and a more thorough presentation on the matter would be appreciated in upcoming meetings.

These things are still in the proposal presentation stage where interest is gauged and potential enhancements and issues are explored. The upcoming meetings will see more fleshed out proposals and potentially more in-depth discussion of the details once the committee decides on the one big thing to include in C++17 and what two other big things to try to get into C++22.

In other potentially more important news, the C++ committee has issued a Call for Papers for more libraries to bring into the standard library specification. Here's a quote from the announcement:

The C++ standards committee is soliciting proposals for additional library components. Such proposals can range from small (addition of a single signature to an existing library) to large (something bigger than any current standard library component).

In a change from past practice, the committee is processing multiple library work items in parallel, and any resulting domain specific technical reports will ship when ready rather than waiting for completion of single large technical report.

This call for proposals in open ended; there is no cutoff date. As a practical matter, the committee is early in the post-C++11 revision cycle, and so the next year is a particularly good window of opportunity for library proposals.

The committee welcomes proposals with or without formal standard wording (often known as "standardese"). This is a change from past practice, and is intended to encourage library developers to step forward with proposals.

The motivating factor here is to dramatically increase the number of libraries that come with a C++ implementation. The committee is favoring library proposals that already intend to standardize existing practice. This means we might be able to see ICU's features made part of the C++ standard with a different (more consistent with the current standard library style) interface. We might also see Boost.Asio's interface and specification as something that comes standard to standards-compliant C++ implementations.

In a similar vein I'm going to also be working with the Networking Study Group (SG4) that will focus on getting a Technical Report (TR) out that focuses on networking libraries. The hope is that we can cover various layers of the networking stack represented well in the C++ standard.

There are now 4 study groups, and SD-3 details what these are and what the Study Groups mean to achieve. Here's the list copied verbatim from the mailing:

WG21 currently has four SGs and appointed chairs.

SG1: Concurrency and Parallelism (Hans Boehm)
SG2: Modules (Doug Gregor)
SG3: File System (Beman Dawes)
SG4: Networking (Kyle Kloepper)

An SG is not a formal project, and so ISO does not formally appoint an Editor. However, some SGs will be working on a document, and those SGs should havesomeone who acts as editor to maintain the working draft of that document, and publish each draft revision as an N-numbered paper. The SG can approve changes to its own working draft, but the understanding is that the final product will be a submission to a full WG21 meeting for further processing.

The Library Working Group (LWG) is working on multiple concurrent efforts to publish logically-grouped TRs that focus on a distinct areas. The idea is that these TRs can be published when they're ready independent of whether or how the C++17 effort is going. Of course changes to the working draft may affect the interfaces of the libraries being defined in the TRs, the agreement has been largely that any new features that go into C++17 will be incremental and non-intrusive to C++11, which means the LWG can focus on TRs that use C++11 as the core language in the libraries specified in the TR.

Call to Action

So this post is not just about a report, this is also a call to action. Let us (the C++ committee members) know what libraries you'd like to see in every C++ implementation, what's lacking, and what you want us to address in the standard library and the language. It was mentioned several times in the meetings that the committee needs more user feedback and representation from actual users, not just the experts. We'd like to hear what people think should be the direction of C++ and how we can make lives of users easier in terms of libraries and actual language features.

Thanks very much and I do hope this update helps.

Rich Pointers: Frequently Asked Questions

noreply@blogger.com (Dean Berris) — Mon, 23 Jan 2012 07:34:00 +0000

Towards the end of last week the Pre-Kona ISO C++ Committee Mailing was released. This featured a few very important papers for the Evolution Working Group (EWG) regarding Concepts (N3351), Modules (N3347), and Static If (N3322). Rich Pointers (written by yours truly along with comments from Lawrence Crowl and Matt Austern) is N3340 got a few questions asked from the EWG mailing list and in Google+ (as well as through emails directly to me). Even though I have responded to these questions as thoroughly as I could, I've found it prudent to post an FAQ to make the issues raised a little more exposure.

Why a new pointer type? Why not just extend RTTI?

If you've read through the whole paper, it doesn't do much to say why a new pointer type is being proposed as a viable solution to the different motivations for the paper (see my other post about rich pointers for background). It does mention some hints that bear repeating and pointing out:

We don't want users to pay for what they don't use -- more precisely, we only want the compiler to generate the type descriptors for types that are used through rich pointers.
We don't want to break existing code that rely on normal pointer semantics that's why we didn't go about changing how pointers work. Backward compatibility is important.
We don't want to tack the implementation onto existing runtime type information (RTTI) for backward compatibility. There's code out there that has been built and linked to rely on std::type_info being a certain size and layout -- changing this on newer libraries/binaries will cause ABI compatibility issues that we don't want to introduce.

Although the proposal does hint that it can use RTTI in cases when upgrading from a normal pointer to a rich pointer, this is only going to be true if RTTI was enabled in the compilation of the module in which the type was defined. This cannot be guaranteed and this is by design as well.

Why not just a smart pointer with an assist from the compiler to generate the descriptors?

This may be feasible and as the paper points out there is a way of achieving this. Unfortunately the following cannot be guaranteed by a library-only implementation:

Deterministic runtime registration of types from libraries. There's currently no way to ensure that a rich_ptr will be able to contain the pointer to the correct type especially if the source is a normal pointer.
Deterministic invalidation of already-registered types. There's no cross-platform and standard way for registering new types in an application's runtime that also guarantees that there's only one definition of a type, and that objects of that single type can be referred to using the correct pointer type.
Clear upgrade path from one version of a type to the latest version of the type. Similar to the above point, there's no clear and cross-platform way of doing this.
Automatic and selective generation of complete types and those types it depends on in its definition. Even with a compiler assist, making this happen with a normal template class and the type for this given template is far too much work for the compiler as opposed to having the compiler treat rich pointers as a special type of pointer.

There are also many more reasons which involve the definition of a cross-platform runtime interface, which will definitely vary from platform to platform -- even though that part will definitely be a library, a library-implemented type cannot get too much special treatment that violates the basic rules of C++.

Why use the '%' character and not another character (like '^' or '@')?

There's really no strong reason to use %, but there are reasons to reject '^' and '@'. The caret character ('^') although ideal to use similar to how Pascal pointers work, unfortunately is already used by Microsoft in their proprietary C++/CLI implementation. Not chosing the caret was an obvious decision to avoid the potential clash and objections with this extension to standard C++. The at character ('@') although has been used in the Ruby programming language, isn't a special symbol (nor a valid symbol) in C++ -- making @ a valid operator character opens up potential major surgery for current compilers and the grammar. In contrast, the percent character ('%') is already a valid symbol and operator in C++ that has a binary operator counterpart -- this is symmetric with the asterisk ('*') which is both a unary and binary operator version.

I personally would also like to think that the percent symbol denotes the presence of two things on different sides of the common line -- a pointer that refers to an object (top circle) and the type descriptor (bottom circle). Of course this is a mnemonic device and symbolism that's very far from existing practice and usage of the percent operator (as a modulo operator), I'd like to think I tried to give it some meaning that's relevant. ;)

Is there an implementation available?

Short answer: Not yet.

Long answer: If the committee agrees that this is something worth exploring, I'm very much poised to go ahead and implement it as an extension to the Clang compiler front-end. At this time the proposal is meant to spark discussion as to whether it's something the members of the committee (and the users) will want to see in the language. Practically speaking, there's no point in going forward with an implementation if people don't think the original idea is worth implementing in the language anyway.

Conclusions

The Kona meeting is exactly two weeks away and I'm going to be preparing more materials for presentation among the members of the committee. I'll also be reading the other papers that are in the pre-Kona meeting to prepare for the discussions that I definitely want to be contributing to in the meetings. You can also expect that I will be writing about the meeting afterwards and will be keeping notes as the meetings go on.

I'm definitely looking forward to the meeting -- if you're going, I do hope to see you there. If you're not going and would like to send your feedback on any of the papers and aren't on the committee mailing lists, please feel free to send in your comments either through the comment form or through email.

Thanks for reading this far and I do hope this helps!

C++ Extensions: Rich Pointers

noreply@blogger.com (Dean Berris) — Fri, 13 Jan 2012 14:35:00 +0000

Some time towards the end of 2011 I officially became a member of the ISO C++ Committee as part of Google's delegation. Around the same time I wrote about a few extensions to C++ that I wanted to propose or at least help champion to the next version of C++. The Kona meeting is just around the corner and I'm just about ready to make a case for a specific new extension to C++ that I wrote a paper about called rich pointers. This post is about the motivations behind the extension.

It's 2012 and I've pretty much started it off with a very ambitious project: to add a new pointer type to C++ with something called "rich pointers". This project aims to achieve the following:

Define a new pointer type that conveys semantics different from the primitive pointers inherited from C.
Associates an object's type to the pointer instead of the object.
Provide a richer reflection and runtime introspection facility for C++ types referred through rich pointers.
Attempts to standardize a runtime interface for managing types and these rich pointers.

Here are a few motivating examples for why a new pointer type would be required instead of shoehorning the reflection information into current RTTI implementations.

Suppose we want to implement serialization on any type of object that has a member function called "serialize", how would we do this from a function that's provided through a DLL?
We're implementing a distributed service and we'd like to use Google Protocol Buffers as the data interchange object but also want to render the information using the Qt GUI framework on the client side. How do we do this with as little overhead as possible and without having to adapt the type of the protocol buffer into something that the Qt GUI framework will understand?
Let's say we want a generic registry of handlers implemented as dynamically linked modules for a server application and we want to minimize the requirement for downtime of the application as we add new functionality to it. How do we ensure that the new types introduced in the dynamically linked modules are dealt with accordingly by the server application?
Suppose we're hosting a just-in-time (JIT) compiler that can compile C++ code in a long-running application. We're doing this so that we can reprogram parts of the application according to runtime constraints/inputs (imagine a query optimizer for SQL databases or for regular expressions) for efficiency concerns. How do we allow the new types defined in the runtime-compiled C++ code to be used in the host application in a type-safe manner?

These are just a number of real-world requirements that rich pointers intends to address and enable in a standardized manner. To get this done I'd have to convince the committee that:

Adding a new pointer type unique to C++ and having rich runtime type information is necessary to enable seamless solutions for these kinds of problems.
Requiring the compiler to generate type information for types referred to via rich pointers is doable and will not lead to overly bloated binaries.
Attaching type information to a pointer instead of the object makes sense.
Current RTTI mechanisms are inadequate to achieve this kind of reflection and runtime type introspection.
Implementing this feature will not require too much heavy lifting for implementors.

This paper also implies and relies on the following being true:

That C++ modules will define how dynamically and statically shared C++ libraries will be defined (maintaining backward compatibility to existing linkers).
That the committee will agree to have a standardized API for accessing details about an application's runtime (for type registration, function registration, certain system characteristics of the machine and implementation, etc.).
Runtime introspection is something we'd like to support at the language level because everyone else who needs this functionality has implemented a library that provides this feature in an inadequate and clumsy manner.

In the coming days I will be updating the post with a link to the actual document hosted in the ISO C++ Committee mailings page. I certainly look forward to shedding more details about this project and update everyone of the progress of this proposal and perhaps post some insider notes about the upcoming meeting in Kona, Hawaii.

Thank you and I do hope to be able to give more details soon.

Update: Moved!

noreply@blogger.com (Dean Berris) — Sun, 04 Dec 2011 09:35:00 +0000

The Wordpress experiment has ended and I'm coming back home to Blogger. I'll be posting more here and would be encouraging others who would like to contribute to let me know so that I can add you as a contributor to the site. Here are a few more updates on what's been going on from my end.

I'm now officially a part of the ISO C++ Committee as an alternate to the Google representation! I'll be going to the Kona meeting in February to present a few papers if things go well.
The C++ Network Library project is progressing well. Check out the progress over at https://github.com/deanberris/cpp-netlib for more information.
I'll be travelling again to the Philippines for the holidays, so there would be a lull as I spend most of the holidays offline. Before the three week break I'll be posting more from here.

Thanks very much for the patience with the moves. I'll try to get more content in to make up for the disruption.

Cheers

New C++ Standard, New Focus

noreply@blogger.com (Dean Berris) — Mon, 31 Oct 2011 12:18:00 +0000

Now that the new version of C++, C++11 is out in the wild as an ISO C++ Standard, there's a renewed focus on getting applications, tools, and books up to date with the new standard. The leaders of the ISO Committee have weighed in on the latest versions of the language. There's Herb Sutter writing about C++11, there's Bjarne Stroustrup's C++11 FAQ, and the upcoming C++Now! conference that's going to be held in Aspen Colorado. Clang is racing to get C++11 support into the stable release of the compiler, and GCC is humming along nicely with it as is Microsoft's Visual C++. Instead of write about any specific things, I would lend my opinion on what needs to be the new focus not for C++11 the language, but the supporting toolchains surrounding this new language -- and the version that's coming after this one.

Having been exposed to a number of programming languages and paradigms I'd like to think that I've seen quite a bit of the programming world. There's the dynamic programming languages like Python, Ruby, Lua, JavaScript, and its ilk, and there's the relatively new static languages like Go, Scala, Haskell, Erlang, among others. Systems programming and application programming have new challenges -- concurrency, consistency, safety, etc. -- that need attention from both the programming language and the libraries that come with the programming environment. There are new platforms that are designed to perform more and more demanding computations in very meager amounts of available resources. The web is ever-growing and there's a lot more things web applications need to do both at the front and back-end of the system. And there are yet a lot more applications to be thought in embedded and large-scale systems that current programming languages have yet to even think about (autonomous computing anyone?).

With these in mind, I think C++ has to evolve into a direction that makes simple things easy and hard things possible. There was a time that C++ was the language of choice because much of the infrastructure that's available now was not available then -- you kinda had to write your own tools back in the day, and C++ was the easier programming language to use for large systems. Unfortunately now, even with the new features that makes it easier to write programs in C++ using the new modern style, a big percentage of the programs still turn out to be either glue code or repetitive boilerplate. If you're writing asynchronous systems, they're made easy with lambda's, efficient containers, consistent memory model, and concurrency primitives (or at least standard libraries) but you end up with a lot of boilerplate because there aren't any sane standard HTTP implementations -- interfacing with C libraries will be a lot more painful with C++11 considering how much of the C++ standard defines things outside of the C standard's scope. C++11 is awesome if you're starting from scratch but when you get to the part where you have to tie libraries together, C++ is still an expert's language.

I'm hinting at two things here that C++ should address in my opinion going forward: a standardized way of defining modules for easier composition and higher level standard libraries with reference implementations. Let's tackle each in order here.

First, standardized modules. The reason programming languages like Go, Haskell, and Erlang are getting traction for back-end programming (in the case of Erlang, it's not really a "systems programming" language unlike Go and Haskell) is because it has a simple and standardized module mechanism. Packaging code in modules that can be programmed against (ala Pascal Units) in terms of interfaces makes modular development really easy. Right now all the module implementations in C++ are vestiges to how the linkers are implemented and implementation specific. There's a lot of details here that's worth fleshing out at a later time (there's a lot of content for the blog) but needless to say it should be simple to import modules into a compilation unit for later linkage -- and this should be made standard across implementations.

One of the problems that standardized modules will solve is that of the definition and implementation of the standard library. The STL is awesome as it is at the moment, but it can be a lot more awesome if there's a defined module layout that will be the same across all platforms and that people can implement shrink-wrapped modules that play well with many other modules in a non-conflicting manner. C++ already has namespaces and it's a mechanical manner of tying namespaces to modules. Extending a module should then be a matter of dependency and linkage. Without going much into details, C++ needs to solve the problem of coupling. Many things need to be thought out, like how the runtime environment deals with types and namespaces and how linkage and application binary interfaces can be made consistent without sacrificing performance.

The second thing that does need attention is that runtime environment. I'm not just talking about garbage collection and I'm not talking about having a virtual machine. What I'm talking about is the management of runtime elements, things like memory, I/O, concurrency, among other things. Although it is not impossible to write OS kernels in C++, the C++ runtime is mostly still the C runtime (i.e. too limited) to make things like a global optimizing allocation mechanism, thread scheduling, implementing co-routines, and magical things that makes systems programming interesting very painful to do. A standardized runtime environment can make writing stand-alone systems in C++ easier and less reliant on the C runtime. This can also finally solve things like module initialization (when dynamically loading things), static initialization, memory management, and even reflection and runtime introspection features that can be implemented in a standard manner.

The runtime should be as programmable as the Lisp environments, as minimalistic as the Lua runtime, and as performance-sensitive as the C runtime. It should also be as open-ended as possible, still allowing for implementation-specific extensions but defining a standard core that others can build upon. There are a lot of things to flesh out in this idea but it's something that needs to happen to make C++ a language that's as viable for systems programming as well as application development with all the goodness of the dynamic languages and the safety of static languages. Short of defining a virtual machine, a standard runtime implementation should definitely be considered for the next version of C++. I don't like virtual machines as much as the next person but having a well-defined runtime environment will make systems programming a lot less hack-laden as it currently is.

With these in mind I'm putting some effort into writing up papers to that effect so that hopefully these ideas get at least thought about in the next ISO C++ Standards Committee meeting in Kona Hawaii. Hopefully I can get comments and ideas and a discussion going with esteemed colleagues and users from around the world who might have an interest in joining an effort to make this a reality. Let's make C++ even better by bringing good ideas from other programming languages into C++ without sacrificing the performance and flexibility afforded by our favorite programming language.

Along with this effort I'm going to work on getting the C++ Network Library reviewed for inclusion into Boost and hopefully be a reference implementation for a high-level network library to be made part of the C++ standard. I think it's about time that C++ gets modern programming features and Web-aware libraries part of the standard to make it more attractive to developers who might want to develop high performance solutions in C++ for the Internet. I'll be supporting the standardization of the Asynchronous I/O library that's already part of Boost so that it can be the standard way of programming asynchronous I/O in C++ and made available to more people who do use C++ for network programming.

It's an exciting time ahead for C++ and I fully intend to help influence that future too.

Minus the Header-Only Dogma

noreply@blogger.com (Dean Berris) — Thu, 25 Aug 2011 14:25:00 +0000

I've been working on the C++ Network Library for a few years already total but have been on-and-off with it until recently. One of the first decisions that I made in implementing the library was to try and do a header-only approach to implementing a network library implementation. At first it made perfect sense -- you get the compiler to optimize as much of the code as it can while being able to expose a very friendly and expressive interface. When it was merely a handful of header files and there was just the notion of a message and the HTTP client, all was fine. I was perfectly happy showing off really simple but powerful use-cases where you'd want a C++ application to be able to make HTTP calls and get the contents without having any knowledge of how networks worked or how to program sockets on either Windows or UNIX-like systems. Until I got ambitious and it became a real project that people wanted to use -- then reality became real.

It's not a secret that I like template metaprogramming. If you do a search on the site on 'metaprogramming' you'll find a number of posts dating back to around 2007 detailing the cool things that can be done with C++ templates. I've seen it first hand make really crazy performance improvements in production while I was still working at Friendster. Until recently the server I re-wrote with almost the whole network layer being a huge template metaprogram was humming along nicely. Unfortunately people didn't have the luxury of three things that I had at that time:

They don't have the same level of expertise with network programming. A vast majority of people who want to use cpp-netlib are people who don't want to implement the HTTP protocol from scratch or use a third-party library that was written in C. That said they didn't have the patience or experience necessarily to debug faults in cpp-netlib's implementation nor did they have the motivation to extend cpp-netlib.

They don't have time to wait for a simple HTTP client example to compile. It just took too much time to try to iterate with cpp-netlib since it was header-only and exercised the compilers too much for something that should be relatively simple.

They didn't buy into the header-only dogma. Much of the world was perfectly fine with linking to external libraries either dynamically or statically.

After getting feedback from BoostCon 2010 about the cost of header-only libraries, I then thought long and hard about how to introduce externally linked libraries into cpp-netlib. What came after that was a process of thinking about how to actually get a header-only library to a point where it can support both a header-only build and a non-header-only build. I imagine there aren't a lot of people who would encounter this problem -- if it's even a problem at all -- but I wanted to point out a couple of things that I stumbled upon while "meditating" on this issue.

Make it Optional

One of the first things I had to do was to make the header-only version of the library optional. The rationale for this is pretty simple: compiling template metaprograms costs compiler time and effort in terms of debugging, so if you want to pay this cost you explicitly say it. This makes casual users happier by not having to pay the heavy cost of the compile time every time. This makes sense if you're in development mode and doing lots of builds often enough to not want to wait too long for each build to finish.

In the C and C++ world, the way to make things optional at the source level is to make it a preprocessor macro. This is the first thing that made itself into the 0.9.0 release, the BOOST_NETWORK_NO_LIB macro which when defined makes cpp-netlib a header-only library. This made cpp-netlib a not-header-only-lib by default. This approach allows me (and other developers) to compile the heavy-compile-cost parts of the library into externally statically linked libs once and link them in at the end. The first things that went into the external lib were the Boost.Spirit based parsers. This had cut down the cost of compiling the tests dramatically -- as much as 60% reduction in compile-time.

The approach has its merits because it makes users who want to use the library in header-only mode explicitly ask for it. One of the reasons people would want to do this is to enable further inlining that the compiler can do in header-only mode when building releases. During debug or development mode, linking an external library is just fine if you have a reasonably fast linker.

Abstract Appropriately

The good thing about being able to move things out into compiled external libraries is that you can program against interfaces. This means abstract base types and later binding. This is completely opposite to the original idea of using static dispatch for most everything that I wanted to achieve when I set out to implement cpp-netlib. When I looked at it long and hard, I saw that trying to do everything through static dispatch isn't so scalable in a number of ways:

Extensibility was static. This meant that if you wanted to add an extension or alternate implementation, you'd have to hook into the library's static dispatch mechanisms. This is not trivial and only the bravest template metaprogrammers will even hazard a try to extend the library through the template dispatch mechanisms.

Horizontally scaling was much impossible. This means adding alternate implementations was severely limited by the static nature of the bindings. For example, I had an idea of being able to statically determine which implementation of a particular string processing routine would be called given a tag type. I can see now that doing things this way makes adding support for things like CString (from Microsoft), QtString (from Qt), Boehm's cord/rope, and countless other string types tragically harder than if I use the normal generic programming approach instead of using type tag dispatch.

Debugging is a pain. Not only does it take long to compile things, it takes long for the compiler to see/say where things went wrong. This makes debugging the library implementation hopelessly dependent on my patience on a given day. There were times when I just literally had no patience for compiler errors that I just left the code alone for days without touching it.

Now with the help of dynamic polymorphism I'm able to appropriately abstract some parts of the library that do need the runtime polymorphism -- things like abstracting a connection whether it's SSL or a normal TCP/IP connection, and maybe later on support other underlying protocols like SCTP, and maybe even UDP if there's a use case for HTTP over UDP. This also makes testing the library much easier than it originally was.

Doing it this way also allows me and other implementors to leverage the best of both the static and the dynamic worlds in the same library. One of the ongoing refactoring efforts that I'm undertaking is the development of a mini HTTP REST Service framework which should come out early September. This would allow for developing a service in a traditional OOP fashion while having the efficiency of a header-only network implementation supporting the HTTP server implementation. The same technique allows us to expose a Boost.Parameter based API on the HTTP Client side and hiding much (if not all) of the internal client implementation in a Pimpl idiom -- so the client implementation can remain template-heavy internally and can be compiled once, and linked to many times.

Revelations

I personally am relieved that I now don't feel like I need to fight and feel the pain of being lonely in a template metaprogramming world. Although there are lots of template metaprogrammers now thanks to the excellent books and Boost C++ Library mailing lists, there is still a lot to be done to get template metaprogramming in the mainstream. I now think that the better strategy for getting it out in the hands of more people is by using the techniques and leveraging its power not just as a power tool but also as an adjunct to more traditional object-oriented programming techniques. I am going to be the first to point out that having been exposed to both OOP and template metaprogramming along with functional programming and generic programming that the synergies between/among these paradigms in C++ is what makes it such a unique and powerful programming language.

Hopefully those who are interested in joining and contributing to an open source library that explores the power and synergies of the different programming paradigms in C++ will join the C++ Network Library project. I now host a weekly Google+ Hangout every Wednesday 11am Sydney time (GMT +11) to discuss with developers and users of the library the direction of the project, what's going on, and how the project can proceed. I also regularly post updates to the C++ Network Library mailing list.

For the next release the target is to simplify much of the internal implementations to make it easier to test and understand for new-comers to the project, and along with that define how others might be able to take on more work in the library ranging from: testing, reporting issues, issue triage, updating the website, implementing new protocols, among many of the community/administrative work that comes with an open source project. We're targeting September 2 as the release date for version 0.10 which should feature better compile times and much better testing.

You can add me on Google+ -- introduce yourself and let me know if you're interested in getting notified/invited to the regular cpp-netlib hangout.

C++0x Is Official: Unanimously

noreply@blogger.com (Dean Berris) — Sat, 13 Aug 2011 08:45:00 +0000

Herb Sutter, one of the rock stars of the C++ development world has posted his thoughts and shares with the world that C++0x has been approved unanimously as an International Standard. From his post:

The final ISO ballot on C++0x closed on Wednesday, and we just received the results: Unanimous approval.

The next revision of C++ that we’ve been calling “C++0x” is now an International Standard! Geneva will take several months to publish it, but we hope it will be published well within the year, and then we’ll be able to call it “C++11.”

I want to extend my thanks again to Bjarne Stroustrup for sharing his work with the world and continuing to help move it forward, and to all of the participants whose hard work went into achieving this important milestone in the history of a great language. Thanks!

-- Herb Sutter on Sutter's Mill

Now the real work begins in moving the world from C++98/C++03 to the more sane and much more wonderful C++0x.

Finally us C++ programmers can start looking forward to modern C++ features that makes programming with C++ just that more pleasurable and powerful.

Here's to a brighter future with C++0x!

cpp-netlib 0.9.1 Released!

noreply@blogger.com (Dean Berris) — Thu, 04 Aug 2011 04:36:00 +0000

Today marks another important day in my calendar as I'm proud to announce the general availability of cpp-netlib 0.9.1. You can now download the source packages from the Github download page. If you want to see what's new you can head on to the online documentation page for what's new in 0.9.1. This release fixes a number of issues with 0.9.0 so if you're still using 0.9.0 you might want to upgrade to 0.9.1. That's not the only reason 0.9.1 is good though, and the rest of this post is about giving a little bit more insight into what's new and what's coming in the coming months for the C++ Network Library project.

For all intents and purposes, 0.9.1 should really be version 1.0 -- but there are a few things holding it back from becoming ready for prime-time as a 1.0 version. Here's a list of things that are in there that makes the release extra special:

Support for streaming body callback handler in HTTP client requests. This means you can perform HTTP client requests and pass a callback function that gets called every time a chunk of data is available from the connection. This allows you to handle larger response bodies without having to fit all the data in memory. This is important for implementing downloaders that need to save data straight to a file or in general when you just want to not have to wait for all the data to be put in memory before processing it. You can find out more about this by reading the updated HTTP Client reference documentation.

Default HTTP client is now asynchronous. This release switches the default HTTP client implementation defined at boost::network::http::client to use the asynchronous features. There are no changes required to the original code that used the default HTTP client implementation, but linking to the new version may require you to link in Boost.Thread. You can easily change the default tag by overriding the BOOST_NETWORK_HTT_DEFAULT_TAG macro at compile time or before you include any cpp-netlib header.

Tons of bug fixes from 0.9.0. You can find the list of issues that have been closed on the GitHub issues page. Needless to say the amount of changes that went into the library between 0.9.0 and 0.9.1 are pretty substantial that would have merited a major number advance, but there are more good things in store to wait for to get into 1.0.

The list above is pretty packed but I'd also like to update everyone on some good news regarding the cpp-netlib project. Needless to say they should be treated as great news but I'm not one for exaggeration. Here are a few updates:

There's now a git mirror up on Google Code Project Hosting. At some point in the future -- depending on how large the community grows and/or whether there's popular support for it -- the Google Code Project Hosting site may just become the de-facto development repository of the project. You can find the Google Code Project here.

I'm now officially working on cpp-netlib as my 20% project in Google! This means I can now devote a measurable amount of time and effort into making sure the project succeeds. This doesn't change the project goals and ideals but hopefully it does change the pace of development as I get more effort into it.

I'm looking to expand the developer pool by offering to mentor interested developers who want to learn the techniques and be part of the project. Since cpp-netlib is not my 100% time project in Google I'm not going to be able to do all the fun work that still needs to be done to make cpp-netlib better. Good news is that I'm making myself available to help mentor developers to get up to speed on the project and have more people eventually able to contribute to the effort.

I've found a willing review manager for cpp-netlib's eventual review for inclusion in the Boost C++ Library. It's going to be just a matter of time now before the review is scheduled and I'm able to set time aside to get the review done and over with. The project will continue to evolve and develop even outside of Boost following the Boost.Spirit and Boost.Asio model and hopefully get more people contributing to Boost through the cpp-netlib project.

If you're interested in the mentoring, please join the cpp-netlib Google group and introduce yourself. Let everyone on the list know what you want to learn and what part of the library or project you want to be involved in. We're currently looking for people who are willing to:

Test the library on the major development platforms. Right now we're only concentrating on Linux and Mac with GCC and Clang, and we're looking for people willing to test on Windows and/or other platforms.

Update the look and layout of the website. This doesn't require much C++ knowledge but should be proficient with HTML, CSS, and has better design aesthetics than I do.

Read/update the documentation. Feedback on the documentation and what's lacking are very important.

Implement new features and/or refactor the internals of the library. This part will be involved and is where I would be doing most of the mentoring and hand-holding as much as necessary.

Thanks for reading so far down and I do hope you'll help spread the word about the new release.

Cheers from down under!

What You've Missed With C++: 2011 So Far

noreply@blogger.com (Dean Berris) — Tue, 05 Jul 2011 13:18:00 +0000

The past few months have been a jam-packed with a few developments in terms of C++. Not only did C++0x get the almost unanimous nod of the industry to be good enough to be called C++11 but there's also a few stories that have come out that need some attention too. There was the Google findings which show that C++ is by far the best performing programming language (if you have enough programmers that know the language well enough to wield it proficiently) and C++ AMP which aims to bring massively parallel C++ programming closer to the mainstream by Microsoft. Also, last May BoostCon 2011 was held with a pretty good lineup of talks that Boost users would be most appreciative of. It seems like 2011 is the year that C++ dusts off its boots and starts marching on again to the top of the programming language hill. Or is it?

For a programming language that started in the 80's and is still being used by and large successfully by the tech companies that matter (I'm talking Microsoft, Google, Apple, Adobe, Facebook) you'd think C++ would enjoy a revered spot in the programming language hierarchy. Unfortunately if you look around and ask programmers what they think about C++ more or less you're going to find people who will have a number of bad things to say about it. Why does a programming language so hated be so effective and still be in use in industry?

First of all I think it all comes from a fundamental fact that good programmers develop good solutions. If you put a good tool (in this case a programming language) in the hands of a good craftsman (in this case a programmer) then the task you're set to accomplish would be accomplished with good enough results. You can still get a good outcome if you have bad tools and a great craftsman or if you have great tools and a bad craftsman. But the best results come from good tools in the hands of great craftsmen. Cathedrals, airplanes, ships, bridges, and all the infrastructure that last are built by craftsmen that had access to the best tools they had at the time they were built. I would argue that the software that stands the test of time (those running in Google or Facebook's servers, Microsoft's software, Adobe's software, Apple's products) are those written by great craftsmen regardless with good tools.

Second reason I can think of is the flexibility of the programming language. There are not a lot of programming languages where you can use it to write device drivers, operating systems, web application systems, realtime monitoring systems, embedded software, desktop applications, imaging applications, and even mobile applications by basing on just a single standard. There's a good reason why the flexible "general purpose" programming languages enjoy the most usage among the craftsmen that know their craft. It just so happens that C++ has enough flexibility to allow pretty much anything to be done with it with features that support the development of easily scalable software -- and I mean scale as in the capacity to grow the software as far as complexity and design is concerned. With a limited programming language that wasn't focused on "usability" in its current version (C++03) it's managed to be one of the most effective tools used by a lot of programmers in industry. Now with most of the shortcomings addressed in the upcoming version of the language, the flexible programming language just got a lot more flexible and usable.

The last thing that I think makes C++ a very compelling programming language to use today and going forward is: for any interesting problem to be solved at scale, you're going to need a programming language that gives you performance, flexibility, and scalability without sacrificing any of these three. I have yet to come across a programming language that comes near the performance profile of C++, supports as many programming paradigms that C++ does, and allows software construction and design to scale favorably as much as C++ does. There are some interesting programming languages that have come out recently but I have still yet to see one that actually can seriously replace C++ as the programming language for "serious software".

The reason C++ is still significant today is because the problems worth solving require as much of the machine as possible while being efficient in doing so. Sure you can use any programming language to write a web application, but if you're trying to handle billions of queries per minute and would want to get the most out of the machines you have at your disposal you're going to use one that allows you to scale your development effort and your application without sacrificing performance. The problems that are worth solving now require massive amounts of computing power and it's actually an advantage to have your application as efficient as you can make it. This is where C++ as a programming language shines.

With C++ in the hands of experienced and good craftsmen, solutions to problems worth solving are efficient, scalable, and maintainable. With a new version coming up I'd say this decade is the one where C++ is more relevant than ever.

cpp-netlib 0.9.0 Released!

noreply@blogger.com (Dean Berris) — Mon, 18 Apr 2011 13:19:00 +0000

After the one-week beta period for the 0.9.0 release, I'm happy to announce that the official 0.9.0 release is now available for general availability at the Github project downloads page. This 0.9.0 release is the first version of the library to be submitted for review to the Boost C++ Library project. Most changes that happened from the beta to the official release involve adding debugging information to critical parts of the server and client implementations as well as documentation and Boost.Build system updates.

This release series will be supported until April 2012, which means all new development for the 1.0 line will happen on a different branch and will continue to contain new implementations and interface-breaking changes over the 0.9.x line. Currently the roadmap for the library is still in flux, but if you're interested in putting in feature requests you can file them through the Github project issues page. Soon enough a wiki page containing the roadmap for the 1.0 release with updated milestones will be released for public availability.

Help Wanted

At this time I would also like to announce that I am currently looking for help in various aspects of the project namely:

Website

Documentation

Testing

Development

This is mostly due to the fact that I've just recently started work at Google Australia, and may not be able to work on the library on a full-time basis. I'll transition more into a maintainer role and will still be doing most of the development on high-impact items.

New Protocols

The 1.0 line will get the HTTP implementation into a state of stability, which should allow other protocols to enter the library. Those that are highest on the list of priority protocols are:

SMTP

XMPP

The focus for 1.0 will be to implement other protocols so that the generic parts of the implementation can be lifted. Also, better installation and integration support for users that might not like the header-only nature of the library along with better customization points for tweaking implementation details would have to be made available as well.

The 1.0 line will be dealing with making the library more flexible in the facilities it uses to achieve the designs it uses. There are a number of other performance and usability points which will have to be addressed to make cpp-netlib adoption in more projects easier.

The Road Ahead

Along with a stability push, the goal for the cpp-netlib project is to attempt to build a community which mimics the Boost C++ Library process, with a more open development/contribution model. The C++ Network Library Project will be started officially with moves to make collaboration and further development of the code a more inviting and more open process. Once these preparations are in place, I'll be announcing what the plan for the foreseeable future of the library and the project will be like.

At the moment the library is looking to adopt other projects that would be useful in the context of a modern C++ library that deals with network-related programming. In the end the goal is to ultimately allow for developing both server and client services that talk in different (and maybe custom) protocols easily with the C++ Network Library. Along with this is an attempt to get the library into Boost as well as maybe part of the Standard C++ Libraries.

It's good times ahead -- I hope to be able to deliver more good news with cpp-netlib development from here down under.

Memcache++ 0.12.2 Out, Last Release by Me

noreply@blogger.com (Dean Berris) — Mon, 11 Apr 2011 06:26:00 +0000

I have just finished tagging and uploading the 0.12.2 release of memcache++ which is a maintenance release that includes one diff away from 0.12.1 -- you can download the library here. This release is the final release that I will be managing and starting now I am on the lookout for others who would be interested in continuing the development of the library. I will still host the project on Github and I will still be available to answer questions but I will no longer be actively developing and adding new features to the library. If you or someone you know might be interested in taking over the project please let me know and join the user group as well. I had fun developing the library when I was still at Friendster and even though it's a little hard to acknowledge that a project is being retired, but my focus now has been put elsewhere -- and more recently the C++ Network Library is really what I'd like to focus on. Thank you to everyone who's checked the code out and those that are using it in production even up to this day.