Distributed Reflections of the Third Kind

Limbo Overview

2010-09-16T10:44:00.001-07:00

It was already sitting on my hard drive, partially completed, but then here it is. An overview of Limbo, the expression language used in Preon to define dependencies between bound fields:

Limbo Overview

Preon repository cloned

2010-08-23T08:37:00.000-07:00

This weekend, I found out that the Preon repository wasn't available for anonymous access. That's just awful. In order to make sure everybody has access, I also started pushing to Github. That way, you can get the latest versions, if you need to. If you agree with me that this is suboptimal, please raise a vote here.

Update: the issue with the Codehaus repository has been fixed. I will continue pushing to both Github and Codehaus for a while, but the blessed repository is the one on Codehaus.

Preon updates

2010-08-08T07:12:00.000-07:00

During the last couple of days, I continued working on Preon again. I can easily see it fit in some of the NoSQL solutions that are gaining traction these days.

However, in order to make things a little bit more manageable, I will no longer be able to retain the compatibility with previous versions, so version 1.1 will probably be renamed to version 2.0 before its release.

Maven groupId changed to org.codehaus.preon.
Limbo was moved into the Preon codebase, and renamed to preon-el.
Some things I am thinking about: Codec interface is probably going to be changed into a class incorporating all (convenience) methods for encoding and decoding. As a consequence, I will need a new name for the abstraction currently called Codec. It hasn't been done yet, but I think the API gets better if I move away from having to use Codecs all over the place. (See Joshua Bloch's bumper sticker API design.

Supporting Maps in Preon

2010-06-20T08:31:00.000-07:00

Just a quick write-up of some ideas that are floating around in my head since some time. I lately tried to use Preon to decode BSON files. Now, BSON is just a binary representation of a data model that is a superset of JSON. And if you think about JSON, then you can probably figure out what the most likely Java representation of the data model would be: that would clearly be a map, potentially containing other maps. However, Preon doesn't support Maps yet. So I started to wonder how to weave that in.

I started thinking about different ways how Maps could be encoded on disk, and realized that it most cases they are actually encoded as lists of entries. It is not necessarily the only or most efficient way, but it's the way things are done in many cases. Let's take JSON as an example:

{
  "firstName":"John",
  "lastName":"Smith","
  "age":25
}

As you can see, it's actually not the internal representation of the map that is encoded here. In that case, you would most likely see a binary tree type of representation. Instead, the entire associative array is encoded as list of key-value pairs.

BSON is no exception to that rule. So we need a way to decode a Map by decoding individual entries and store them in the result Map, one by one.

I have been hacking around for a little while, and got this working:

public static class Sample {

    @BoundList(type= SampleEntry.class)
    public Map<String,Integer> data;

}

public static class SampleEntry implements Map.Entry<String,Integer> {

    @BoundNumber(size="8")
    private int keyLength;

    @BoundString(size = "keyLength")
    private String key;

    @BoundNumber(size="8")
    private Integer value;

    public String getKey() {
        return key;
    }

    public Integer getValue() {
        return value;
    }

    public Integer setValue(Integer value) {
        Integer originalValue = value;
        this.value = value;
        return originalValue;
    }
}

In the example above, the Sample class has an attribute called 'data', mapping Strings to Integer values. The @BoundList annotation might seem a little out of place on this element, at first. However, in this particular case, the key value pairs are essentially encoded as a sequence of entries. So, in that sense, it might not be all that out of place at all.

The new CodecFactory that as added to Preon will be triggered by Map type of attributes that have an @BoundList annotation with the type of element set to a subtype of Map.Entry. Behind the scenes, it will first use the @BoundList annotation to create a Codec capable of decoding the List, and then copy each of the key value pairs into the result Map.

The critical question of course is how the Codec responsible for reading all entries should know how to decode these entries. The answer is simple, and pretty obvious: you just do it the regular way, by making sure there is a class representing the entry with the Preon annotations set correctly. This gives you the flexibility to encode sequences of map entries any way you like. In the sample case, every entry contains of a unsigned byte stating the size of the key (in number of characters) followed by the actual key encoded in US-ASCII, followed by the value encoded as an unsigned byte.

All in all, it seems to be work pretty well. There are a couple of gotchas though. When implementing this example, it turned out that we still suffer from the PREON-9 bug. That's annoying. As a result, there are some things that cannot show yet.

Preon Going Codehaus

2010-03-19T05:54:00.000-07:00

There have been quite a lot of changes in Preon during the last couple of weeks. Not so much in terms of source code, but rather in where we are planning to go (project hosting) and who's going to work on it.

First of all, I'm happy to say that Samuli Saarinen joined the project. He already fixed numerous bugs, so I'm happy to welcome him aboard.

Next to that, we also decided to move Preon to Codehaus. This is going to affect Preon in a number of ways:

All project resources will be hosted at a single location. So no more searching on Kenai, Sourceforge, Atlassian, etc. (I'm sure that's a relief for many people.)
I expect the package and Maven artifact names will change at some point to reflect the new location.
We might move from Subversion to Git.
The existing mailing lists will probably go away to be replaced by Codehaus mailing lists.
We will have a continuous integration build hosted at Codehaus.
The SNAPSHOT repositories will move somewhere else.
Limbo and Pecia might be moved back into Preon, to ease further development.

There is a chance that you will be affected by these changes. We will try to do our best to make it as least disruptive as possible though. So don't expect any big changes before an upfront announcement.

In summary, I just want to say that I think this is all excellent news. Sure, it will take some time before it stabilizes, but once the entire migration is done, we will have an environment that will help us to grow Preon in a healthy way in the future.

OOPSLA slides uploaded

2010-01-21T08:49:00.000-08:00

OOPSLA Talk on Preon

View more documents from Wilfred Springer.

The Size of the Slice

2010-01-11T23:04:00.000-08:00

During the last couple of months, I have been going back and forward on how to deal with slicing while encoding. If it comes to clearing your mind, nothing helps as explaining it to others (well, perhaps there are some things that help even better), so here goes nothing:

Slicing

First question: what is slicing anyway? In Preon, slicing is basically creating a partial view on a BitBuffer. If you happen to have a BitBuffer with 512 bits, and you slice it, what you get is a new BitBuffer. If your current position is on bit 9, and you create a slice of 16 bits (hardly realistic, but it's just easier to work with smaller numbers), then you get a BitBuffer that contains 16 bits. On position 0, it will have the value of position 9 in the original BitBuffer, and so on.

Use of Slices

Slices are useful in situations in 1) you have a collection of things and 2) you can't predict how many items it has, but you can predict how much space it occupies in total. Basically, if you first slice the BitBuffer, you can basically just keep reading until you the end of the BitBuffer. In reality, the BitBuffer can contain way more data, but you can never read beyond the end of the slice.

The Size of the Slice

There are essentially two different ways to define the size of the slice: the size of the slice could either be a static value, or it can be defined in terms of data read upstream. As an example of the second case: there might be a situation in which the size of the slice is defined by a value encoded as an int right in front of the start of the actual slice.

Ways to Define a Slice

Out of the box, Preon provides two different ways to define a slice. There's a '@Slice' annotation, and a '@LengthPrefix annotation. The second annotation existed before the first one, and is not as powerful as the first one. So while I was thinking about all of this, I did consider dropping @LengthPrefix entirely.

So how does it work?

The @Slice annotatation is defined like this: @Slice(size="..."). In this case the size is a Limbo expression, so it can either be a static value or an expression that incorporates data read upstream.

The @LengthPrefix annotation is defined like this: @LengthPrefix(size="...", endian={ByteOrder}). In this case, the size is not the size of the actual slice, but the size of a value represented as an integer that holds the size of the slice. (See the example below.)

So, you could say that with @Slice you can do everything that can be done with @LengthPrefix and a little more, but not the other way around. So far, there doesn't seem to be a reason to keep @LengthPrefix any longer; it seems we can just deprecate it. But there is a little more to that. Read on.

Encoding

Encoding a slice is in a way more complicated than decoding it. Suppose that the slice is defined to have a fixed size. In that case, the size of the data to be encoded within that slice should not be bigger than the size of the slice itself.

In case the size of the slice is a function of data read upstream, then it becomes even more complicated. If the data within the slice requires a certain amount of bits, then that function needs to result in that number of bits. Consequently, the variables used in that function need to have proper values, but since those variables are read from bits upstream, those bits need to have the proper value as well. So, the space required by downstream data requires upstream bits to have a certain value.

It's imported to note that this is a specific case of some challenge that has been reported before, in a previous post. The @LengthPrefix shines a different light on it though.

The following two snippets are essentially the same:

// Example 1:
@BoundNumber(size="8") int size;
@Slice(size="size") @BoundList byte[];

// Example 2:
@LengthPrefix(size="8") @BoundList byte[];

The Codecs generated by Preon will be almost identical. The main difference is that, in the first case, the size attribute is something that is accessible by the client. In the second case, it is data that will be read from the BitBuffer, but it will never be exposed to the Preon user.

Because of the information hiding in case of @LengthPrefix, it will be easier to preserve the dependency between the size attribute and the actual size of the slice. Encoding the slice in case of @LengthPrefix could work like this:

Determine if the size of data structure from the slice can be determined before it is encoded. In case it can, simply write the size of the slice to the BitChannel first, and then continue writing the data structure that needs to be encoded.
Otherwise, write the entire data structure to a temporary BitChannel backed by an in memory buffer. Then, before flushing the contents of this buffer into the target BitChannel, determine the required size of the buffer, and write that value to the target channel first. Then write the contents of the temporary buffer into the BitChannel.

So, this will guarantee that the encoded representation will always be correct. Something that is currently impossible when using @Slice. For @Slice, we it is possible to write to a temporary buffer, and determine the required size, but then there is not much we can do with that information, since it's data that could have been changed. (And having the ability to preserve dependencies between fields that can be changed is currently deliberately left out of the Preon roadmap.)

Summary

Trying to summarize my thoughts. Although there is some benefit in using @LengthPrefix when encoding, I'm still considering taking it out. @LenghtPrefix is only going to be a benefit in a few cases, whereas the problem that manifests itself when using @Slice exposes a challenge that 1) is way harder to solve, but also 2) much more awarding to solve.

If I take that route, then there will not be a reason for having the ability to write the slice to a temporary buffer to determine it's size. It's possible to write to a BitChannel directly, and then throw an exception if the number of bits required exceeds the size defined by the slice.

Preon on JRockit

2009-10-28T19:07:00.001-07:00

After my Preon talk earlier today, Alex Buckley warned me that reflection is not going to guarantee that the fields will always be returned in the same order. Preon currently relies on the fields to be returned in the order in which they were defined, to some extent.

That is, if this is the data structure you defined:

public class Image {
@Bound int width;
@Bound int height;
}

Then Preon will expect that the reflection API will also return the fields in this order.

I think I sort of knew this in the back of my head, but it never failed any of the tests, so I stopped worrying about it. So, the question is, should I start worrying now? It's clearly undesirable to depend on some coincidental properties of a Java VM, but are there actually VMs that will return a different field ordering? With IBMs VM being based on Sun's VM, it's unlikely they will differ. So that leaves us with JRockit.

In the end, I figured it would be wise to have an automated test that at least guarantees that the current setup also wors on JRockit. It doesn't solve the problem, but it does provide some degree of guarantee that the problem will usually not manifest itself. And it turns out, the test succeeds.

Maven Archetype for Simple Preon Project

2009-10-15T12:17:00.000-07:00

It's there. Just got to http://preon.flotsam.nl/getting-started.html and follow the instructions. This is by far the easiest way to get started.

Preon Encoding Roadmap

2009-10-13T12:14:00.001-07:00

In my previous post, I highlighted some of the challenges and questions to be answered before Preon can be made to support encoding as well. The main problem is not so much that it is hard to understand how the data should be encoded. That's all pretty clear. The meta data gathered by Preon provides sufficient detail to make that fairly easy to do. No, the real problem is in preserving consistency.

If you really try to imagine the ultimate goal, then it's immediately clear that it will be close to impossible to get there in one step. So what do you do? You break things up in phases. That's what I tried to do tonight:

The picture above depicts the different stages for getting closer to where I want Preon to be. Read it from the bottom up.

Phase 1: Writing all data to a stream

Currently, the Codec only defines a decode operation. We clearly need an encode operation as well. The decode operation is going to take the object passed in and write it to the output channel. Period. By adding this operation, you will be able to encode data, however when you actually make changes to the data, you're basically on your own.

Here are some of the complications that could occur if you change the data:

In case Preon loaded an instance of its own LazyLoadingList, then trying to modify that list is going to throw exceptions.
If you replace the value of an attribute that is used in Limbo expressions, there's not only a chance that you write corrupted data, but there's also a chance that you will not even be able to continue to read the data any longer. Remember, Preon is decoding lazily. It might not even have read the data that you are about to write. By changing the attributes that Preon uses in calculating the starting point of a section of data to read, you might get in trouble.

Phase 2: Binding to public accessors

We need to be able to understand when code outside of Preon is making changes to an object that was loaded using Preon, because if it does, then we can no longer afford to drop a cached version of that instance. However, currently Preon only binds to fields, and not to bean-type accessors. It's going to be close to impossible to track changes to those private fields, but it will be possible to track changes to those fields if Preon binds to the accessor methods rather than the fields. So, we will probably need that feature to be there.

Phase 3: Copy on change

Maybe this is not the right term. In some cases, all we need to do is make sure that we hold on to a cached copy indefinitely, until data is persisted. In other cases, we will need to make sure that we actually replace the entire copy of an existing object and replace it by something else.

Phase 4: Consistency checks while writing

As I said, making sure that we preserve consistency over the entire file is going to be one of the biggest challenges. The previous step has made sure that we can actually change the file and write it again, but it's not going to guarantee that whatever is going to be written is consistent. For that, we need something else. This phase is about adding a feature that will check consistency while writing.

(Consequently, data early in the file will always prevail over data later on in the file. If the file first contains an integer denoting the size of a list following, and that value is greater than the actual size of the list when the list needs to be written, the list either needs to be truncated or grow, or an exception will have to be thrown. This phase is about making sure an exception is thrown.)

Phase 5: Rewrite only if required

In many cases, it's not going to be required to first load data into an object and then write it to output again. If the object encoded did not change, then we can just stream the from the original source. (In this case, the BitBuffer.) This is - hopefully - going to be an optimization that is going to pay off big time.

Phase 6: Autocorrecting

In phase 4, I already said that there will be cases in which you ideally want the list of elements to grow or shrink if the attribute that denotes the number of item in the list is updated. This phase is about considering solutions like these. It will probably be quite hard, if not impossible at all, but it's worth to take it into consideration.

Phase 7, 8, 9, 10, ...

O man, if only I would have time.

Inserting Data Using Preon

2009-10-11T12:45:00.000-07:00

I started work on supporting encoding data the other day. There are quite a few complications, and I noticed that just jotting them down is not going to help or anyone else to come up with some solutions. First I need a better description of the problem.

So I decided I would add a couple of examples that are slightly more detailed, in order to be able to highlight the problems, and explain the solutions that I'm currently thinking of.

The first example is about a simple image format. (I'm just so grateful we have images. Where would I have been without them. They always seem to serve as be the best examples in cases like this.) It only defines a slot for the number of pixels and then defines the color values for those pixels.

The example above illustrates an instance of this model, based on a 400 pixel image. The orange region highlights the first pixel value in the byte stream. It shows the relation between the object instance and the corresponding location in the byte stream.

Now, if the above would have been decoded using Preon, then the code would have probably looked a little like this:

class Image {
@Bound int nrPixels;
@BoundList(size="nrPixels") List<Color> pixels;
}

class Color {
@Bound byte red;
@Bound byte green;
@Bound byte blue;
}

It clearly shows the link between the nrPixels attribute and the pixels attribute. Now, suppose that Preon would allow you to modify the data and then encode the data. What would that look like in this case? The image below is trying to capture it:

So, only inserting a single pixel would already require quite some changes. In fact, it would basically require a rewrite of the entire byte stream.

The nrPixels attribute would change from 0x190 to 0x191. (See pink region.)

The entire pixel array would have to be shifted to the right in order to allocate space for the new pixel to be inserted.

That new pixel would have to be inserted. (See orange region.)

The question that I'm trying to answer myself is how Preon should deal with this. There are a couple of challenges, given the way Preon works.

If the List pixel 'array' would have supported the insert operation, then the 'nrPixels' attribute is not automatically updated if you would insert a new pixel.

Changing the nrPixels attribute would not automatically allocate a new element in the list of pixels.

Given the fact that Preon writes to private fields of objects, it's going to be pretty hard to notice updates to any of these fields at all.

Now, that's quite a few challenges. My first thoughts on this:

Since the List pixels attribute will cause Preon to insert an instance of its own List class, it should be possible to intercept changes to that list. If the size of the list is based on an attribute value read upstream, then it should be possible to update that value with every change to the pixels list.

However, this would only work in case that attribute value is only defining an encoding/decoding attribute for that specific List. If there are other data elements read downstream that are based on that same attribute value, it would be impossible to update it automatically.

In cases like these, Preon could still be able to check the dependencies as a constraint and abort the encoding process if the constraints are violated.

Changing Preon Modules

2009-10-10T02:56:00.001-07:00

I know this could be painful, but I will go ahead and do it anyway. The last couple of weeks I spent some of my sparetime to work on a capability for outputting data from Preon. And I planned to support streaming output first. Now, that requires a couple of new abstractions that I first added to a module called preon-bitchannel. But I think I'm going to collapse preon-bitbuffer and preon-bitchannel into one new module: preon-io.

The two abstractions for reading bit'stream' type of content are just so intertwined that it hardly makes sense to have two different modules for it. For instance, one of the capabilities that will turn up in BitBuffer is the capability to 'stream' content directly to an output channel.

Now, why is this important? It's a little hard to explain, but I will give it a go. First of all, you should be aware of the fact that Preon is loading data lazily. That means that - in some cases - rather than loading data aggressively, it will keep a pointer to a position in the BitBuffer, and only start reading from that position if the application is actually accessing that data. Now, if you would be able to update data, but you would leave that specific segment untouched, then encoding would be as simple as writing from the BitBuffer directly to the output channel. And an implementation could do that extremely efficiently if the buffer and channel abstractions are designed to work with each other.

Any way, a code example probably would help a lot in this case. Sit tight. Something is coming.

Parameterized Byte Order

2009-09-26T00:46:00.001-07:00

I started checking out Lua byte code a minute ago. (Thanks to Stackoverflow for finding its location.) Now, I haven't started reading all of the details yet, but there is one section that already is proving to be troublesome.

1 byte Endianness flag (default 1)

• 0=big endian, 1=little endian

1 byte Size of int (in bytes) (default 4)

1 byte Size of size_t (in bytes) (default 4)

1 byte Size of Instruction (in bytes) (default 4)

It's the byte order. It means that - if I would try to decode Lua bytecode using Preon - I would have to make some changes to the annotations.

Currently, the annotations for decoding numeric data look like this:


@BoundNumber(byteOrder=LittleEndian)
@BoundNumber(byteOrder=BigEndian)

... but now that byteOrder needs to be defined by something read before. I guess there are a couple of ways to deal with it:

Option 1: Have the ability to change the 'global' default byte order setting as a consequence of reading a value

Now, global doesn't really need to be global. It might be good enough if it would be a default for the current 'lexical scope'. So basically, if you would read a block of data that would indicate a change in the byte order, then Preon could basically respect those changes for anything read within that block.

Option 2: Have the ability to refer to ByteOrder enum values read previously from @BoundNumber annotations

Now this is harder. Not undoable, because this is exactly what you can already do for reading the number of bits for instance:


@BoundNumber(size="header.numberOfBits")

For byte order, it could be similar:


@BoundNumber(byteOrder="header.byteOrder")

However, this would mean that instead of typing byteOrder as an enum value of type ByteOrder, it would be a String. That would be less compile time safe. And it would require an incredible number of complex pointers pointing to that header attribute.

But maybe there is a third option. Add a new ByteOrder enum value called Default, basically indicating that the ByteOrder is defined somewhere else. Currently, the default is LittleEndian. But maybe that default needs to be set as an annotation as well.

Anyway, there is a lot to think about. I will see if I can answer this one today.

Preon Encoding Started

2009-09-22T11:59:00.000-07:00

I started work on supporting Preon the other day. One of the first things I will have to do is to define the interface that will be used to write data. Picking that interface is not trivial. One of the ideas that I suggested last year is not to try to keep Preon symmetrical; that is, it probably makes sense not to preserve the random access type of access to the data when writing data, but write to a streaming type of interface instead.

There are a couple of reasons why that would make sense:

In-place editing is just going to be way to painful: if you would a couple of elements to a List, you would need to move blocks of data around. I can hardly imagine what that would look like, and it's probably not going to be very efficient.
Now, regardless on what the repercussions would be in terms of moving blocks of data around in memory and on disk, it would also imply changing all pointers that still have a reference to other places in that buffer of bytes. So, suppose that you would have two lazy loading lists after each other. Then in order to be able to lookup elements of the second list on the fly, that list will keep a reference to its starting point. That reference will be based on the offset of the start of its content relatively to the beginning of the file. Now, inserting elements in the first list also needs to result in updating those references.
Preon aims to be threadsafe. That complicates matters further. As a result of adding an element to a list, thousands of pointers might potentially require an update, but from the perspective of the client consistency would need to be preserved during that transaction. This calls for an STM type-of solution.

I'm still not sure if it would be totally ridiculous to even support in-place editing, but it's just going to be to hard at this stage. Having the ability to 1) process changes, and 2) to persist all of those changes to disk in a single snapshot would already be quite a challenge, but it would be an awesome feature.

Tomorrow, I will try to add some thoughts on the shape of the interface that I am currently considering.

Preon Presentation at OOPSLA 2009

2009-08-30T06:39:00.000-07:00

I will present Preon at OOPSLA 2009. That's going to be fun. I have been wondering if it would be possible to squeeze in encoding before October 25.

Getting the Ethernet Frame

2009-08-13T01:36:00.000-07:00

A couple of days ago, I blogged about using Preon to take a Snoop file apart. It's interesting that it actually worked, but then again, it hardly retrieved any valuable data. Given that most of the data floating around on our networks is actually ethernet based, it would be nice if Preon would actually be able to take apart the data sections as well.

Obviously, I wouldn't be telling you all of this, if Preon would not be able to do that. So let me show you how. The full code is listed below.

The important bits are in line 60.

The code that we used to have there was this:


@Slice(size="(packetRecordLength - 24) * 8")
@BoundList(size="includedLength")
private byte[] packetData;

In the new version of SnoopFile, that changed into:


@Slice(size = "(packetRecordLength - 24) * 8")
@BoundObject(selectFrom = @Choices(alternatives =
 @Choice(condition = "outer.header.datalinkType==DatalinkType.ETHERNET", type = EthernetFrame.class))
)
private Object packetData;

Basically, instead of decoding the packet data into a byte array, it's now immediately turned into an object. An object that actually has the relevant data already decoded. Now, don't be alarmed by the type of packetData. In the new code it's declared to be of type java.lang.Object. That's just to make sure that it will be able to cover for all of the types of packet data that you could possibly encounter. In the common case, it will be ethernet frames. But Snoop is able to capture data for other data link types as well, and it's fairly unlikely that these different types of packet data would have commonalities to be captured in a base class.

So, even thought he type of packetData is java.lang.Object, Preon will actually decode packet data in objects of a particular type, based on the data link type. The selectFrom attribute on @BoundObject contains the rules for picking the appropriate type of object. Currently, it only covers for Ethernet packages. It basically reads: if the data link type read as part of the header is Ethernet, then decode the data into an instance of EthernetFrame.

The EthernetFrame itself is defined like this:


public static class EthernetFrame {

   @BoundList(size = "6")
   private byte[] destinationAddress;
  
   @BoundList(size = "6")
   private byte[] sourceAddress;

   @BoundNumber(size = "16")
   private int type;

   @BoundList(size="outer.includedLength - (6 + 6 + 2)")
   private byte[] data;

}

Not much to explain here. It's pretty much straightforward Preon code. Note that the number of bytes is calculated based on an attribute of the packet record. Since the EthernetFrame is read as part of the PacketRecord, you *can* actually refer to the 'outer' context, and reference attributes of that outer context.


public class SnoopFile {

@Bound
private FileHeader header;

@BoundList(type = PacketRecord.class)
private List<PacketRecord> records;

public FileHeader getHeader() {
 return header;
}

public List<PacketRecord> getRecords() {
 return records;
}

public static class FileHeader {

 @BoundBuffer(match = { 0x73, 0x6e, 0x6f, 0x6f, 0x70, 0x00, 0x00, 0x00 })
 private byte[] identificationPattern;

 @BoundNumber(byteOrder = ByteOrder.BigEndian)
 private int versionNumber;

 @BoundNumber(size = "32", byteOrder = ByteOrder.BigEndian)
 private DatalinkType datalinkType;

 public int getVersionNumber() {
  return versionNumber;
 }

 public DatalinkType getDatalinkType() {
  return datalinkType;
 }

}

@ImportStatic(DatalinkType.class)
public static class PacketRecord {

 @BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
 private long originalLength;

 @BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
 private long includedLength;

 @BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
 private long packetRecordLength;

 @BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
 private long cumulativeDrops;

 @BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
 private long timestampSeconds;

 @BoundNumber(byteOrder = ByteOrder.BigEndian, size = "32")
 private long timestampMicroseconds;

 @Slice(size = "(packetRecordLength - 24) * 8")
 @BoundObject(selectFrom = @Choices(alternatives = @Choice(condition = "outer.header.datalinkType==DatalinkType.ETHERNET", type = EthernetFrame.class)))
 private Object packetData;

 public long getOriginalLength() {
  return originalLength;
 }

 public long getIncludedLength() {
  return includedLength;
 }

 public long getPacketRecordLength() {
  return packetRecordLength;
 }

 public long getCumulativeDrops() {
  return cumulativeDrops;
 }

 public long getTimestampSeconds() {
  return timestampSeconds;
 }

 public long getTimestampMicroseconds() {
  return timestampMicroseconds;
 }

 public Object getPacketData() {
  return packetData;
 }

 public static class EthernetFrame {

  @BoundList(size = "6")
  private byte[] destinationAddress;

  @BoundList(size = "6")
  private byte[] sourceAddress;

  @BoundNumber(size = "16")
  private int type;

  @BoundList(size="outer.includedLength - (6 + 6 + 2)")
  private byte[] data;

  public String getDestinationAddress() {
   return render(destinationAddress);
  }
 
  public String getSourceAddress() {
   return render(sourceAddress);
  }
 
  private String render(byte[] address) {
   StringBuilder builder = new StringBuilder();
   for (int i = 0; i < address.length; i++) {
    if (i != 0) {
     builder.append(':');
    }
    builder.append(Integer.toHexString(0xff&address[i]));
   }
   return builder.toString();
  }

 }

}

public static enum DatalinkType {

 @BoundEnumOption(0)
 IEEE_802_3,

 @BoundEnumOption(1)
 IEEE_802_4_TOKEN_BUS,

 @BoundEnumOption(2)
 IEEE_802_5_TOKEN_RING,

 @BoundEnumOption(3)
 IEEE_802_6_METRO_NET,

 @BoundEnumOption(4)
 ETHERNET,

 @BoundEnumOption(5)
 HLDC,

 @BoundEnumOption(6)
 CHARACTER_SYNCHRONOUS,

 @BoundEnumOption(7)
 IBM_CHANNEL_TO_CHANNEL,

 @BoundEnumOption(8)
 FDDI,

 @BoundEnumOption(9)
 OTHER,

 UNASSIGNED
}

}

Enums in Preon

2009-08-12T07:10:00.001-07:00

You might have noticed in my previous entry that I introduced a new annotation to map enum values to their encoded representation. I already thought about that a couple of times, but never found time to do it. Now I introduced it, I notice that there is actually something missing here. Question is: will it be possible to introduce it without overhauling the entire framework? I think it is possible.

The Problem

So what is actually missing here? Well, with the latest addition of @BoundEnumOption, you can easily map an enum value to its encoded representation. That's nice; now you can have a typed collection of enum values instead of arbitrary integer values.

However, there is a gotcha. What will you will see a lot is that the enum value read influences the way you are supposed to read the remainder of the file. In the snoop example, this is no different. Currently, the example doesn't do anything with the actual data packets, but if it did, then it would have to decode it differently depending on the value of datalinkType.

In the past, you could easily use @If or @Switch and refer to the value of datalinkType, as long as it was an integer, String or boolean. However, now you need to compare that value to another enum value, and that's something that wasn't supported yet.

The Requirements

We want to be able to do something like this:


@BoundNumber(size="8")
private DatalinkType datalinkType;
…
@If("datalinkType == DatalinkType.ETHERNET")

This means that Preon needs to be able to compare enums, and that it also needs to be able to resolve references to enum values.

The Challenge

Having the ability to compare enums doesn't look like a huge problem. In fact, I just wrote an additional test in LimboTest, and it seems to be working fine:


    public void testComparingEnums() {
        EasyMock.expect(defs.getType("a")).andReturn(Direction.class).anyTimes();
        EasyMock.expect(defs.getType("b")).andReturn(Direction.class).anyTimes();
        EasyMock.expect(resolver.get("a")).andReturn(Direction.LEFT).anyTimes();
        EasyMock.expect(resolver.get("b")).andReturn(Direction.RIGHT).anyTimes();
        EasyMock.replay(defs, resolver);
        assertTrue(condition(context, resolver, "a == a"));
        assertFalse(condition(context, resolver, "a == b"));
        assertTrue(condition(context, resolver, "b == b"));
    }

    public static enum Direction {
        LEFT,
        RIGHT;
    }

The difficulty is somewhere else: it's in resolving the enum values. Problem is Direction.LEFT is not something Limbo is capable of dereferencing; it's just not in lexical scope. So, the question is, do we need another version of Limbo now? One that is capable of resolving enum value references? The good new is: no, you don't.

The reason for that is that Limbo is extensible. It defines something called a ReferenceContext that needs to be implemented to allow Limbo to execute in a context.

Let's look at an example to clear things up a little. Let's assume that you want to evaluate "foo.bar + 3". When you feed this expression to Limbo, it will first try to see if this expression makes sense at all. So it will check if it *will* actually be able to resolve something named 'foo', and if that thing has an attribute called 'bar'. If it doesn't then it will just fail directly. Now, in order to check if there is something called 'foo' in its context, it will simply call selectAttribute("foo") on the current ReferenceContext. If that ReferenceContext decides that something like that is referenceable, it will return a Reference object representing the "foo" reference. Next, Limbo will check if there is actually an attribute "bar" on the type of object referenced by the Reference just created. If so, then it will return a Reference to that property.

The important thing to note here is that - whatever Limbo is trying to do - all requests always hit the ReferenceContext first. Once the References are 'linked', it will call resolve(…) on the references to resolve the values on an instance of that context.

So, if we want to put the enums into lexical scope, all we need to do is rewrite the ReferenceContext. And our ReferenceContext is defined by Preon, so we don't have to rewrite Limbo; we just need to write a small adjustment in Preon, in order to integrate Limbo slightly differently.

The Solution

The actual code required to make this work is a little bit more than I'm willing to list here. (It turns out you need a decorator for RefenceContext, and then two additional Reference implementations. One to refer to Classes, and one to reference static fields on a class. The solution works in the same way as outlined above. The only thing that you need to add is an @ImportStatic annotation, at the top of the class from which you are referencing the enum values:


@ImportStatic(DatalinkType.class)
public class SnoopFile {

  @BoundNumber(size="8")
  private DatalinkType datalinkType;
  …
  @If("datalinkType == DatalinkType.ETHERNET")

Snoop Using Preon

2009-08-10T04:03:00.000-07:00

Just before my holiday started, Adriaan Thomas sent me an email, asking me about using Preon for decoding Solaris snoop files. That sounded like an interesting challenge. So, this weekend, in my spare hours, I wrote some code to see what that would look like. The code required is given below, and based on RFC 1761.

Now, if you want to decode a snoop file, that's pretty easy: you just create the Codec, and then pass that Codec to one of Codecs decode operations:

Codec<SnoopFile> codec = Codecs.createCodec(SnoopFile.class);
SnoopFile capture = Codecs.decode(codec, ...);

This is the definition of the SnoopFile class.

public class SnoopFile {

@Bound
private FileHeader header;

@BoundList(type=PacketRecord.class)
private List<PacketRecord> records;

public FileHeader getHeader() {
return header;
}

public List<PacketRecord> getRecords() {
return records;
}

public static class FileHeader {

@BoundBuffer(match = { 0x73, 0x6e, 0x6f, 0x6f, 0x70, 0x00, 0x00, 0x00 })
private byte[] identificationPattern;

@BoundNumber(byteOrder=ByteOrder.BigEndian)
private int versionNumber;

@BoundNumber(size="32", byteOrder=ByteOrder.BigEndian)
private DatalinkType datalinkType;

public int getVersionNumber() {
 return versionNumber;
}

public DatalinkType getDatalinkType() {
 return datalinkType;
}

}

public static class PacketRecord {

@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long originalLength;

@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long includedLength;

@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long packetRecordLength;

@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long cumulativeDrops;

@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long timestampSeconds;

@BoundNumber(byteOrder = ByteOrder.BigEndian, size="32")
private long timestampMicroseconds;

@Slice(size="(packetRecordLength - 24) * 8")
@BoundList(size="includedLength")
private byte[] packetData;

public long getOriginalLength() {
 return originalLength;
}

public long getIncludedLength() {
 return includedLength;
}

public long getPacketRecordLength() {
 return packetRecordLength;
}

public long getCumulativeDrops() {
 return cumulativeDrops;
}

public long getTimestampSeconds() {
 return timestampSeconds;
}

public long getTimestampMicroseconds() {
 return timestampMicroseconds;
}

public byte[] getPacketData() {
 return packetData;
}

}

public static enum DatalinkType {

@BoundEnumOption(0)
IEEE_802_3,

@BoundEnumOption(1)
IEEE_802_4_TOKEN_BUS,

@BoundEnumOption(2)
IEEE_802_5_TOKEN_RING,

@BoundEnumOption(3)
IEEE_802_6_METRO_NET,

@BoundEnumOption(4)
ETHERNET,

@BoundEnumOption(5)
HLDC,

@BoundEnumOption(6)
CHARACTER_SYNCHRONOUS,

@BoundEnumOption(7)
IBM_CHANNEL_TO_CHANNEL,

@BoundEnumOption(8)
FDDI,

@BoundEnumOption(9)
OTHER,

UNASSIGNED
}

}

Preon Class File Format

2009-03-29T23:58:00.000-07:00

A couple of weeks ago, I blogged a little about Preon's CodecDescriptor, and that it should be changed. This weekend, I completed the new interface, and made sure the entire documentation generation process is now using that interface. I'm quite pleased with the result, although I noticed one area where it can be improved.

The description below is what the new setup produces for the Java Classfile Codec produced from the POJO classes here. Note that the output is a little messy, but that's mostly Blogger.com's style. The actual output looks a little better and can be downloaded here. The same goes for the hyperlinks. The hyperlinks down below don't work. But they do work in the original source document.

Class file

Class file is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Magic

a 32-bit integer value (big endian).

32 (4 bytes)

Minor version

a 16-bit integer value (big endian).

16 (2 bytes)

Major version

a 16-bit integer value (big endian).

16 (2 bytes)

Constant pool count

a 16-bit integer value (big endian).

16 (2 bytes)

Constant pool

a list of elements.

The number of elements in the list is the difference between Constant pool count of Class file and 1.

The particular choice is based on a 8-bit value preceeding the actual encoded value. If 7, then Class cp info will be choosen. If 6, then Double cp info will be choosen. If 9, then Field ref cp info will be choosen. If 4, then Float cp info will be choosen. If 3, then Integer cp info will be choosen. If 11, then Interface method ref cp info will be choosen. If 10, then Method ref cp info will be choosen. If 12, then Name and type cp info will be choosen. If 8, then String cp info will be choosen. If 1, then Utf8 cp info will be choosen.

(unknown)

Access flags

a 16-bit integer value (big endian).

16 (2 bytes)

This class

a 16-bit integer value (big endian).

16 (2 bytes)

Super class

a 16-bit integer value (big endian).

16 (2 bytes)

Interfaces count

a 16-bit integer value (big endian).

16 (2 bytes)

Interfaces

a 32-bit integer value (little endian).

The number of elements in the list is Interfaces count of Class file.

Interfaces count of Class file times 32

Field count

a 16-bit integer value (big endian).

16 (2 bytes)

Fields

Field info.

The number of elements in the list is Field count of Class file.

(unknown)

Method count

a 16-bit integer value (big endian).

16 (2 bytes)

Methods

Method info.

The number of elements in the list is Method count of Class file.

(unknown)

Attribute count

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes

Source file or Deprecated.

The number of elements in the list is Attribute count of Class file.

The particular type of data structure is selected based on the value of 16 leading bits. These bits are interpreted as an unsigned int. The table below lists the conditions, and the data structure assumed when these conditions are met.

Condition	Data structure
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "SourceFile"	Source file
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Deprecated"	Deprecated

(unknown)

Class cp info

Class cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Name index	a 16-bit integer value (big endian).	16 (2 bytes)

Double cp info

Double cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Value	a 64-bit integer value (big endian).	64 (8 bytes)

Field ref cp info

Field ref cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Class index	a 16-bit integer value (big endian).	16 (2 bytes)
Name and type index	a 16-bit integer value (big endian).	16 (2 bytes)

Float cp info

Float cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Value	a 32-bit integer value (big endian).	32 (4 bytes)

Integer cp info

Integer cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Value	a 32-bit integer value (big endian).	32 (4 bytes)

Interface method ref cp info

Interface method ref cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Class index	a 16-bit integer value (big endian).	16 (2 bytes)
Name and type index	a 16-bit integer value (big endian).	16 (2 bytes)

Method ref cp info

Method ref cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Class index	a 16-bit integer value (big endian).	16 (2 bytes)
Name and type index	a 16-bit integer value (big endian).	16 (2 bytes)

Name and type cp info

Name and type cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Name index	a 16-bit integer value (big endian).	16 (2 bytes)
Descriptor index	a 16-bit integer value (big endian).	16 (2 bytes)

String cp info

String cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
String index	a 16-bit integer value (big endian).	16 (2 bytes)

Utf8 cp info

Utf8 cp info is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Length	a 16-bit integer value (big endian).	16 (2 bytes)
Value	A sequence of characters, encoded in ASCII. The number of characters of the string is Length of Utf8 cp info.	8 times Length of Utf8 cp info

Name

Description

Size (in bits)

Length

a 16-bit integer value (big endian).

16 (2 bytes)

Value

A sequence of characters, encoded in ASCII.

The number of characters of the string is Length of Utf8 cp info.

8 times Length of Utf8 cp info

Constant value

Constant value is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Attribute length	a 32-bit integer value (big endian).	32 (4 bytes)
Constant value index	a 16-bit integer value (big endian).	16 (2 bytes)

Synthetic

Synthetic is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Attribute length	a 32-bit integer value (big endian).	32 (4 bytes)

Deprecated

Deprecated is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Attribute length	a 32-bit integer value (big endian).	32 (4 bytes)

Field info

Field info is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Access flags

a 16-bit integer value (big endian).

16 (2 bytes)

Name index

a 16-bit integer value (big endian).

16 (2 bytes)

Descriptor index

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes count

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes

Constant value, Synthetic or Deprecated.

The number of elements in the list is Attributes count of Field info.

Condition	Data structure
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "ConstantValue"	Constant value
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Synthetic"	Synthetic
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Deprecated"	Deprecated

(unknown)

Line number table entry

Line number table entry is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Start pc	a 16-bit integer value (big endian).	16 (2 bytes)
Line number	a 16-bit integer value (big endian).	16 (2 bytes)

Line number table

Line number table is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Attribute length

a 32-bit integer value (big endian).

32 (4 bytes)

Line number table length

a 16-bit integer value (big endian).

16 (2 bytes)

Line number table

Line number table entry.

The number of elements in the list is Line number table length of Line number table.

Line number table length of Line number table times 32

Local variable table entry

Local variable table entry is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Start pc	a 16-bit integer value (big endian).	16 (2 bytes)
Length	a 16-bit integer value (big endian).	16 (2 bytes)
Name index	a 16-bit integer value (big endian).	16 (2 bytes)
Descriptor index	a 16-bit integer value (big endian).	16 (2 bytes)
Index	a 16-bit integer value (big endian).	16 (2 bytes)

Local variable table

Local variable table is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Attribute length

a 32-bit integer value (big endian).

32 (4 bytes)

Local variable table length

a 16-bit integer value (big endian).

16 (2 bytes)

Local variable table

Local variable table entry.

The number of elements in the list is Local variable table length of Local variable table.

Local variable table length of Local variable table times 80

Code

Code is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Attribute length

a 32-bit integer value (big endian).

32 (4 bytes)

Max stack

a 16-bit integer value (big endian).

16 (2 bytes)

Max locals

a 16-bit integer value (big endian).

16 (2 bytes)

Code length

a 32-bit integer value (big endian).

32 (4 bytes)

Code

a 8-bit integer value (little endian).

The number of elements in the list is Code length of Code.

Code length of Code times 8

Exception table length

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes count

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes

Line number table or Local variable table.

The number of elements in the list is Attributes count of Code.

Condition	Data structure
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "LineNumberTable"	Line number table
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "LocalVariableTable"	Local variable table

(unknown)

Exceptions

Exceptions is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Attribute length

a 32-bit integer value (big endian).

32 (4 bytes)

Number of exceptions

a 16-bit integer value (big endian).

16 (2 bytes)

Exception index table

a 32-bit integer value (little endian).

The number of elements in the list is Number of exceptions of Exceptions.

Number of exceptions of Exceptions times 32

Method info

Method info is composed out of several other smaller elements. The table below provides an overview.

Name

Description

Size (in bits)

Access flags

a 16-bit integer value (big endian).

16 (2 bytes)

Name index

a 16-bit integer value (big endian).

16 (2 bytes)

Descriptor index

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes count

a 16-bit integer value (big endian).

16 (2 bytes)

Attributes

a data structure selected from a list of 4.

The number of elements in the list is Attributes count of Method info.

Condition	Data structure
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Code"	Code
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Exceptions"	Exceptions
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Synthetic"	Synthetic
the value (a String) of the nth element of Constant pool of Class file (with n being the difference between the value of the first 16 bits and 1) equals the String "Deprecated"	Deprecated

(unknown)

Source file

Source file is composed out of several other smaller elements. The table below provides an overview.

Name	Description	Size (in bits)
Attribute length	a 32-bit integer value (big endian).	32 (4 bytes)
Source file index	a 16-bit integer value (big endian).	16 (2 bytes)

Preon changes

2009-03-24T00:35:00.000-07:00

The last couple of weeks, I spend most of my commuting time on redesiging the CodecDescriptor in Preon. And I think I'm getting there. Find the results below. I am currently trying to work this into the different existing Codecs. Not even halfway yet, but I think I will be able to get there this week.

Now, one of the important changes that this new CodecDescriptor introduces is the Documenter. I want to rename that into something a little bit more sensible. Anybody with a good idea on that, please send your suggestions. ;-)

The Documenter has been in Pecia for a while, but you could only use it in a paragraph context. I am making changes to Pecia now to use it all over the place. It would be good to have another name for it though. (Did I already mention that?)

Anyway, the Documenter - or whatever its name is going to be - is an object that is capable of rendering itself into a certain context. It works like a callback. If you pass the Documenter to - say - a paragraph, then the paragraph will turn around and ask the Documenter to render itself.


package nl.flotsam.preon;

import nl.flotsam.pecia.Contents;
import nl.flotsam.pecia.Documenter;
import nl.flotsam.pecia.ParaContents;

public interface CodecDescriptor {

   /**
    * An enumeration with different adjectives.
    */
   public enum Adjective {
       A, THE, NONE;
      
       public String asTextPreferA() {
           switch(this) {
               case A: return "a ";
               case THE: return "the ";
               default: return "";
           }
       }

       public String asTextPreferAn() {
           switch(this) {
               case A: return "an ";
               case THE: return "the ";
               default: return "";
           }
       }
      
   }

   /**
    * Returns an object capable of writing a one-line summary of the data
    * structure. Expect the summary to be printed at the beginning of a
    * paragraph, but make sure the paragraph is ended in such a way that more
    * lines might be appended to that paragraph, if required, by some other
    * component. I.e. make sure you end with a dot-space. (". ") Typically
    * starts with {@link Adjective#A}. 
    *
    */
   <C extends ParaContents<?>> Documenter<C> summary();

   /**
    * Returns an object capable of rendering a short reference to the type of
    * data for which the Codec provides the decoder. This reference should
    * <em>at least</em> include a reference to the type of data decoded by
    * 'sub'-Codecs. The {@link Adjective} argument allows the implementor to
    * generate a correct reference, such as 'a list' instead of 'an list'.
    *
    * <p>
    * Note that implementers should assume that the particular piece of data
    * that is going to be referenced here will be detailed further along the
    * road. Unless {@link #requiresDedicatedSection()} returns
    * <code>true</code>, that could be within the same section.
    * </p>
    *
    * @param adjective
    *            The adjective to use; <code>null</code> if no adjective should
    *            be used.
    */
   <C extends ParaContents<?>> Documenter<C> reference(Adjective adjective);

   /**
    * Returns an object capable of writing detailed information on the format
    * to the document section passed in. Typically implemented by writing a
    * (couple of) paragraph(s), and forwarding to the CodecDescriptor of a
    * nested {@linkplain Codec}. Note that - while forwarding - the descriptor
    * has the option to replace the way the buffer is referenced.
    *
    * @param bufferReference
    *            A String based human readable reference to the encoded data.
    */
   <C extends Contents<?>> Documenter<C> details(String bufferReference);

   /**
    * Returns a boolean indicating if the type of data for which the Codec
    * provides the decoder should be documented in a dedicated section.
    *
    * @return A boolean indicating if the type of data for which the Codec
    *         provides the decoder should be documented in a dedicated section:
    *         <code>true</code> if it does; <code>false</code> if it doesn't.
    */
   boolean requiresDedicatedSection();

   /**
    * Returns the title of the section to be rendered, in case
    * {@link #requiresDedicatedSection()} returns <code>true</code>.
    *
    * @return The title of the section to be rendered, in case
    *         {@link #requiresDedicatedSection()} returns <code>true</code>.
    */
   String getTitle();

}

Codec Descriptors in Preon

2009-03-02T09:39:00.001-08:00

The sole purpose of this entry is get my head around something that has been laying there rotting for quite a while now, and it's about time I get rid of it. I just haven't figured out how to do it yet. So, this is a note to self. Not only reminding me to make some changes (hurry up man!), but also to explain myself what the situation is a like right now, and how I feel I could get rid of it.

First of all, what is a CodecDescriptor. A CodecDescriptor is an interface. It's the interface that needs to be implemented by all Codecs. That is, all Codecs need to be able to return an implementation. It may be implemented by the Codec itself, but it might also be implemented by something else, outside of the codec.

Now, why do we even have an interface like that? The answer is simple. A Codec needs to be able to describe itself. This is just because of the way Preon works. If you construct a Codec for a certain type of data structure, then you essentially create a Codec that delegates to other Codecs, until you hit some of the elementary types defined by the framework. One of the ambitions of Preon is to have something that prevents you from having to maintain documentation on the encoding format. That documentation is generated by the Codecs themselves. They describe themselves, and - if they depend on other Codecs for realizing part of the work - they will delegate generation of documentation to other Codecs.

This approach has been the subject of quite some debate in the past. People argued that using something like a template language would have been good enough. I doubt it.

First of all, the inner workings of a particular type of Codec might be quite involved. Externalizing the bits responsible for generating documentation would break the encapsulation big time. Logic that is closely related is now spread across two different places.
Second, the Codec will often depend on a number of parameters. For instance, the Codec decoding numeric values requires to have the number of bits set, the byte order, the target data type, etc. All of that is also required for the documentation. Consequently, the template would require access to all of this data, and therefore the Codec would need to publically expose all of this, again breaking the encapsulation.
So you don't want to loose encapsulation, but you do want polymorfism. Whenever a Codec of lists needs to document itself, it needs the ability to say something about the type of elements it's decoding. In terms of Preon, that's just another Codec. When the Codec of lists is documenting itself, it should be able to just ask the element Codec to state something about itself, without being required to understand what that Codec is actually doing. (If you break that restriction, you are going to end up with a kazillion marker interfaces, and a lot of additional code in the Codec of lists.) So, if polymorfism is what you want, the object oriented approach makes a lot of sense.
In many cases, you would render things differently in a different context. In fact, one way of rendering might be ok in one context, but totally ridiculous in another context. If would for instance make perfect sense to generate a list item in a list, but it would be ridiculous to generate a list item in an image. With all of this, compile time checks come in handy. Not a strong case for template languages.

Now, as you may have noticed, I've drifted away from the question why a Codec needs a separate CodecDescriptor. After all, if they share so much in common - as I argue above - then why not have it al included in one class? In all honesty, I am not sure if I will be able to answer this question. The only thing I can say for sure is that the CodecDescriptor interface has had a comletely different lifecycle than the Codec so far. In many cases, if I didn't care about the actual description yet, I had the Codec return a general purpose Codec descriptor. Maybe not all that sensible, but it allowed me to continue on the important bits, without focusing too much on continuosly implementing boilerplate code.

Anyhow, enough about that. Let's take a look at the actual interface itself. This is it:

public interface CodecDescriptor {
    <T> void writeReference(ParaContents<T> contents);
    String getLabel();
    <T, V extends ParaContents<T>> V putOneLiner(V para);
    boolean hasFullDescription();
    <T> Contents<T> putFullDescription(Contents<T> contents);
    String getSize();
}

Doesn't look all that complicated, does it? The problem is that the operations are not documented. Not here, and not in the actual source code. There are processing expectations however. So I will try to capture those by looking at the operations one by one:

void writeReference(ParaContents contents): I think that this operation is expected to do is to generate a reference to the type of data structure it is supporting into the target document.
String getLabel(): I think what this operation is expected to do is to generate a refeference to the type of data structure it is supporting. However, since all this operation is capable of doing is return a String, that basically prevents it from generating a real link. (A link in the Pecia definition of the word. That is, something that eventually will be rendered into an HTML anchor tag, or a DocBook xref.)
ParaContents putOneLiner(ParaContents para): I think what this operation is expected to do is to genenerate a single short description of itself into the output document. As far as I remember, this was done to allow you to generate a short description in a table for a field, and then make sure it includes a reference to something defined in detail elsewhere. Sort of similar to the way JavaDoc works.
boolean hasFullDescription(): Called to check if the next operation is implemented in a meaningful way.
Contents putFullDescription(Contents contents): Similar to putOneLiner. However, in putOneLiner, it would be impossible to write outside a single paragraph. In putFullDescription, you can basically write a number of paragraphs, include tables, images, etc.
String getSize(): I think it is expected to return a human readable description of the size. Is it still required? That's hard to tell. Probably not, now we have getSizeExpr() on Codec.

So, as you can see, the contract is quite fragile. And there are a couple of other problems:

In some cases, you want the references to be inlined. In other cases, you want them to be part of the start of a sentence. Do you need to start whatever is generated with a capital? Or do you expect the framework to recognize when to change whatever you are generating into something starting with a capital?

As you can see, there is quite a bit of stuff that needs to be clarified or redefined. Question is, how? I will start to take a look at it tonight, and see if I can slowly refactor this to something that makes a little bit more sense, and is a little bit more helpful.

Writing a Java byte code decoder without actually writing it

2009-03-01T12:21:00.001-08:00

During my talk about Preon at Devoxx last year, I mentioned that I was working on capturing Java's class file format in Preon. This weekend, I took another shot at it, and it's starting to come together quite well. :-)

The source code itself is too much to publish here (what else would you expect from something capable of reading byte code), but if you're interested, I suggest you check out the latest version here. And as you will see, it does not actual contain any Java code reading byte code; it's all done in the Preon declarative way.

Bit Syntax for Java (I)

2009-02-24T12:24:00.000-08:00

Erlang's bit syntax is pretty cool. Now, Preon is aiming to provide something with similar ease of use. Which one is easier? You decide.

One of the examples in the excellent Erlang book shows how to decode an MPEG header, using Erlang's bit syntax. This is what it looks like:

decode_header(<<2#11111111111:11,B:2,C:2,_D:1,E:4,F:2,G:1,Bits:9>>) ->
Vsn = case B of
    0 -> {2,5};
    1 -> exit(badVsn);
    2 -> 2;
    3 -> 1
end,
Layer = case C of
0 -> exit(badLayer);
1 -> 3;
2 -> 2;
3 -> 1
  end,
%% Protection = D,
BitRate = bitrate(Vsn, Layer, E) * 1000,
SampleRate = samplerate(Vsn, F),
Padding = G,
FrameLength = framelength(Layer, BitRate, SampleRate, Padding),
if
FrameLength < 21 ->
  exit(frameSize);
true ->
  {ok, FrameLength, {Layer,BitRate,SampleRate,Vsn,Bits}}
end;
decode_header(_) ->
exit(badHeader).

This is the same code, but then in Java, using Preon:


public class MpegHeader {

@BoundNumber(size="11", match="0b11111111111")
private int frameSync;

@BoundNumber(size="2")
private int mpegAudioVersionId;

@BoundNumber(size="2")
private int layerDescription;

@Bound
private boolean crcProtected;

@BoundNumber(size="4")
private int bitRateIndex;

@BoundNumber(size="2")
private int sampleRateFrequencyIndex;

@Bound
private boolean padded;

@Bound
private boolean privateBit;

@BoundNumber(size="2")
private int channelMode;

@BoundNumber(size="2")
private int modeExtension;

@Bound
private boolean copyright;

@Bound
private boolean original;

@BoundNumber(size="2")
private int emphasis;

}

Codec codec = new Codecs.create(MpegHeader.class);
Codecs.decode(codec, buffer);

Now, there are obviously a lot of differences. The Erlang example also includes calls to functions defined elsewhere to determine the bitrate and frame lenght. The Preon example does not have that. If you would need it, you could imagine implementing it as methods of the MpegHeader class.

Also note that the example does match on the first 11 bits, expecting them to be ones only, similar to the Erlang example. The notation @BoundNumber(size="11", match="0b11111111111") basically says that this field will be decoded from 11 input bits, as long as it matches the bit pattern "11111111111". Or to be more precise, as long as it matches the numeric value of 11 "1" bits, interpreted as an integer. If that criterion isn't met, the Codec will throw a DecodingException.

I will try to work the other examples into Java with Preon examples as well, but it's interesting to see how the two approaches compare. What would you prefer? And why?

Preon versus Google Protocol Buffers

2009-01-15T01:13:00.000-08:00

Preon was mentioned in a discussion about implementing Google Protocol Buffers in JBoss. The author considers Preon to be slightly off-topic, but perhaps interesting. It makes you wonder if Preon actually bears some similarity with Google Protocol Buffers. Let's take a look at that.

Preon like Google Protocol Buffers is about marshalling between the in-memory representation of a data structure, and its binary encoded representation. In that way, they are similar.
Google Protocol Buffers is not just about encoding and decoding data: it also defines an RPC model. Preon does not; it's about the binary encoded representation only. That way, they are different.
Google Protocol Buffers already has encoding implemented. For Preon, it's clear how it should be done, but since it is not backed by a large corporation yet (anyone?), I haven't been able to work on encoding yet. In that sense, they are different.
Google Protocol Buffers relies on .proto files for defining the data structure. Preon doesn't. Preon allows you to define the mapping from a Java class to its binary encoded representation inside the class itself, relying on annotations. In that sense, they are different.
Google Protocol Buffers generates source code. Preon doesn't. Preon builds the entire Codec at runtime. In Preon, the only objects that you have to deal with are the objects you wrote yourself. In that sense, Google Protocol Buffers and Preon are different.
Google Protocol Buffers supports multiple languages. Now this is an interesting one. Since the Codec constructed by Preon captures exactly how the data is represented in a binary way, it should be fairly straightforward to generate code from it for different languages. Now, that would of course be a little unusual, but nevertheless doable. It that sense, they are currently different.
Google Protocol Buffers is not extensible. Well, it is extensible, but not in the way Preon is extensible. It does not allow you to use other types of compression techniques, unless you do that in the application layer. (Let's just say, accept a byte array, and interpret that byte array at he compressed representation of something else.) Preon is extensible. It's like allowing the protocol buffer language to be extended to support new types. In Preon, you can create your own CodecFactory, and hook that up with the framework. From that point on, sky is the limit. In that sense, they are different.
Google Protocol Buffers supports decoding the full data structure all at once, in a single pass. Preon was designed to decode data just in time, only if we need it. That was done deliberately. Preon originates from decoding map data. You don't want to decode a full map into memory all at once. It would blow up the heap, if you need to stay within the boundaries of a Java 32-bit VM. So Preon will try really hard not to decode all data all at once. In that sense, Google Protocol Buffers and Preon are completely different animals.
Preon allows you to define dependencies between different fields, and have that influence the way data is decoded. In that sense, it's different from pretty much anything else out there - apart from the Flavor derivatives. So if you first read a numeric attribute, then you can reuse that attribute further along down the road, for instance by interpreting it as the number of elements in an array Preon has to read. For media content, this is fairly common. In typical serialization mechanisms, this is pretty uncommon. The expressions you can use to define these dependencies can be as complicated as you like. In that sense, Google Protocol Buffers and Preon are totally different.

In summary, Google Protocol Buffers and Preon are quite different, and address different needs. Preon is better suited for decoding existing media types. Google Protocol Buffers is better suited for integration projects.