Endeavors in GPLFlash2

Project moved…

2006-05-14T13:32:00.000-07:00

GPLFlash2 has faded into the mist. Folks who still want to follow development on a FLOSS SWF interpreter should check out gnash.

More Dead Air, and More to Come

2005-07-31T22:30:00.000-07:00

All right, folks; sorry again for the huge delay in posting. I've been letting my subconscious turn over the design (read: slacking off), and there are a few changes that will need to be made to some of the interfaces that I've defined so far. I'd really like to get at least the SWFIO design done before the end of the summer, but I don't think that's going to happen. Which leads to my next announcement…

I'm going in tomorrow (Monday) to negotiate a short-term, full-time job at a software company I have prior dealings with. Short of dire budget problems or a major catastrophe, I suspect I'll be employed starting August 11th to September 27th, and school will be starting up directly afterward. Furthermore, I'm going to be on a road trip and unavailable from August 4th to the 10th. Thus, I'm probably not going to have too much time to do anything really substantial for the rest of the summer.

SWFIO Filters Design

2005-07-27T09:04:00.000-07:00

All right, I sat down and hammered out the basics of the SWFIO Filters package. I still haven't documented the semantics of all of the calls yet, but I know what they are (I just need to write them down). Once I get the semantics documented, I can write the unit tests — after that, I can implement the code, or leave that for someone else to do while I move on to designing other pieces.

I've updated the Umbrello diagram on the server (check the links at the left), and have also generated an image of the diagram (click the image below to see the full-scale picture). Enjoy :).

Cairo and GPLFlash

2005-07-18T08:30:00.000-07:00

All right; I'm going to really drive the "Cairo will/won't work for us" argument into the dirt. I think it will work for us, and I'm prepared to do the research to back that hunch up.

This email is a bit deceptive on a couple of points, though not intentionally so. Cairo, first off, is a vector drawing library, which means it talks in terms of paths (curves/lines), strokes, fills, and transforms. Both SWF and SVG are vector-based formats; SWF uses only paths, strokes, fills, and transforms, whereas SVG uses these in addition to primitive shapes.

The big issue for correctly rendering SWF shapes is how fills are performed. The aforementioned email states that there are "left/right area[s]"; drawing on the spec, I'll elaborate a little. The specification says that there are two fill types that a path can have: "fillstyle0", which is applied to the left of the vectors in the path, and "fillstyle1", which is applied to the right of the vectors in the path. However, due to the examples and language used after this statement, I am under the impression that "fillstyle0" is for interior fills, and "fillstyle1" is for overlap/self-intersection fills, and that the left/right nomenclature was purely for the sake of example. The only way to find out will be to run a test through MM's player; I'll try to construct such a test in the near future.

Update: I have constructed tests. I have been totally unable to get MM's player to render "fillstyle1" and "fillstyle0" simultaneously. MM's player, for all intents and purposes, behaves exactly like "even/odd" filling, which Cairo provides. If anyone is interested in getting the test files and source code used to generate those tests files, ask me for a URL in IRC (any link posted here would probably go stale too fast).

Here's a table that compares SWF's functional needs with the drawing tools that Cairo provides:

SWF-required	Cairo
RGB/A Solid fill	Provided (sample)
Linear Gradient	Provided (sample)
Radial Gradient	Provided (sample)
Clipped Bitmap Fill	Not provided
Tiled Bitmap Fill	Provided (sample)
Stroke width	Provided (sample)
Stroke color	Provided (sample)
Moving Without Drawing	Provided (sample)
Affine Transforms	Provided (sample)
Drawing Lines	Provided (sample)
Drawing Curves*	Provided (sample)
Even/Odd Filling	Provided (sample)

*Note that the Bezier curves must be converted from quadratic (SWF) to cubic (the rest of the world :p). This is a straightforward transformation.

The only thing I can see standing in the way of using Cairo is the "clipped bitmap fill," which takes a 1 pixel outline of the bitmap, then extends that outline as far as it needs to go to fill the area. I can't easily generate a test for this functionality to see how it works in MM's player, so I don't know if it will be easy to emulate (using something like imagemagick) or not.

Overdue Update

2005-07-17T09:04:00.000-07:00

Sorry about all of the dead air lately. I have been rather distracted these last few days, so I don't have much to show :p. There are still a couple of news-worthy tidbits, despite my lack of concentration.

ffmpeg Dependency Issues

First off, thanks to willem for helping out with this matter.

So, it turns out that ffmpeg actually generates .pc files (for the pkg-config system), but how those files are handled varies from distribution to distribution. Under Debian, an ffmpeg-config script is installed, and other under distributions, the .pc files could be installed. Under Gentoo, none of those files is installed. Thus, I've re-written our ffmpeg test so that it looks for "ffmpeg-config" first, "pkg-config avcodec avformat" next, and then finally tries to just right-out look for "-lavcodec -lavformat" (or "-lavcodec_pic -lavformat_pic").

The main reason we have to use ffmpeg-config or pkg-config first is because on systems like Debian, the ffmpeg libraries are cross-linked to lots of other optional libraries. Thus, there is a variable list of -l flags that must be used in addition to "-lavcodec -lavformat", and it would be foolish to try to brute-force all of the possible combinations.

Edits & Corrections

kermit_ has kindly donated his time and skills as a technical writer. He's already made a few adjustments to the GPLFlash homepage, and he's presently working on annotating and cleaning up the contents of gplflash2/doc/. Thanks kermit_!

NPAPI updates

I've made various updates to the NPAPI docs (html, pdf, and texinfo source). The initial documents were totally, flat-out wrong in some respects. I could really use a Windows programmer and a Mac programmer to help me write the platform-specific parts of the guide. Anyone interested?

Scouting Documentation for NPAPI

2005-07-12T19:32:00.000-07:00

I'm writing something that I've dubbed a "scouting document" — documentation that covers a whole topic that will affect us in the future, but only a subset of which I'm interested in at the moment. In this case, I've decided to take a detour and start working on a thorough and comprehensive NPAPI guide (since even the official documentation has some holes and misinformation). When someone in the future goes out to re-write our plugin (as I'm sure will happen eventually), this document will hopefully be the only thing they need to understand the requirements on the plugin side of things. I've been working on it for about two days now, and hope to have my research and first draft completed within a week. For those interested in alternate formats, the texinfo source document and pdf rendering are also available.

Explaining SWFContext

2005-07-11T11:12:00.000-07:00

Someone, whilst looking over the diagrams in a previous post, asked me why the SWFContext package depended on the SWFIO package. The answer to that initial question is because the SWFContext package will (or has the high potential to) handle SWF tags, structures and/or primitives. I think it would be highly unlikely that, throughout the entire package, not one class or sub-package would handle objects from SWFIO. Handling (sans a higher-layer abstraction or wrapper) is enough to cause dependency.

The path that our discussion took, however, went towards "ActionScript only" coding, and how it wouldn't make sense to have just tags for keeping context in that case. This is entirely true: in a file where you have only two tags (an ActionScript and an End), the vast majority of the work is not going to be done with tags; it'll be done in the ActionScript VM, using context-saving methods that are designed for that purpose. However, note that there is no conflict here: SWFContext is a package, not a class. I used packages on purpose, because I really don't know enough about how the system should work overall (yet) to create classes.

Now, I surely haven't thought deeply about AS and how to design its implementation yet — and I most certainly didn't think about the situation of a completely programmatic SWF. However, the design does allow for these "future requirements" by creating appropriate extensibility points. To implement AS, we shouldn't need to change anything in the SWFContext package at all, nor should we have to change anything in the TagLogic package (aside from implementing the AS tag handler). Creating things inside of AS will be managed by the AS VM; the context will be allocated by that VM, and could then be stored in a variety of places (such as a void* in an STL map container, either in the TagData object itself, or back in some context object in SWFContext).

Now keep in mind that I made up most of the last paragraph you just read (that is, I hadn't planned it out in advance). The design is far from being coherent, let alone complete — it's just a rough roadmap for the time being. In fact, a very important detail (having a SWF- and/or tag-central location where multiple "plugins" can store independent contexts) came out of the exercise. I'm nowhere near touching AS, or rendering, or user input, or sound, or A/V sync, or any other of a million other broad and nuanced areas that the final player will encompass. That's why getting questioned and challenged along the way is ultimately a good thing — other folks can see the potential mistakes I'm making, and that gives me the chance to plan out and ensure I don't make that mistake. That is, after all, what design is about ;).

Asking the Mozilla Devs

2005-07-11T00:03:00.000-07:00

All right; the long and short of it is that the "simple" plugin from last time didn't register or otherwise work. As such, I decided to pop over to irc.mozilla.org and see if I couldn't straighten things out directly with the developers. Apparently, the XPCOM plugin interfaces are deprecated >_<. So… there's no reason not to just use the plain-jane C NPAPI interface. I'll go ahead and design with that in mind.

Pulling Mozilla CVS

2005-07-10T08:12:00.000-07:00

So, after looking at several "real-world" examples, it turns out that all of them (including the mplayerplug-in project) use NPAPI in order to actually register and communicate with the browser. There are, however, sample plug-ins in the Mozilla repository that appear to not use the old NPAPI at all. I've been examining these plugins, but have not had any luck getting my own version to show up in about:plugins. I'm currently building the whole Mozilla suite from CVS now in order to see if the sample plugins actually register properly, or if it's just me doing something wrong in my implementations.

For the curious, here is the link to the "simple" sample plugin that I'm trying to build. I think I've learned a lot about the architecture thus far, but since I have yet to get a plugin working, I can't be sure. As such, I'm going to refrain from documenting my "findings" from studying that plugin until I'm sure that what I think I've learned is fact ;)

I'm also starting to wonder if this whole process is worth it. Since so many other plugins use the old NPAPI, why not go ahead and use it for GPLFlash? The old API certainly seems like it would be easier to use. I guess the big thing is the scripting interface that can be exposed with the new API, though that can be done in conjunction with using the old API calls for plugin registration. Ah well — the "simple" plugin sample just finished compiling (with a little bit of tweaking), so I'm off to test it.

Pulling Real-World Examples

2005-07-09T15:04:00.000-07:00

Well, I decided I'd go see how other folks have implemented plug-ins, and use that as a basis for building my own. I started out with the plug-ins that I have installed: gxine and librsvg. Both of these use the "old" NAPI style. Then it occurred to me that someone had mentioned another plug-in that we should borrow from — the mplayer plug-in. I fetched the source for that plug-in again, and lo and behold — it uses XPCOM. I went back and grep'd my logs to see who suggested that; thanks tgc! You had the right idea, you were just ahead of me in the scheme of things. I'm going to spend today alternating between doing housework, and picking apart this plug-in.

XPCOM and Gecko Plug-Ins: Help?

2005-07-09T03:52:00.000-07:00

So, it looks like the interface I currently have a link for (on in my "reference" links on the left) is for the "pre-5.0" plug-in system. Apparently, there's a new system based on "cross-platform component object modules" (XPCOM). This means that it's OO, interface-based, and roundabout ;). Unfortunately, there doesn't seem to be very good documentation on how to use the XPCOM system to make a plug-in, so I'm a bit stuck. I was preparing to write a little program to figure out how Mozilla/Firefox's plug-in system works, so that I could better design the SWFIO::Filters::NAPI::Read class.

Does anyone have familiarity with the XPCOM model in Mozilla? Does anyone know of a good reference for writing a plug-in based on this model? If not, I'll do my best to document my own path.

On an unrelated note, I'm going to try and be more regular with my updates. It appears as though folks are actually reading this thing now, and it's rude of me to be too far off in my corner without updating you all on what I'm doing. I'll do my best not to dilute the posts, but I'll also be more forthcoming with the snags I've hit, and the intermediate steps I'm going through.

that's all for now :)

Anon. Comments

2005-07-08T13:53:00.000-07:00

This is a bit of a delayed announcement, but I have turned anonymous comments on in order to allow a wider range of people to respond to the posts.

A Walk Through Design Lane

2005-07-06T21:06:00.000-07:00

All right, I finished documenting my topmost level of design, as well as the first (of many) second-level design diagram. I'll be presenting these in this post; the original information is all available in the current design document.

Top-level design

Because of the dependencies, the packages should be implemented in this order:

SWFIO
SWFContext
Frontend (can be done concurrently with 1 and 2)
TagLogic

Package Descriptions

SWFIO

A strictly data-oriented interface to the SWF format.

Handles reading and writing all primitive data types, and data-only structures from an SWF stream.

SWFContext

Provides different levels of containers that, as a whole, represent the state of an entire SWF file.

The SWFContext package is intended to provide a common access point for external SWF processors (such as SWF interpreters, writers, or even dynamic filters). Some of these applications will want the SWF container to be smart (in the case of writers), whereas others will want the container to be as dumb as possible (in the case of interpreters and some special filters). This is why SWFContext is a package, and not just a single class.

Frontend

A normalized interface to shuffle data to and from the user.

Any implementation of the Frontend package will need to provide functionality for reading keyboard events, reading mouse events, displaying graphical objects, playing sounds, and possibly reading graphical objects (webcam) and reading sounds (microphone).

TagLogic

A behavior-oriented interface to the SWF file format.

TagLogic requires SWFIO for the TagData package it contains. Each TagLogic class representing a flash tag has a corresponding SWFIO TagData class embedded within it. TagLogic also requires the SWFContext package, since tags may need to refer to and modify other tags in the SWF file. Finally, TagLogic also requires an appropriate Frontend package (which could be implemented multiple times, allowing for the Frontend to be switched) in order to get input from the user, and provide the appropriate visual/audio output.

SWFIO package

Because of the dependencies, the packages should be implemented in this order:

Filters
Datatypes
Structures
TagData

Package Descriptions

Filters

Provides a framework for normalized I/O and drop-in data filters.

This package defines filter classes that can be "chained" together, similar to a pipe on the command line. Filters can perform an in-line function (like encryption/decryption, compression/decompression, executing a SED script, etc.), normalize access to an existing data source/sink (STL iostreams, C file I/O, NAPI asynchronous I/O, etc.), and/or provide "special" data extraction/insertion methods (SWF primitive data types).

Datatypes

Provides a way to access SWF primitives that are not directly provided by the SWF filter.

Some primitive data types (most noticeably the fixed-point types) are directly derived from other SWF primitives. As such, these types are not really primitive, but alternate interpretations of "true" primitive types. This package sits on top of the SWF filter, and provides the primitives that the SWF filter does not directly support.

Structures

Contains classes for data structures that are used by several different SWF tags.

The structure objects in Structures do not have any behavior other than asserting that the structure values are always within their bounds and providing correct serialization/deserialization to/from an SWF filter.

TagData

Contains classes that read and write entire SWF tag objects.

The tag objects in TagData do not have any behavior other than asserting that the tag values are always within their bounds and providing correct serialization/deserialization to/from an SWF filter.

I'm Back!

2005-07-06T19:36:00.000-07:00

All right; now that I'm back from my hiatus and settled in, I'm ready to go back to doing some serious work. Here's what I'm planning to take care of, in order of importance:

Getting more work done on the new design.
Assisting newbies on IRC and the mailing list
Helping the wiki move along.
Reviewing the "detailed" design document for the current system.
Replying to E-mails. My apologies to folks on this one, but crafting an E-mail is a 30 minute to 2 hour job for me, and I can't really work on anything else in that period. Catch me on IRC!

Expect updates on the design soonish. I'll try to generate some png files for important parts of the system for those who can't or don't want to muck with Umbrello.

General Updates

2005-06-29T17:34:00.000-07:00

Well, I've been busy over the last couple of days, mostly working on getting the GPLFlash Wiki populated with some base-line information. I've also filled in the final sections of my knowledge management paper; feedback on this paper would be welcome :).

Also, I'm probably going to be out of touch for the next 5 days — going to visit relatives tomorrow, and then I'll be attending a four-day LAN party directly afterwards. I'll see what I can do between rounds at the party, but communication is likely to be sporadic; don't count on anything ;).

MoinMoin: The Tentative Winner

2005-06-25T10:05:00.000-07:00

So, now that I've tinkered around with it a bit, it looks like MoinMoin could come out as being the wiki that powers GPLFlash. Briefly, here are the pros and cons of the system:

Pros
- Seems very hack-able/extensible
- Log in works (as previously mentioned)
- User and group permissions are flexible, and lend themselves to delegation and the autonomous operation of sub-groups.
- Multiple indexing schemes should help both site users and maintainers to find what they need.
- Categories look like they have potential (but I have not tried the feature yet)
- With some hacking, RSS support now allows for fine-grained monitoring of the site for page updates
- The wiki syntax and macros provide a robust set of tools to create, reference, and re-use content within the site
- Skinnable with CSS, so people can color to taste. Also allows for the possibility of creating accessible versions of the site (e.g., "high contrast," "large print").
- File uploading/attachments
- Extensive documentation
Cons
- No database, which means the files eat our web quota, we have a file permissions nightmare, and the system is liable to be slower.
- CGI, and the server doesn't support FastCGI (at least for Python). This is a disadvantage to PHP, which can have a single (or many) long-running threads that can cache things in memory and share info between processes. CGI requires a new process to be started for every request, and any caching must be done to disk.
- Renaming a page makes all existing links to that page turn into "create this page" links. No redirect is inserted, and the links themselves aren't changed to point at the new page. This may be fixable by modifying the RenamePage plug-in.

If anything else springs to mind, I'll be sure to update this post. In the mean time, I'm probably going to go about re-installing MoinMoin and setting up the site for a test pilot.

So Many Wikis, So Little Time

2005-06-24T18:59:00.000-07:00

Well, I've been installing and uninstalling wikis like crazy over the last couple of days, along with the assistance of Thomas. For future reference, here are the systems that we've tested, and why we've rejected each:

UseModWiki: No real user handling/page control. "logging in" was not possible; a new user was always created.
MediaWiki: Couldn't stay logged in.
TWiki: Could create users, but no passwords; unable to log in.

Currently, I've got MoinMoin installed. It was a bit of a pain to set up, but now that it is installed, it looks like it may actually have real user support (!). I wouldn't have thought that such a basic feature — logging in — would be so hard to find. I'll continue tinkering with MoinMoin and see if it's all-around adequate for our needs.

Update: Other wikis that were rejected:

WackoWiki: (not installed) Documentation indicated that groups must be completely administered via configuration files on the server; no wiki interface, and no delegation.
PhpWiki: Difficult to install, and difficult to administrate; spotty documentation.

GPLFlash and Knowledge Management

2005-06-23T10:31:00.000-07:00

Well, I decided to start putting a little more work into the knowledge management paper I'm doing for school, and I'm almost finished with it. I've decided to put the work in progress (PDF) up for anyone who's interested — if anyone would like to comment on it, please feel free.

More Rotten Fruit

2005-06-22T03:29:00.000-07:00

ArgoUML can't differentiate between classes with the same name but in different namespaces. Hello, guys; remind me what the whole point of namespaces is? ArgoUML also has a UI that is so painful to use, I almost want to slam my computer to the floor in frustration. Drag-and-drop works, but only some of the time in some cases; click-to-edit works, but sometimes the changes take and sometimes they don't; while working, internal parts of the program seem to crash and can only be fixed by a restart… the thing feels like it's taking lessons from a younger M$. I swear I have never been this angry at a program before — and it takes a lot to get me angry, especially at a computer.

I think that the most frustrating thing is that the program seems like it could be vastly useful if the UI weren't so bad… and the software weren't so buggy… and the model elements interacted in an intuitive fashion… and it could import XMI right… I don't know how, through all these flaws, it manages to seem useful, but it does pull it off, and it's utterly infuriating. Unless you want to bash your head against the wall, do not use this software.

So, here's the make/break list for the software I've tried thus far:

ArgoUML: Namespace blind. Using it makes me see red.
Umbrello: Bad C++ generation.
Gaphor: No XMI import, no code generation, and it seems things must exist in a diagram to exist in the model overall.

So, I'm bust for tools. I think I'll go back to Umbrello, since it's pretty close, and very simple to use… it just has crappy code gen. But I think I can live with that *sigh*. Can anyone else recommend a good tool?

Rotten Fruit

2005-06-21T23:33:00.000-07:00

So, I got through a good chunk of design in Umbrello, only to realize that it doesn't have any concept of a const member function (i.e., a "query" stereotype method). I'm a very const-aware person, and the lack of this feature makes the whole tool utterly worthless to me. What's even more aggravating is that I can't open the XMI file in ArgoUML. So much for interoperability… I'm currently hand-copying the information over to ArgoUML (which does handle the query stereotype), and I'll post that XMI file when I've finished. If you haven't noticed, I'm keeping the design link on the "links" list; the last updated time will help you figure out if you have the newest version or not.

Fruits of Labor

2005-06-20T14:33:00.000-07:00

Well, I've found a diagram editor that I really like, and it spits out files that seem to be standardized/portable (XMI). So, I've gotten some very rough (and by no means complete) gplflash models done, at long last. You can read the files with Umbrello, or with other tools that can read XMI (like ArgoUML, or Gaphor)

I've also tarballed my current SWFIO concept, which at the moment provides access to only the SWF primitives.

Update: I forgot to mention that feedback is requested; sending in diffs and ideas would be much appreciated!

Coming Attractions

2005-06-17T20:37:00.000-07:00

I've retargeted my school project to incorporate GPLFlash2 — that way, I'm really only working on one project, and getting real work done with the project (instead of just fake academic-land "work"). It will be a small (fewer than 15 pages) report on Knowledge Management, and how the GPLFlash project could greatly benefit from it. I'll make it available as an sxw and a PDF as soon as I've finished it.

SWF Rendering: Part 1

2005-06-17T11:42:00.000-07:00

Now that I have the basics down, I've started to take a deeper look into SWF overall, in order to get a better feeling of where I'm going and what design decisions need to be made along the way. As part of this process, I've decided to write a set of articles explaining how the SWF rendering process works. This will serve three purposes:

It helps crystallize the idea in my mind
It clarifies the ideas for people who have only cursory knowledge of the subject
It exposes my ideas to critique, so flaws can be worked out

In this article, we'll go over SWF interpretation up to and including definition tags. The next article will be dedicated to the interpretation of non-ActionScript control tags, and possibly the visual model that the SWF format should be interpreted with.

Step 1: Decoding Tags

Before anything else can happen, the data from the file must be parsed into data structures in memory. There are three major parts to this on the file wide scale:

Header extraction
Decompression (dependent on header)
Tag extraction

"Tag extraction" is a bit vague, however; the RecordHeader data structure means that we could extract all of the tags simply by pulling the appropriate number of bytes out of the stream and store them in a character array. This wouldn't be very useful, however, since we still wouldn't know what information is trying to be conveyed by the file. Thus, we have a little more to do on the per record scale (header included):

Know what type of record we're extracting
Know the SWF primitives (their type, order and meaning) that make up the record
Extract each primitive to a corresponding, size-compatible, in-memory variable.

Once we can extract tags in a meaningful manner, we have enough infrastructure to start processing them. Because SWF is a streamable format, we don't need to have the whole file before we start working.

Step 2: Definitions and The Dictionary

As definition tags are discovered, they are then added to the "dictionary." This is a static repository for all of the objects with definition tags (called "characters"). Other tags (called control tags) generally refer to one or more tags in the dictionary, applying transforms in a cumulative filtering (or "delta based") fashion. This makes implementing the dictionary straightforward, as all it has to do is allow new characters to be added, and report the values of existing characters.

SWFIO: Layer 0 Completed

2005-06-16T22:03:00.000-07:00

Well, I've just got my lowest-level I/O code working. It now both writes and reads the integral SWF data types (UI8, UI16, UI32, UB, SI8, SI16, SI32, SB), and the hard bits (that is, the bitfield operators) pass a suite of tests that is thorough enough to make me comfortable that the code works. The only thing I can think of that I'm missing is a mechanism to quickly pull out a very large quantity of data (such as an embedded JPEG or MP3), but I figure that's a feature I can add later if I really need it.

The next thing up is layer 0.5, or the "fixed point" data type. After that will be the first substantial layer — layer 1, the structure layer. I'm eager to get to that point, and start getting the structures out of the way. I think it will be very easy going, since the nasty part (interfacing with the data stream at a bit-wise level) is done.

Some other misc. stuff does need to get done to the layer 0 code before it's really "completed" — specifically, I need to add logging and create better unit tests for the non-bitfield operators. There exist tests for all of the operators at the moment, but the tests for the aligned accesses are wimpy (since the code implementing them is almost laughably trivial). Ah well.

Better Design Through Layering

2005-06-13T15:09:00.000-07:00

Well, I finished the lowest level of my SWF I/O code (well, technically there's no I yet), and have been taking the time to think about what I'm doing and why I'm doing it. The nice thing about writing parsing code is that it's fairly straightforward, and you can be pretty much guaranteed that you'll need it later; thus, you can start hacking it up first (to handle that "gotta code" urge), and simultaneously contemplate what to do next.

Someone asked me the other day what I thought the design flaws were in the current GPLFlash2 implementation, and I didn't give them a good answer. I've been thinking about it, and have started to define exactly what it is that I don't like about the current design. The major things that I think are missing are the use of modularization and layering.

As an example, the Swf_decoder class does absolutely everything for decoding SWF. In fact, it knows every data struct and every class (!) that could be stored in the SWF. Furthermore, it knows the exact order and type of data that needs to be pulled out of the SWF — the objects don't read themselves in, the Swf_decode object encapsulates (IMHO, inappropriately) all of the knowledge of all of the tags. This "everything at once" approach is difficult to test, difficult to maintain, and difficult to get consistent and predictable quality out of.

I propose a design that takes a much more modularized and layered approach to the problem. Instead of having one great big monolith that tries to do everything, I envision building up a system that depends on discreet services. The layers move from data to presentation in a gradient, fanning out as it moves closer to the "end result" that the user sees.

The first bit that I'm working on is the "bitstream" reader and writer. This is the absolute lowest layer, and acts as the most general interface to an SWF file. It provides only the functions needed to read and write the integral SWF data blocks: unsigned integers (little endian; 8, 16 and 32 bits), signed integers (little endian; 8, 16 and 32 bits), unsigned bitfields (big endian; 1-31 bits long), and signed bitfields (big endian; 1-31 [2-32 with sign] bits long). The other data types (fixed point integer and bitfield) can be derived from the types already included. In truth, we need only provide "integer" functionality without sign (sign is unimportant, and simply a matter of interpretation), but for the sake of consistency, I've provided calls for both signed and unsigned integers. Since handling of signed bitfields is different than that of unsigned bitfields, both functions are necessary.

This interface is the absolute bare minimum needed to interpret the contents of an SWF file. By choosing a very tight set of functionality, we can ensure better quality — every one of these functions has a set of unit tests, as well as an integration test to ensure that they play nice together. This is an application of the "do one thing and do it well" principle — the SWF stream wrapper doesn't need to know or care about tags or non-primitive data structures; all it needs to know are the primitives.

Now, I've been talking about layering, but so far I've discussed only one component. Well, the SWF stream wrapper is the lowest layer in the system I'm envisioning. Everything else is built on top of it, so it's absolutely critical that it provides rock-solid functionality. Because it's so small, we are much more able to build it up to the level of stability that an entire system can depend on. The very next layer is also thin: it's the set of structures that emulate the "missing" core types (fixed-point numbers). This layer need only know about the SWF bitstream wrapper, and doesn't care about anything else. The next layer up will be the data structures. They don't need to know about one another — they just need to know how to read themselves out of a SWF stream wrapper, using the fixed-point classes if needed.

As you can see, we're slowly building our way up. We now have the very core types, the fixed-point types, and the data structures (Rectangle, RGB, RGBA, Matrix, etc.), all of whom know how to use the lower layers to construct themselves. There is no interrelation between things at the same layer, so bugs are isolated — the bug either exists in the class itself, or in subset of objects that it uses from the layer below it(1).

Moving up one more layer, we finally get to tags. The tags, like the structures, will know how to read themselves in off the stream — nothing fancy in the tag classes themselves, they just use the lower layers to fill in their data when constructed with a SWF stream wrapper, or when asked to extract() themselves from a SWF stream.

The alert readers in the crowd will probably be asking at this point, "But Quandary, how will we know what tag to read off the stream? If the tags read themselves in, they won't know until it's too late if their object type is at the start of the stream!" Well, to put it shortly, the tags themselves don't actually care. In fact, the tags don't even need to have a variable field for their ID (it can be a static const), because they aren't going to read the ID off the stream. It's not the job of the tag to know if it's next up on the stream — that's someone else's job (we'll come back to this later).

So, now we've got three layers (and one mini-layer), with each class in each layer being nice, compact, focused, and (mostly) isolated from all of the other classes in its layer. All this infrastructure doesn't do much, though — actually, all of these "layers" are really sub-layers of the overall system's data layer. The data layer's responsibility is to read data in from a provided input stream and put it into objects that the rest of the system can understand and use in other operations. Having a solid data layer that can stand independent of the system is a crucial aspect for reusability — it is difficult to re-use data layers if they take actions for or make assumptions about other parts of the system.

So, getting back to the "how can a tag read itself in?" issue. That problem starts moving out of the pure data realm and starts treading into the realm of behavior, or the application logic layer. The problem is still in the data layer, because the SWF specification tells us how the file is structured and how it must be read in order to make sense. However, the bit that actually makes decisions about how to decode the stream uses the tags — it is not a part of the tags. I've settled on using a "class factory" pattern to solve the discreet problem of reading a tag in from the stream. Given that we know a tag is at the start of the stream (and there will be a process that knows when this is the case; we haven't discussed it yet) we can have a RecordHeader read itself off the stream (and hey, it'll even automagically handle long versus short record types! The joys of encapsulation). We can then pass that RecordHeader to the TagFactory function(2), which will use the tag type in the record header as an index into an array of function pointers. Each entry in the array will point to a static "create" function, to which we can pass an SWF stream wrapper (and tag length) to and get back an appropriate tag. In the cases where we cannot decode the tag, the corresponding table entry will point to the create function of the generic tag class. This will eat the unknown tag off the stream so that file processing can continue.

So, we've worked further and further up the chain, and this is all starting to look like a nice bottom-up design. We now have a mechanism to create tags from the SWF stream wrapper, so all that's left is the top-level controlling class that provides all the glitzy "look, it's a parser!" feel. This top-level class will be pretty straightforward: take a filename or an existing stream, create the stream (in the former case), wrap the stream with the SWF wrapper, tell the header to read itself in, check for validity, then go into a loop of reading RecordHeaders, passing them to the TagFactory, and getting back nice, pretty tags. And for anyone who's thinking "Hey! You glossed over compressed SWFs!" — yes, you're right, I did.

So, let's take the opportunity to extend the system. What happens when we have a compressed SWF file? Well, the data stream starts off uncompressed, then switches to compressed. So, all we have to do is modify the SWF wrapper class to have a new method (like decompress()) that instantiates a lower layer wrapper class — one that does on-the-fly ZLib decompression — passes that lower layer wrapper the current stream, and then replaces the current stream with the ZLib-wrapped stream. From that point on, everything is exactly the same as it was before. In fact, we don't even need to implement this functionality directly in the SWF wrapper class — the high-level control class can just toss out the SWF wrapper, re-wrap the stream in a ZLib wrapper stream, and create a new SWF wrapper class to wrap the ZLib wrapped stream.

1: This is somewhat of an oversimplification; because the classes share a common data store (the SWF stream), a bug (reading/writing too much or not enough data to the stream) can cause later reads and writes to not return the expected results.

2: Note that it would be an (understandable and common) error to make the TagFactory function a method on, say, the Tag class. Tag should not have to know each and every class that derives from it. The whole point of derived classes is that the parent has no clue what's being implemented under it — any class that could benefit from inheriting the data and behaviors that the parent provides should be free to do so without having to notify the parent. TagFactory is a domain (if not an application) specific tool that works with and around tags — it is not part of a tag (in much the same way an emergency dispatcher is not a policeman or firefighter).