I’ve been following Carmack’s tweets on porting RAGE to iOS and was intrigued by his and other mentions that memory mapped files work so well on the platform.

Mapped files have the advantage of being cached by the OS (in kernel memory space) and paged in and out of memory as necessary. On a desktop platform or console, the unpredictable seek time of the disk makes this less feasible, but with the gauranteed solid state disk of apple products it becomes an interesting proposition.

Since the mapping is in kernel space it can actually remain cached across application launches which is a nice performance win.

The Test

Wooords has a 1MB resident english dictionary for checking word submissions against. The dictionary is a array of sorted 64bit ints, to look up a word a simple binary search is performed.

The dictionary is an ideal candidate for moving to a memory mapped file as the OS can cache the commonly hit parts of the data file and leave the rest on disk (although I’m not sure how smart the caching algorithm is? Is it page based?)

To test these theories out I implemented the required calls and timed the binary search for a variety of words.

As we can see the mmap array has similar performance but there’s an initial spike as the data is loaded by the kernel and a couple smaller spikes during usage.

For dictionary lookup the extra latency now and again is not a problem and therefore the choice to use mapped files easy.

Technical Implementation

To use a memory mapped file, simply open the file as usual and then call the mmap function. Once you’re done with the data munmap it and close the file.

Map

FILE* fp = fopen("file.dat", "rb");
fseek(fp, 0, SEEK_END);
size_t length = ftell(fp);fseek(fp, 0, SEEK_SET);
int fd = fileno(fp);
off_t offset = 0;
void* data = mmap(NULL, length, PROT_READ, MAP_SHARED, fd, offset);
fclose(fp);

Arguments

Unmap

munmap(data, length);

Conclusion

Using memory mapped files can reduce your process memory footprint with minor modifications to your code. It can however cause small load stalls, so the best thing to do is to benchmark it in your application.

Overall its another great tool in the iOS developers toolbox.

Unfortunately Wooords requires a lot more than 25 (the current GameCenter limit) leaderboards as the  game is going to ship with a lot more puzzles. This has led me to consider a number of other options. After reading a number of great posts on GAE, I decided to it was the best fit for my needs and I like Python. Heroku could be another good alternative if you prefer Ruby on Rails.

Implementation

For a basic leaderboard implementation all we need to be able to do is submit and query scores. More complex features like querying friends scores can easily be built on top of this.

Message Format

I chose json as my message format as I already had support for it built into my engine. Unfortunately GAE runs python 2.5, which doesn’t support json natively, however I downloaded simplejson (which works identically to the 2.6 version) and it just worked. All communication is done via HTTP post requests to the google cluster.

Names and UserIds

Each user requires a unique id in the system, rather than forcing to users to create an account I decided to use the existing GameCenter information to store the UserId and their name. When the game starts up it ‘registers’ the user with the server, this simply records their id and name and gives me the option to send back a Message of the Day. If the user is running an older version of iOS or a device that does not support GameCenter, the system fallsback to using the device id and allows the user to pick a name.

Security

With a simple json message format it won’t take long before somebody starts sniffing your network packets and then attempts to submit a fake score. To counter this all messages that require security checking contain an additional auth token. This token is based on a number of things, the most important being a server key that changes regularly.

The system consists of two tokens, a client key and a rolling server token.

When the client wishes to send a secure message it first requests an authentication token from the server, this token is only valid for a short period of time to reduce the chance of somebody sniffing and then using the token. This server token is then combined with the client token and used to sign the contents of the message.

When the server receives the request, it checks the signature against the contents of the message, if it matches the message is processed, otherwise it is ignored.

It is important to obfuscate the client token in the executable so that the key cannot be obtained with a hex editor.

Other Advantages

The custom leaderboards can now store a lot more than just score. I track time, word count and a bunch of other statistics as well.

After implementing the leaderboards a bunch of other things became a lot easier, for example I could now store user profiles in GAE and have cross device syncing. When the game starts, it simply requests the profile from the server, if the profile received has more points than the profile on the device it replaces the local profile, otherwise we send out profile to back to the server.

All in all working with AppEngine has been a very easy process, things will get a bit more interesting when the app goes live.

In this article I’m going to give a brief overview of how and why I chose to use OpenAL.

Why OpenAL?

OpenAL provides a simple cross platform API for playing audio. Developing directly for OpenAL ensures compatibility across all Mac, Windows and Linux desktop machines too, which is extremely useful for development as audio and art assets can be tested directly on desktop hardware without the need for the ios simulators.

OpenAL also automatically handles the device mute button, which is a handy side affect.

Which file format?

On desktop systems, ogg-vorbis would probably be the best choice for audio encoding as is completely free and has great compression ratios. However on mobile platforms using too mich CPU time for audio decompression reduces framerate and adds unnecessary battery drain.

The Quicktime IMA4 format reduces audio file size to a quater of its PCM equivalent while only requiring a small amount of code¹ in your application which makes it very attractive for this platform. Apple provide a number of functions for working with audio files which makes implementation a lot easier, however because of my requirements for application to run on non Apple systems I had to implement all the low level functions. I found the older AIFF file format a lot easier to work with than the more modern CAF format.

It should also be noted there are open source libraries to help with loading audio like libsndfile, however their license (LGPL in the case of libsndfile) make them too restrictive for static linking in an closed source iOS application.

Implementation

I chose to implement all sound playback as streaming sounds, since iOS devices use solidstate disks, seek time is negligible. No samples are ever loaded completely into memory.

A single OpenAL audio source supports multiple buffers, load any number of buffers into the source and OpenAL will play through the buffers in order. When a buffer has been been played, detach the buffer, fill it with new data and queue it for the source. 3 buffers of 48k each works quite well.

When a sample is played, load and decode enough data to fill all the buffers (with IMA4 simply multiply the compressed size by 4 to get the uncompressed size) and start the playback.

To handle the recycling of the buffers set up a second thread in the audio engine that runs every quater of a second looking for buffers that have been used on all playing sound sources and fills them.

Its a good idea to implement the system without IMA4 support first (hint: WAV files), get that working and then add decompression as a second step. For convenience leave the support for playing uncompressed files in, your content developers will probably get annoyed having to compress files simply for testing.

Converting files

The easiest way to compress your audio files to IMA4 is to the command line afconvert tool provided with every Mac. The converter takes just about any input format (including MP3) and writes a multitude of different codecs and file formats. In this case we use AIFC and IMA4.


afconvert -f AIFC -d ima4 input.wav output.ima4


Conclusion

Implementing OpenAL is straight forward and provides a great cross platform alternative to CoreAudio. Using IMA4 will greatly reduce the size of your application without using too much CPU for decompression.

Source code to the IMA decompression can be found here.

UPDATE: I’ve since discovered that this IMA decoder doesn’t work exactly like the apple implementation which means it causes a small amount of noise that is only audible when playing very quiet tracks. I suggest looking at the Apple sample here.

Motivation

I’m using libRocket to render the UI for Wooords on iOS. libRocket comes with great Python integration, that allows the programmer to do just about anything with it from script, however Python is a bit heavyweight for my needs on mobile devices, it’s more suited to the desktop or console usage.

I need to build a simple system where by menu flow and events can be triggered from RML (libRocket’s markup language, think HTML with extensions) so interfaces can be prototyped and mocked up without touching the c++ code – think live reload, fast iteration.

The Model-View-Controller pattern…

“ The pattern isolates “domain logic” (the application logic for the user) from the user interface (input and presentation), permitting independent development, testing and maintenance of each (separation of concerns).” – Wikipedia

The two parts I’m going to describe below are the Controller and the View, the Model is the internal data structures specific to your game.

Views and Controllers

In my case libRocket is managing the view, as defined in RML files and the c++ programmer is implementing the controller.

The libRocket document will send events to its attached controller, which will update your model (drink a potion, disable sounds, etc) and then update the view accordingly.

The engine should support a way of registering a named controller with a c++ class, that will be invoked at runtime. You could do this with an instancer, or have all your controllers permanently resident. When a screen is required the engine looks up the controller by name, instances it if necessary and requests libRocket to load the corresponding view (RML file). I recommend keeping the file name of the RML file the same as the controller name.

A number of screens do not require any special behaviour (think Main Menu, Pause Menu), you should consider providing a generic controller implementation that implements basic common functionality that the more complex menus could derive from.

Hooking into the event system

By default libRocket does not handle inline RML events. Adding the Python plugin allows you to use Python events, however we want to implement our own event handling. We need to register a custom event listener and its corresponding instancer. This instancer is invoked whenever libRocket parses an event in RML (onclick, onload, onkeydown, etc¹). You should pass in a pointer to your engines UI manager class (in my case UIContext), so that it can be notified when an event fires.

Then somewhere in your initialisation code you’ll need to tell libRocket about your EventListenerInstancer

After implementing the above code you should be able to define a onclick event on a button in RML as follows:

<button onclick=”hello”>My Button</button>

If you load this view and click the button, processEvent on your UIContext will get called with the text “hello”. In this way, you can generically send events to your UI system from RML files.

Basic control script

Sending text strings through to your UIContext is all very well, but you probably want to be able to have some basic control from RML.

The text strings coming through from RML can be any format you like, I chose to implement a simple parser that understands C like function syntax. This way I can separate commands with semicolons and pass as my arguments as necessary.

I’ve implemented 3 basic commands

• open(<menu name>): This opens a menu, by looking up the controller and associated view
• close([menu name]): Closes the named menu, if no menu is specified, it closes the menu this RML is associated with
• sendevent(<event name>, [menu_name]): Sends the event to the named menu, or the current menu if no name is specified.

The open and close functions are pretty self explanatory. The sendevent function simply calls the virtual method processEvent on the specified controller with a string. Its up to that controller to interpret the string as it sees fit. For example my pause menu calls sendevent(pause, game) in its onload event, which sends the pause event to the game controller whenever the pause menu is opened.

Conclusion

Hooking into the libRocket event system and providing a very simple scripting interface allows you to prototype and build your whole menu flow without touching c++ code. Once the menus are defined you can then go back and define custom controllers for each screen and add the required events for triggering more complex behaviours from the views.

Over the last couple of weeks I’ve been preparing for the libRocket open source release. One of the things that confused me for awhile was how to go about building FreeType for iPhone/iOS. After a fair bit of trial and error I came up with something that allowed me to be a universal i386/i686/arm6/arm7 binary, so thought I’d document it here.

Compiling the library

We have to compile the library four times, once for each architecture.

i386

$ ./configure CFLAGS="-arch i386"
$ make

This forces configure to use the i386 architecture and build the library. The built library is located at objs/.libs/libfreetype.a. We copy this to our top level folder and build the next architecture.

$ cp objs/.libs/libfreetype.a libfreetype-i386.a

x86_64

Similar build setup for x86_64, notice the addition of ”make clean”, we want to completely remove the i386 code.

$ ./configure CFLAGS="-arch x86_64";make clean;make
$ cp objs/.libs/libfreetype.a libfreetype-x86_64.a

arm6

Arm6 is used on iPhone 3G and earlier. There are quite a few arguments that need to be passed to configure to make it build for arm6. I’m going under the assumption that you’re targeting iOS 3.2 using gcc-4.2 as your compiler. If this is not what you want, please update the arguments below accordingly.

$ ./configure --prefix=/usr/local/iphone --host=arm-apple-darwin --enable-static=yes --enable-shared=no CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc-4.2  CFLAGS="-arch armv6 -pipe -mdynamic-no-pic -std=c99 -Wno-trigraphs -fpascal-strings -O2 -Wreturn-type -Wunused-variable -fmessage-length=0 -fvisibility=hidden -miphoneos-version-min=3.2 -I/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk/usr/include/libxml2 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk" CPP=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/cpp AR=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/ar LDFLAGS="-arch armv6 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk -miphoneos-version-min=3.2"
$ make clean;make
$ cp objs/.libs/libfreetype.a libfreetype-arm6.a

arm7

Arm7 is used on iPhone 3GS and newer. We do the exact same options as above, but pass arm7 as the architecture (remember to update the CFLAGS and the LDFLAGS)

$ ./configure --prefix=/usr/local/iphone --host=arm-apple-darwin --enable-static=yes --enable-shared=no CC=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc-4.2  CFLAGS="-arch armv7 -pipe -mdynamic-no-pic -std=c99 -Wno-trigraphs -fpascal-strings -O2 -Wreturn-type -Wunused-variable -fmessage-length=0 -fvisibility=hidden -miphoneos-version-min=3.2 -I/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk/usr/include/libxml2 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk" CPP=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/cpp AR=/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/ar LDFLAGS="-arch armv7 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.2.sdk -miphoneos-version-min=3.2"
$ make clean;make
$ cp objs/.libs/libfreetype.a libfreetype-arm7.a

Bringing it all together as a universal library

We now have 4 individual libraries. To combine them into a single universal library use the ”lipo” tool.

$ lipo -create -output libfreetype.a libfreetype-i386.a libfreetype-x86_64 libfreetype-arm6.a libfreetype-arm7.a

And thats it,  you can check which architectures are in a library with the -info argument.

$ lipo -info libfreetype.a
Architectures in the fat file: libfreetype.a are: armv6 armv7 i386 x86_64

So where was all this space going? I’d already disabled any static plugins that I didn’t need, leaving OgreMain, RenderSystemGLES, Plugin_ParticleFX, FreeType, OIS and FreeImage.

Taking a look at the iPhone dependencies folder, there were 2 libraries that jumped out at me, FreeImage (1.3MB) and FreeType (500kb).

I decided the quickest way to get my executable size down was to remove these two dependencies. Unfortunately my current code base relies on FreeType for fonts, so I set that aside as a second phase, however removing FreeImage should be trivial as all I needed to do was disable it in Ogre, rebuild and then provide a custom ImageCodec for the image types I used (only TGA).

I rebuilt Ogre and my application, disabling ZIP archives at the same time and I’m pleased to say it reduced the size of my executable by 1.5MB!

I’ve placed my custom TGA Codec in the Source Code section of this site, you may find it useful.

In the coming weeks I plan to remove FreeType and switch completely to Bitmap based fonts.

Recently I stumbled across the announcement of Ogre3d 1.7 and there were two features that were of great interest to me.

• MIT License
• iPhone support (OpenGL ES Renderer)

Switching to the MIT license means Ogre3d can be statically linked to an application without requiring your code to be open source, previously it was LGPL, which as far as I understand it that you didn’t have to release any code that linked with Ogre3d dynamically (not possible on the iPhone)

Adding a native OpenGL ES Renderer to means any code written for Ogre would run with little modification on the iPhone.

The Ogre3d engine technology is a lot more advanced than my own tech and the content pipeline is so much more mature. Ogre3d has a very large active community who have developed many plugins and addons and been used in a couple of high profile PC titles. The Ogre Team is also currently working on an OpenGL ES 2.0 implementation (used by the iPhone 3GS), with full shader support. Being able to run the game on iPhone, Mac and PC is a great advantage.

Being a more advanced engine means it probably wont be quite as quick as a custom built solution, however I should be able to spend a lot more time in game code, rather than engine code.

For me, its a no brainer and I’m sure my artists are going to enjoy the more streamlined pipeline.