Migration from Posterous required three major steps:

1. Exporting posts from Posterous
2. Importing posts into Octopress
3. Setting up redirection from Posterous to my new blog

Follow along below to back up your own Posterous blog.

Export posts from Posterous

This part was quite straightforward. I used the Backup tool in Posterous’s control panel to create a .zip archive of my blog’s data. There’s a delay of a few minutes to hours between requesting a backup and the backup becoming available, so check back periodically. Although Posterous shuts down tomorrow, it’s not too late to request a backup of your old Posterous space. According to the Posterous shutdown announcement, the backup feature will be available until May 31, 2013. If you haven’t done so already, back up your old space ASAP.

Once you have your backup, a bit of postprocessing is needed because Posterous’s generated XML in wordpress_export_1.xml appears to strip extended characters outside of 7-bit ASCII. Even simple things like M- and N-dashes might be replaced with “???”.

1
2
3

cd /path/to/space-[numbers, name, etc.]
cat head.xml posts/*.xml > fixed_export.xml
echo '</channel></rss>' >> fixed_export.xml

Import posts into Octopress or Jekyll

Importing, editing, and fixing links within dozens of posts by hand seemed rather tedious and boring, so I wrote a script in Ruby to do the job for me. This script loads the fixed-up RSS feed generated in the previous step, then generates files under source/_posts. Images within posts are downloaded into post-specific directories, organized hierarchically by date. Encoded versions of videos are downloaded from Posterous (this is likely to stop working on Apr. 30!). Links to other posts on the same blog will be adjusted to point to their new location on the new blog.

To use the script, save it as posterous_import.rb in your Octopress or Jekyll blog’s base directory, then run the script with the path to fixed_export.xml generated in the last step:

1
2

cd /path/to/new/blog
./posterous_import.rb /path/to/space-[numbers, name, etc.]/fixed_export.xml

Complete usage information is included in the script.

Set up redirection from Posterous

Posterous doesn’t explicitly support redirection to a new blog. It does, however, allow one to use a custom domain with a Posterous blog. Once you set up a custom domain with Posterous, all of your old XYZ.posterous.com URLs will redirect to the custom domain, which is expected to point to Posterous’s own servers. We can cleverly exploit this behavior to try to transfer search engine rankings to our new custom blog. Unfortunately there may not be enough time left for Google to crawl your old blog and find the redirections before Posterous shuts down.

Set up permalink redirectors

Since Octopress defaults to permalinks with dates, it’s necessary to redirect the top-level Posterous shortlinks to the correct location on the new blog. I added an option to posterous_import.rb to do just this (--links must be the first argument to the script):

1
2

cd /path/to/new/blog
./posterous_import.rb --links /path/to/space-[numbers, name, etc.]/fixed_export.xml

This will create a directory under source/ for each post in the Posterous backup. Within each of those directories an index.html file will be generated that contains a redirection to the post’s new location.

Redirect feed.xml to atom.xml

I recommend setting up a 301 redirection from /rss.xml to /atom.xml. Octopress generates /atom.xml, while Posterous used /rss.xml. The file formats may not be the same, but most, if not all, feed readers can handle all the major feed formats.

This will allow subscribed RSS readers to find your new blog’s feed (until Posterous stops redirecting tomorrow). I used the S3 management console to create the redirection for my blog; your own hosting solution will have its own method.

Point Posterous at the custom domain

Here’s where the magic happens. Using Posterous’s control panel, I clicked Spaces at the top, clicked the gear icon next to my blog’s entry under Your Spaces, then clicked Space Settings in the popup menu. The first entry on the settings page is “Name Your Space”. Click the big Edit button to the right of your blog’s name and URL. At the bottom of the new page, there is a “Custom Domains” section. Here you can tell Posterous your blog is hosted at the domain of your choosing.

Thankfully, the Posterous engineers decided not to verify that the domain we enter points to Posterous. Instead of entering a domain that points back to Posterous, we’ll enter the domain that points to our new blog. At this point, all requests to XYZ.posterous.com/url will be redirected to our new blog.

Verify it works

If you’re just completing this process now, it’s likely that Google and other search engines won’t crawl the redirections in time. However, since I migrated my blog several weeks ago, I can check to see whether my new blog has replaced the old one in search queries. I’m hoping that GoogleBot caches the 301 results after Posterous shuts down, for the sake of all those links on news sites to my old blog.

My old blog used to rank on the first page of results for some searches related to home automation. Sure enough, when I try these queries now, the new blog shows up instead, and my new blog’s traffic has risen to match the old blog’s levels. At this point the new blog is running smoothly.

The Automation Controller will be reduced from $325 to $250 for the same time period. Quantity discounts will also be adjusted; get in touch using the Contact Us page to place an order or to inquire about quantity discounts.

If you’ve been waiting to outfit your home or business with Kinect-based automation, this is a great opportunity to get started!

new in build 43 beta

• Hue bridge discovery and registration
• Basic control and identification of Hue lights and groups
• Automation rules – set Hue lights and groups based on Zone events
• Minor page layout improvements
• Fixed a bug that can occur if the power fails at just the right time

There’s a good bit of work left to do to get it ready for production, but I just tested the first implementation and it works great! I welcome your feedback on the UI design for automation rules via the Nitrogen Logic Contact Us page.

More details to come in the future…

You can update your Depth Camera Controller’s firmware by clicking the Check for updates… link on the controller’s Settings page, downloading the appropriate file, then uploading it to the controller using the firmware update section of the Settings page.

new in build 35

• Added a confirmation prompt before deleting a zone.
• Tweaked the page layout for improved consistency.
• Added Surface Area to the zone attributes displayed in the web interface (this should have been there a long time ago; Surface Area values were already calculated and reported through the TCP/IP interface on port 14308).
• Fixed the display of zone attributes when a zone is added in another browser window or from a different device.

So what have you been doing all this time?

I’m pleased to announce the release of version 1.0 of a new application, the Nitrogen Logic PC Remote. This application allows you to run commands on any Java-enabled PC (Linux, Mac, or Windows) in response to zone events from one or more Depth Camera Controllers. Don’t worry, you can still disable the Java plugin in your browser.

I wrote this application to allow the Depth Camera Controller to interface with home automation systems that use PC command lines to deliver events to the automation software. One example of such a system is iHouse Home Automation. Thanks to a dedicated user’s feature requests and beta testing, this application is now available for the benefit of all Nitrogen Logic users.

What can I do with the Nitrogen Logic PC Remote?

The application allows the creation of conditional “triggers”, or command lines that will be executed in response to a certain parameter (such as whether a zone is occupied, or a zone’s surface area) reaching a defined value. The command line’s parameters may include the controller address, zone name, parameter name, and parameter value that triggered the command line, through the use of substitution variables (e.g. %ipercent% anywhere in a command’s argument will be substitued with the triggering parameter’s value as a percentage of its defined range, rounded to the nearest integer).

By building a list of triggers for the desired zones and parameters, you can control a command-line-based automation system using a Depth Camera Controller. In addition to controlling an automation system, you can also do things like start a video player when someone walks into a zone, shut down a computer when someone leaves a zone (if you are running the application as a user with shutdown rights), run custom scripts on the PC in response to changes in the room’s ambient light level – basically anything you can do from a PC’s command line can be triggered by zone events from a Depth Camera Controller.

The Nitrogen Logic PC Remote is also designed with security in mind. It uses user-defined command lines and connects directly to a Depth Camera Controller (instead of the other way around) to minimize the risk of commands being run by any malicious device or application that finds its way onto your private network.

Where do I get it?

The Nitrogen Logic PC Remote can be downloaded for free from its product page. If you find any interesting uses, bugs, or cool ideas for the Nitrogen Logic PC Remote, please be sure to get in touch.

Customers who purchased a “prototype” Depth Camera Controller can install the latest firmware update, build 31, to get exactly the same software and features as the retail Depth Camera Controller.  Click the “Check for updates” link on your controller’s Settings page, or contact Nitrogen Logic, to obtain this new firmware version.

new in build 31

• Very minor improvements to web interface layout.
• Fixed slow loading times in the experimental WebGL view (accessible from a link in the Depth Camera Controller’s main setup page footer).
• Event log is preserved across controller reboots.

Things to look forward to in the new year: easier ways to buy a Nitrogen Logic controller, Philips Hue support, and more.

Have a great start to a great year!

1. Interested party provides hardware they want to control on a temporary or permanent basis.  Control documentation for the hardware must be available.
2. Nitrogen Logic makes the necessary changes to the controller firmware.
3. The new firmware version is released, and temporarily provided hardware is returned.

With ambient light values included in xAP messages, customers using HomeSeer, Loxone, and other home automation systems dependent upon the UDP-based xAP protocol can now use relative brightness levels in each zone to trigger automation actions.

The event log provides an overview of recent activity.  You can see camera disconnections and timeouts, zone entry and exit events, xAP status changes, and firmware update attempts.  One great use for the event log is finding the exact XYZ coordinates where a piece of furniture is accidentally activating a zone.

Update instructions will be e-mailed to existing customers.  Read on for a detailed list of changes, complete with pretty pictures!

new in build 23

• Ambient light sensing in xAP and the web interface
  • Ambient light values are now included in the Level field of xAP messages, with an adjustable update rate:
    
    
    
    A sample message:

    xap-header
    {
    v=12
    hop=1
    uid=FFEDB20A
    class=xapbsc.event
    source=n2l.depth.theater.cam:Lightbright
    }
    input.state
    {
    State=OFF
    Level=150/1000
    }

  • Ambient light values are displayed in a zone’s information box on the web interface:
    
  • Ambient light sensing zones are visible on the Video tab (browsers without WebGL support will see a black-and-white image).
    
    Note: the video camera has a different field of view from the depth camera, so a zone’s ambient light sensing and depth sensing do not cover the exact same area of the room.
• Event log
  • The last 100 camera status and zone entry/exit events are visible in the newly added event log:
    
  • The center of gravity listed in zone events can help identify and correct zones experiencing intermittent spurious activation.
• Performance improvements and other minor changes
  
  • Improvements allow the creation of more zones with more activity, while still maintaining an acceptable frame rate.
  • The Overhead, Side, and Front views now use a crosshair cursor to indicate that a zone can be created by clicking and dragging.

analysis

The Depth Controller web interface uses the same text-based protocol on port 14308 that is made available to users for custom integration.  As such, every time a zone’s contents change, the web frontend has to parse a line of key-value pairs describing the change.  The Zone.new line measures the total amount of time spent parsing zone information from the backend.  This total includes time spent in kvp, which is the step of parsing a line of key-value pairs into a Ruby hash, and Zone.normalize!, which makes sure all of a zone’s attributes have the right data type.

The numbers show that, even with the web interface open and watching the overhead view, zone parsing is taking more time than either of the image processing steps for generating the overhead view (41.39% for zone parsing vs. 22.96% and 20.03% for overhead generation and compression)!  This code had already gone through a couple of rounds of optimization before it ever reached the public, but has become a bottleneck once again.  It looks like finding a way to attack both kvp and Zone.normalize! will knock out almost 29% of that 41% total.  Additionally, the get_zones task, which executes rarely but runs for 110ms straight, includes some zone processing, so improving the speed of Zone.new will also shorten get_zones and cut down on intermittent jitter.

To understand how to remedy the situation, I prepared benchmarks of each of the methods I could use to speed up kvp.  In addition to different kvp implementations, I tested JSON, YAML, and Ruby’s eval function, as well as a standalone Zone class that didn’t rely on hashes at all.  After some testing, I wrote a simple state machine-based version of kvp in C that avoided all use of regular expressions.  It also parsed integers, floating point numbers, and strings directly, thus eliminating the need for Zone.normalize!.  These are the results on my desktop i7 (ARM is slower, but the proportional relations hold):

conclusion

The C-language state machine-based kvp is the clear winner, resulting in as much as a 6.5x speed improvement.  With all overhead accounted for, the new key-value parser can process zone updates 2.4 times faster than the original code.  The final word will come from the results on the controller itself:

There you have it.  With the new and improved key-value pair parser, Zone.new finds its rightful place below the CPU-intensive image processing tasks, and the web UI feels much snappier.  We gained nearly all of our expected 29%.  All this was accomplished without changing the backend protocol.  There are still optimizations that could be made in the new kvp code, but if zone processing becomes a bottleneck again, I will switch to something like MessagePack (initial testing shows a further 2x speedup over the new kvp).

the problem

Since the introduction of color image support in the Depth Controller, the low-level camera server on the controller has been experiencing occasional hangs.  These hangs could be reproduced quite easily by switching to the Video tab on the controller’s web interface, then moving or resizing a zone.

The camera server’s built-in watchdog detects these hangs, resetting the depth camera when they occur.  Every time the camera is reset, there’s a several second delay while the software and camera reconnect, and sometimes the first few seconds of data from the depth camera are skewed by about half a meter.  With these hangs happening quite often when a zone was adjusted on the Video tab, it was clearly necessary to do something about the problem.

the investigation

Based on the symptoms, I came up with a few possible explanations:

1. The cause might be within the Kinect itself, that repeatedly stopping and starting its RGB camera was causing it to hang.
2. A bug in libfreenect was causing an invalid command to be sent to the camera.
3. The controller didn’t have enough USB bandwidth or CPU power to keep up with all of the depth and video data, causing data to be lost.  The resulting data loss caused the camera to hang.
4. A race condition between the server’s network thread and camera processing threads was leading to a deadlock.

The only way to narrow down the cause was to gather more information.  I started by adding better logging facilities to the camera backend that would also allow me to dump low-level debugging information from libfreenect.  If I could find a common set of events that always happened right before the camera stopped, I would most likely eliminate hypothesis #1.  The massive amount of data generated by the new logging features made debugging directly on the controller untenable.

Hypothesis #1 seemed unlikely because the bug was associated with specific behavior on top of video support.  One might also expect to see USB disconnection messages in the kernel dmesg output; the lack thereof further discredited hypothesis #1.

I upgraded to the latest version of libfreenect, in case the problem was #2, a libfreenect bug that had been solved upstream.  The updated version of libfreenect looked like it supported a 320x240 mode (the libfreenect.h header defines a constant called FREENECT_RESOLUTION_LOW), which would use 25% as much data as 640x480.  I was hoping that by switching to 320x240 the problem would go away, confirming hypothesis #3.  Unfortunately libfreenect doesn’t actually support a low resolution mode, and there are some claims that the Kinect itself lacks a 320x240 mode (despite the Xbox 360 press indicating that the Kinect’s launch resolution was 320x240, with a later software update to obtain 640x480).

I reasoned that if I could reproduce the bug on my desktop PC I would eliminate hypothesis #3, so I ran a pair of active USB extension cables from my desktop to the Kinect.  At first the bug did not occur on my PC, even when I opened multiple browsers to the Video tab and thrashed dozens of zones around like a madman:

I tried using valgrind’s thread checking tools to identify the problem, but they made the camera server run so slowly that the watchdog timer fired constantly and the interface was unusable.

Meanwhile, I was still working on further improving performance to allow more zones to be created and monitored, trying to get the Depth Camera software working on the Raspberry Pi, and looking for other ways of reducing USB bandwidth requirements.  In every case I ran into a dead end.  The performance “improvements” turned out to be neutral, I found the Raspberry Pi is incapable of working with the Kinect, and as mentioned before, 320x240 support was out of the question.

All this discouragement over several days was getting rather annoying, so I spent quite a few hours playing the awesome Black Mesa Mod to clear my mind and get a fresh perspective.

the cause

I kept thinking about hypothesis #4 — if indeed there was a race condition on the controller under a certain amount of stress, maybe I could trigger the same condition on my PC by running a stress test.  Since the hang would occur while simultaneously watching the Video tab and moving a zone, I opened my browser to the Video tab, then ran the following command to simulate rapidly changing a zone:

$ (while true; do echo 'setzone 34,xmin,-1.589'; done) | nc localhost 14308

Success!  In less than a minute the hang was triggered.  Hypothesis #3 was eliminated.  But wait, I forgot to enable the detailed logging features!  I produced a debug build, set the appropriate environment variable to enable ultra-verbose logging, and started gdb (the low-level camera backend process is called knd):

$ KND_LOG_LEVEL=10 gdb ./knd

In order to get an exact snapshot of the program at the time of the hang, I set a breakpoint in the watchdog callback, automatically printing a stack trace of each thread at the instant the watchdog fires (there are five threads in the camera server):

(gdb) break watchdog_callback
(gdb) commands
Type commands for breakpoint(s) 1, one per line.
End with a line saying just "end".
> info threads
> thread 1
> bt
> thread 2
> bt
> thread 3
> bt
> thread 4
> bt
> thread 5
> bt
> end

Then I launched the camera server with the run command, reloaded the Video tab in my web browser, and started my stress test.  Within minutes, I had logs and stack traces for several hangs.  It was time to look for a pattern.

Every trace was the same, with the obvious exception of timestamps and thread IDs.  Here are the first few lines from a trace (manually colored for clarity and visual appeal):

2012-10-05 03:53:00.992642 -0600 - main_thread - Camera spew: Got video frame of size 307200/307200, 162/162 packets arrived, TS a5ffa17f
2012-10-05 03:53:00.993029 -0600 - video_thread - Camera debug: Write Reg 0x0005 2012-10-05 03:53:01.941985 -0600 - watchdog_thread - knd.c:89: watchdog_callback():  Timed out: at least 0.949313668s since last update.
[Switching to Thread 0x7ffff6934700 (LWP 9212)]

Breakpoint 1, watchdog_callback (data=, interval=0x7ffff6933e80) at knd.c:91
91              if(info->stop) {
  Id   Target Id                                        Frame
  5    Thread 0x7ffff5004700 (LWP 9215) "kndsrv_thread" __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:136
  4    Thread 0x7ffff5805700 (LWP 9214) "video_thread" pthread_cond_timedwait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:216
  3    Thread 0x7ffff6006700 (LWP 9213) "depth_thread" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
* 2    Thread 0x7ffff6934700 (LWP 9212) "watchdog_thread" watchdog_callback (data=, interval=0x7ffff6933e80) at knd.c:91
  1    Thread 0x7ffff7fbd720 (LWP 9211) "main_thread" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86

Every single hang was preceded by the same event, a write of 0x00 to register 0x0005 on the camera.  According to the OpenKinect.org Protocol Documentation, writing 0x00 to register 0x0005 disables the RGB camera stream.  It seems there’s a problem at the point of disabling the video feed after taking a snapshot.

It’s also notable that each thread is in the same function every time, and apart from the watchdog thread, they are all lock-related functions.  Hypothesis #4 grows ever stronger.  The next step is to identify which locks are held by which threads (Java makes this much easier than C, but this isn’t Java).  We will pay particular attention to the main and video threads, which were the last two threads to print any data.

We’ll need to compare the stack trace from each thread to look for common ancestors, then read the source code near each entry to identify mutexes and semaphores waited for or held by each thread.  Here are the traces, manually colored and annotated with lock information:

Thread 1 — main_thread

Holds something in libusb.
Waiting for the video_empty semaphore.

#0	sem_wait ()
#1	video_callback (dev=, videobuf=0x644880, timestamp=2784993663) [waits for the video_empty semaphore, locks the video_in_use mutex, posts the video_full semaphore]
#2	iso_callback (xfer=0x641ee8)
#3	?? ()
#4	?? ()
#5	?? ()
#6	libusb_handle_events_timeout ()
#7	freenect_process_events_timeout (ctx=0x63b750, timeout=)
#8	freenect_process_events (ctx=)
#9	vidproc_doevents (info=0x63b580)
#10	main (argc=, argv=)

Thread 2 — watchdog_thread

Happy as a clam.

#0	watchdog_callback (data=, interval=0x7ffff6933e80)
#1	watchdog_thread (d=0x63b3e0)
#2	start_thread (arg=0x7ffff6934700)
#3	clone ()
#4	?? ()

Thread 3 — depth_thread

Holds the depth_empty semaphore.
Waiting for the depth_full semaphore.

#0	sem_wait ()
#1	depth_thread (d=0x63b580) [waits for the depth_full semaphore, locks the depth_in_use mutex, posts the depth_empty semaphore]
#2	start_thread (arg=0x7ffff6006700)
#3	clone ()
#4	?? ()

Thread 4 — video_thread

Holds the video_in_use mutex.
Holds the video_empty semaphore.
Waiting for something in libusb.

#0	pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1	libusb_wait_for_event ()
#2	libusb_handle_events_timeout ()
#3	libusb_handle_events ()
#4	libusb_control_transfer ()
#5	fnusb_control (dev=, bmRequestType=, bRequest=, wValue=, wIndex=, data=, wLength=12)
#6	send_cmd (dev=0x63ba00, cmd=3, cmdbuf=, cmd_len=4, replybuf=0x7ffff5804e30, reply_len=4)
#7	write_register (dev=0x63ba00, reg=, data=)
#8	freenect_stop_video (dev=0x63ba00)
#9	video_thread (d=0x63b580) [waits for the video_full semaphore, locks the video_in_use mutex, posts the video_empty semaphore]
#10	start_thread (arg=0x7ffff5805700)
#11	clone ()
#12	?? ()

Thread 5 — kndsrv_thread

Waiting for the video_in_use mutex.

#0	__lll_lock_wait ()
#1	_L_lock_883 ()
#2	__pthread_mutex_lock (mutex=0x63b670)
#3	request_video (info=0x63b580) [locks the video_in_use mutex]
#4	getbright_func (client=0x7ffff7e07010, command=, argc=, args=)
#5	parse_line (line=0x7ffff7e1f4ed "getbright", client=0x7ffff7e07010)
#6	parse_buffer (count=131072, client=0x7ffff7e07010)
#7	knd_read (arg=0x7ffff7e07010, buf_event=)
#8	knd_read (buf_event=, arg=0x7ffff7e07010)
#9	bufferevent_readcb (fd=19, event=2, arg=0x643ab0)
#10	event_process_active (base=0x63a8f0)
#11	event_base_loop (base=0x63a8f0, flags=0)
#12	event_base_dispatch (event_base=0x63a8f0)
#13	kndsrv_thread (d=0x61a1a0)
#14	start_thread (arg=0x7ffff5004700)
#15	clone ()
#16	?? ()

Even though the traces always have the same functions at the top of the stack, the final trace differs occasionally in the exact call stack. Specifically, most of the time the kndsrv thread was processing a video image for the getvideo command, but sometimes it is handling a getbright command instead. Both commands need to access video imagery.

Also interesting is the fact that two threads have libusb_handle_events_timeout() in their call stack; one under freenect_process_events() where you might expect it, and another under freenect_stop_video().  You may recall that every hang happens right after the camera is told to stop the video stream.

Anyway, as you can see, we have a classic deadlock: Thread 4 (video) holds video_empty, needs something in libusb.  Thread 1 (main) holds something in libusb, needs video_empty.  The other threads are operating as expected, waiting for work that will never come.  Hypothesis #4 is confirmed.

the solution

When working with third-party libraries in threaded code, it’s easy to run into concurrency issues where they aren’t expected. It is clear that libfreenect is not reentrant. One solution would be wrapping all access to libfreenect in a mutex, to ensure that only one thread at a time tries to access libusb. But this is rather like fighting fire with fire — you have to plan very carefully to ensure your fire breaks are in the right place. Better to avoid concurrent and recursive access altogether, in this case by moving all calls to freenect_*() functions into a single thread, outside of any libfreenect callbacks.

After building and uploading a new firmware image with the new changes and reconnecting the Kinect to a Depth Camera Controller, video support is fully functional and the hangs are nowhere to be found.  Mission accomplished.

The SheevaPlug has a fast enough CPU, excellent USB support, and gigabit Ethernet, but no GPU.

The D2Plug includes a GPU (with unknown driver availability), but costs too much.

The older Beagleboard had good enough USB support, but the USB bus sometimes died on Rev.C3 boards, it has no network interface, a slower CPU, difficult GPU drivers, and a higher price than the SheevaPlug.

The Raspberry Pi has a slower CPU, but a good GPU and a great price.  Unfortunately it only has 100mbit Ethernet attached via USB and terrible USB support (can’t even get a single coherent frame from the Kinect).

The Odroid-X has loads of CPU power and a reasonable price for what it provides, but USB-based 100mbit Ethernet.

The BeagleBone looks promising, but I wasn’t able to find GPU drivers as easily as for the PandaBoard or Raspberry Pi.  It also requires a separate power supply.

It seems the platforms with good I/O (e.g. the Marvell platforms with gigabit Ethernet, good USB support, eSATA, and even PCI express) either lack GPUs or are priced too high, and the platforms with good GPUs lack drivers and satisfactory I/O (e.g. Raspberry Pi).  Someone please make a low-cost ARM system with real (not USB-connected) Ethernet, solid USB 2.0 or 3.0 support, an enclosure, and a GPU with easily accessible drivers!

I began this journey in May of 2009, when I left school and started down what would be the most enlightening, and most terrifying path I’ve ever traveled. I’ve had the excitement of seeing satisfied users putting my products to amazing uses.  I’ve felt the pain of trying (but always succeeding) to make ends meet during leaner times.  Last year, what was meant to be a one-off hack with the Kinect turned into 15 minutes of fame and, a few months later, my primary product line.  I’ve put everything I’ve got into this vision. Nitrogen Logic is my brainchild, my baby.

I have learned and built a lot during this time.  I’ve learned that working with a team can significantly amplify one’s potential.  I’ve learned what it’s like to have an unexpected publicity breakthrough, only to fade back into the shadows.  Most importantly, I’ve learned that there’s a big difference between having a vision, and communicating it.  In other words, I am not a salesman.

Now, over three years down the road less traveled, I come to a fork.  When I started working on Nitrogen Logic, consumer automation appeared to be stagnant. Since then, products and services like IFTTT (If This, Then That) and Belkin’s WeMo, and Kickstarter projects like Ubi and SmartThings have filled some of the niches I set out to address.  Additionally, home security firms and ISPs are jumping on the home automation bandwagon.  I find myself running into the limits of the processing power and USB bandwidth available on low-power controllers.  Meanwhile, there are recruiters e-mailing, companies growing, and opportunities knocking.  The temptation to rejoin the ranks of the well-paid, salaried workers with strange and unfamiliar benefits like “dental insurance” grows.

As each month goes by, one question looms ever larger in my mind: Do I continue with Nitrogen Logic, seeking new customers and partners to keep the vision going?  Or do I move on, put Nitrogen Logic on the back burner, and look for another way to make my vision a reality?  The answer to this question may lie with you, dear readers, customers, and fans.  Should I shut everything down? Switch to selling software?  Go open source?

Have you been inspired by my Kinect hacks?  Are you a current or future customer of Nitrogen Logic?  Do you work for a company interested in adding automation logic to your products?  As I ponder the future of Nitrogen Logic, I welcome your opinions and advice.

Video support has been a long time coming, mostly due to potential privacy concerns.  To help mitigate privacy issues, this firmware version makes use of the Kinect camera’s LED to indicate whether images are being captured.  If the LED is green, the system is in standard control mode.  If the LED is yellow or red, depth or video images are being accessed, either through the web interface on port 8089 or through the low-level backend on port 14308.  If this is insufficient, future versions may add access control.

Support for ambient light sensing in the Automation Controller and xAP interfaces is on the way.  Existing customers should contact Nitrogen Logic to receive this firmware update.

Finally, I will be announcing an important change to the Nitrogen Logic product lineup in the near future.

new in build 20

• Video image support:
  • Ability to retrieve raw Bayer images from the controller backend on port 14308.

    help
    ...
    getbright - Asynchronously returns the approximate brightness within each zone.
    getvideo - Grabs a single video image.
    ...

  • Ability to retrieve PNG-compressed Bayer images from the web interface on port 8089, at [controller-name.local]/video.png.
  • Ability to view full-color, demosaiced images in the Video tab of the web interface (browsers that do not support WebGL will see the raw Bayer image).
• Ambient light sensing:
  • Non-photometric brightness for each zone is available from the controller backend on port 14308.
    Example with lights off:

    getbright
    OK - Requested brightness for each zone
    BRIGHT - bright=4 name="Captains_Chair"
    BRIGHT - bright=3 name="Projector"
    BRIGHT - bright=4 name="Office_Door"
    BRIGHT - bright=4 name="Theater"
    BRIGHT - bright=4 name="Theater2"
    BRIGHT - bright=4 name="Theater3"
    BRIGHT - bright=5 name="Stairs"
    BRIGHT - bright=4 name="Laundry"
    BRIGHT - bright=4 name="Touchscreen"
    BRIGHT - bright=4 name="Theater_Bright"
    BRIGHT - bright=4 name="xAP_Stress"
    BRIGHT - bright=3 name="Lightbright"

    Example with lights on:

    getbright
    OK - Requested brightness for each zone
    BRIGHT - bright=398 name="Captains_Chair"
    BRIGHT - bright=378 name="Projector"
    BRIGHT - bright=557 name="Office_Door"
    BRIGHT - bright=420 name="Theater"
    BRIGHT - bright=400 name="Theater2"
    BRIGHT - bright=240 name="Theater3"
    BRIGHT - bright=240 name="Stairs"
    BRIGHT - bright=221 name="Laundry"
    BRIGHT - bright=392 name="Touchscreen"
    BRIGHT - bright=480 name="Theater_Bright"
    BRIGHT - bright=376 name="xAP_Stress"
    BRIGHT - bright=643 name="Lightbright"

• Kinect LED indicates image capture mode:
  • Green indicates normal zone detection mode.
  • Yellow indicates depth images are being accessed.
  • Red indicates video images are being accessed.

To view in 3D, zoom out using your browser’s zoom feature, then look through your monitor (not cross-eyed) until the red dots align, or you start to see 3D, sort of like a random dot stereogram.

how it was made

Update: The Curiosity rover has stereo pairs of cameras that help it avoid hazardous obstacles.  I placed the raw images from each stereo camera side by side, applied a partial lens correction distortion, adjusted the image levels, added the red alignment dots, then selected and colorized the sections of the image by hand using the Curves, Levels, Hue/Saturation, Color Balance, and Colorize tools.

As an aside, I will be looking for a new blogging platform soon, as Posterous has become increasingly unreliable since their acquisition by Twitter.  For example, the media upload and link creation interfaces in the post editor no longer work for me.  The embedded YouTube videos also seem to be missing from my older posts.  I welcome any suggestions.

Here are some of the changes in these new firmware versions:

automation controller v0.9.2-7

The Automation Controller firmware includes enhancements to existing logic objects, as well as the addition of one new processing object. There are also minor changes to the controller’s web status page on port 4567.

Parameter Control object

• The Parameter Control plugin was modified to preserve its exported parameters when its size is changed in Palace Designer.
• Added optional Reset input pins to Parameter Control objects, selectable using the Reset Pin parameter. This permits the use of automation logic to clear out an override value set via parameter, and restores the input pin’s value on the output pin.

Float Value object

• The parameter range of the Float Value plugin was changed from -FLT_MAX..FLT_MAX to -inf..inf.

Phyiscal Model object

• Added a Physical Model plugin to the Filters category. This object simulates physical attributes of a point mass on a single axis, such as position, velocity, etc. while moving towards a target value.
• The input pins control the target value, while the output pins reflect the current simulated position. There are limits imposed on the physical attributes (e.g. maximum velocity).
• Thus far, velocity is simulated. Acceleration will be added in the future.
• One intended purpose of this object is for light dimmers. Using a Physical Model object to control light brightness causes fades from very dim to very bright to take a long time, while fades between similar brightness levels happen quickly.

Multi-zone Value object

• Added X Proportion, Y Proportion, and Z Proportion attributes to the Multi-zone Value plugin that interfaces with the Depth Camera Controller.  These attributes result in output pin values between 0.0 and 1.0, based on the zones’ center of gravity in each dimension, regardless of the size of the zone.  The primary use for these attributes is 3D user control interfaces, such as the Gesture-like Light Control demonstration I released last year.

depth controller v19

Performance improvements have been made both to the controller backend and to the web interface on port 8089. Some changes were made to the low-level protocol on port 14308.

Web Interface

• Coordinate grid and cursor location overlay on Overhead, Side, and Front views.

xAP Support

• xAP support is much faster, so the controller will run reliably on a heavily loaded xAP network.
• xAP events are generated at the full camera frame rate, up to 30Hz, rather than at a fixed rate of up to 10Hz.

Backend

• The sub command now sends ADD and DEL events when a zone is added or removed.
• There is now a small hysteresis used when detecting whether a zone is occupied.
• The maximum zone reported by the zones command is now selected by surface area instead of pixel count.
• Some enhancements were made that should improve frame rates on systems with many zones.

The complete updated version of xap_ruby, including the new parser, is available on Github. Benchmarks follow.

updated parser - x64

$ ./parser_bench.rb 10000
Running 10000 iterations of ParseXap.simple_parse
10000 iterations: cpu=0.360000 clock=(0.363838)
27484.72689933285 per second

Running 10000 iterations of ParseXap.parse
10000 iterations: cpu=10.640000 clock=(10.647138)
939.2195243898797 per second

Running 10000 iterations of node.to_hash
10000 iterations: cpu=0.280000 clock=(0.279407)
35790.08087596572 per second

updated parser - ARM

$ ./parser_bench.rb 1000
Running 1000 iterations of ParseXap.simple_parse
1000 iterations: cpu=0.800000 clock=(1.048829)
953.4446317047145 per second

Running 1000 iterations of ParseXap.parse
1000 iterations: cpu=21.990000 clock=(29.733424)
33.63218476814936 per second

Running 1000 iterations of node.to_hash
1000 iterations: cpu=0.720000 clock=(0.837403)
1194.1684162215195 per second

conclusion

It appears that using a generic parser backend like Treetop is disadvantageous for simple parsing tasks like the xAP protocol. The upside of the Treetop-based parser is that it provides full validation of the message syntax, with better error messages for invalid xAP messages. This new parser will allow development of the Depth Camera Controller’s web interface to continue.

With a bit of profiling I discovered that the controller’s web server was spending all its time in the xAP support code, specifically the Treetop-based xAP parser.

xAP

First, a bit of background on xAP.  xAP is a horribly inefficient protocol for smart home devices that just happens to be supported by a lot of DIY home automation platforms.  Reasons why xAP is inefficient:

1. Everything is all broadcast, all the time.  Each xAP message is sent via UDP broadcast to the entire network, so all xAP devices and applications have to deal with every single xAP message on the network, even messages destined for other devices.
2. xAP messages are text-based. Parsing values from text can be slow.
3. xAP Basic Status and Control, the schema used by the Depth Camera Controller and supported by automation packages like HomeSeer, sends only a single binary and/or numeric value in each packet. That’s 100+ bytes just to send a single bit.
4. xAP has some unusual multi-device address wildcarding, which can be time consuming to process.

Reason number 1 is the probably the most significant factor in the Depth Camera Controller’s performance problems—every message we send, we also receive and have to process. However, we can’t be sure until we’ve actually tested the parser’s performance. So, I threw together a quick benchmark of just the xAP parser, and tested it on my PC and a controller:

original parser - x64

$ ./parser_bench.rb 10000
Running 10000 iterations of ParseXap.parse
10000 iterations: cpu=9.690000 clock=(9.696733)
1031.2751730063692 per second

Running 10000 iterations of node.to_hash
10000 iterations: cpu=0.280000 clock=(0.275799)
36258.3475033779 per second

For whatever reason, the benchmark ran significantly slower when run with realtime priority via chrt or schedtool (perhaps something to examine some other day). Here are the numbers for the controller itself:

original parser - ARM

$ ./parser_bench.rb 1000
Running 1000 iterations of ParseXap.parse
1000 iterations: cpu=19.690000 clock=(25.032248)
39.94846963781098 per second

Running 1000 iterations of node.to_hash
1000 iterations: cpu=0.660000 clock=(0.733073)
1364.121077297844 per second

Yikes! Some of the new features I’m working on, which were designed to reduce average CPU usage and response delays, could generate 300 (30fps times 10 zones) xAP messages per second or more. Obviously the frame rate will drop as CPU usage increases, but if we’re only parsing 40 messages per second at 80% CPU usage, it’s no wonder the system is getting bogged down. The EventMachine event queue in the controller’s web server is filling up with unprocessed xAP packets faster than it’s being emptied by the parser.

to be continued aftermath

With the problem identified, I have some optimizing to do. I’ll write a followup post detailing how I resolve the performance problem with the xAP parser.

Update Jul. 12, 2012: I have written a new trivial parser for xAP messages that improves performance by a factor of 28 or more. Details are in my next blog post (linked in the previous sentence).