• 10am – Ken Yarmosh Stop Pinching Me: Getting Gestures Right in Your Application
• 11am – Jon Blocksom iOS Concurrency: NSOperationQueue and Grand Central Dispatch
• 1pm – Mark Gerl Recipe for Dynamic Content
• 2pm – Mark Pospesel Enter the Matrix
• 3:30pm – Chris Brown Improving App Monetization Through Data
• 4:30pm – James Norton Embedding Lua Script in iOS

Mark Pospesel wrote up his thoughts on the conference here.

Sometimes perfection seems too much to ask. Don’t despair; rejoice at all the milestones you have already met. The following Feltz levels are defined:

1. Sits in the editor without harm.
2. Correctly syntax-highlighted
3. Successfully saves to disk
4. In CM
5. Compiles
6. Links
7. Execution reaches main()
8. Produces discernable output
9. Does something arguably correct before crashing
10. Runs to completion.

Last weekend (Friday and Saturday), I attended RubyNation 2008, the first regional Ruby conference in my area. The speakers were excellent and it was great to meet up with folks from the Ruby community who I haven’t seen in awhile. As always, I left feeling energized and excited about my Ruby projects.

There were two main themes of the conference: the ‘official’ one was Ceremony vs. Essence, while the ‘unofficial’ one was Test all the fucking time. The first theme was the subject of the opening and closing keynotes. The second came from Bryan Liles lighting talk on the first day and kept coming up afterwards (audience ‘hecklers’ would ask presenters how often they should test).

Agenda

Neal Ford’s talk on Ceremony vs Essence was one of the best talks of the conference and a great way to kick it off. He criticized our industry for failing to look backwards and leveraging the lore of the past. Then he proceeded to look even further back discussing Plato’s doctrine of ideas, Aristotle’s essential properties vs accidental properties, Galileo’s use of experimentation to verify ideas, and William Occam’s principle that the explanation of any phenomenon should make as few assumptions as possible (KISS). Don’t confuse essential complexity (we have a hard problem to solve) vs accidental complexity (we’ve made the problem hard to solve). Some of the new patterns he presented were:

• Chicken Cognition: no memory, can’t learn from your mistakes
• Frozen Caveman: burned by something and was flash frozen at that point in time, everything revolves around that issue
• Rubik’s Cubicle: love to solve puzzles, not real issues

During his keynote, Neal suggested learning from the lore of our past by reading classics like The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition), Smalltalk Best Practice Patterns, The Pragmatic Programmer: From Journeyman to Master.

Next up was Russ Olsen who gave a talk about DSLs, both external and internal. Domain specific languages continue to be a hot topic in Ruby and the developer community in general. Ruby is very well suited to both external DSLs (involving a parser) and internal DSLs (use the Ruby parser and build language infrastructure). Ruby is great for DSLs because it doesn’t require parenthesis or semi-colons, it supports procs, the reflection capabilities (method_missing, instance_eval, etc), etc. Rich Kilmer called DSL building, syntax driven development.

Glenn Vanderburg presented on the culture of innovation within the Ruby community. The Ruby community is seeing so much innovation because Rubyists are novelty junkies, the languages opens possibilities that deserve to be explored, the requirement to integrate with IDEs is hampering innovation in Java (Java developers won’t use it until it’s integrated within their IDE), and things are just easier to accomplish in Ruby (low ceremony). He said that sometimes its good to start over as being forced to reinvent can be an opportunity. To close out his talk he did have a warning: When learning new things costs are easier to spot than benefits, but when inventing new things benefits are easier to spot than costs.

Yehuda Katz from Engine Yard spoke about living on the edge. As an aside, it appears that Engine Yard is trying to be a mini-Google in that they appear to have hired many of the top Rubyists as well as some very savvy operations folks. Yehuda said that we should expect Merb 1.0 by the end of summer, introduced the DataMapper ORM, YARD (a replacement for rdoc), and Johnson (a JavaScript/Ruby bridge). He urged those present to take the time to think things through and not rush an API (or software) out.

During the Lightning Talks we heard from Chris Bucchere about BDG’s social network for conferences; David James of Community Goals about state machines, Simon Kaczar of Prognosoft about libncurses-ruby, and Bryan Liles about testing (where the meme test all the fucking time appeared). Bryan’s was one of the best talks of the conference, I highly recommend seeing him present.

The first day closed with Giles Bowkett’s presentation on Archaeopteryx (a Ruby MIDI generator). The music it generated (through Reason) was pretty good. The software makes heavy use of lambda functions and so Giles has aliased lambda to L to reduce typing. His talk covered a wide range of topics and there’s no way I can do it justice. Hopefully he’ll have it recorded (since he’s given the presentation a few times now) and post it on the ‘net.

The second day opened with Rich Kilmer of InfoEther who provided a look back at the early days of Ruby (from his involvement in 2001), the conferences, and all of the various projects he’s worked on with Ruby. Rich always has fun stories to tell and I enjoyed his talk immensely. I won’t go through the entire chronology, but he ended by stating that the community needs to develop tools to deal with the dynamic nature of Ruby in order for wider adoption of the language, though he did note that Ruby is becoming mainstream. He also announced HotCocoa, a DSL for Cocoa application development.

Mike Furr spoke about issues he encountered when writing a Ruby parser for his DRuby project. These were somewhat interesting traps, pitfals, and corner cases for where the language is ambiguous. His slides are available here so you can see specific code examples.

David Keener walked through installing, configuring and running JRuby. Unfortunately, I have no interest in JRuby. I started working with it at my previous job to add Ruby scripting to a Java application, but since I no longer program in Java nor have an operational environment that is setup to deploy Java, I just don’t care.

Tools for your Ruby toolbox by David Bock was an interesting tour of three Ruby projects: StaticMatic, Sinatra, and GServer. StaticMatic looked interested for static website development, but then Mike Clark wrote about Webby and it looks even better. Sinatra appears to be a very light weight alternative to Rails and in combination with a static web site tool like StaticMatic or Webby could be extremely useful. GServer is a generic server which takes some of the pain out of setting them up. Dave’s an engaging speaker and I always enjoy his presentations as well as speaking with him in the hallways at conferences.

The second day of Lightning Talks was even better than the first. Soren Burkhard from Hawaii Business Consulting had a great overview of starting your own business. Glenn West spoke about several Rails plugins that he uses on a regular basis: ActiveScaffold, Role Requirement, ActiveAuthentication, File_column, PDFtoRuby, DotR, and Tabnav. Jesse Shock from Engine Yard spoke about high availability planning: multiple sites, geographic diversity, data partitioning, sharding, etc. He said you need to establish costs up front so the customer knows which availability techniques they can afford. Yehuda Katz, also from Engine Yard, spoke about testing (in the context of Merb) and how you should use BDD for internal as well as external interfaces. The last of the lightning talks was by Bryan Liles on how to be a manager, his slides are available here.

Finally, Stu closed out the conference with a talk on bad Ruby. Overall, Ruby is good but that there are some bad practices/decisions that could come back to bite us later as Ruby adoption grows. Some of the examples he provided were: constants (you can’t change them so testing becomes difficult), class attributes (prefer instance attributes on the eigenclass), direct access of instance variables, and proc workarounds (e.g. passing more than one block to a method). The code examples from his talk are available here.

Luis de la Rosa was kind enough keep track of all of the URLs that folks were throwing around and has posted them here: – delicious: http://delicious.com/tag/rubynation – magnolia: http://ma.gnolia.com/tags/rubynation

Overall, it was a great conference and I look forward to next year’s RubyNation in June 2009.

• BuzzHubb: A Better College Social Network. Next generation group management for colleges and universities (it is similar to Facebook in that regard &emdash only students can join).
• Heekya: The Wikipedia For Stories. Social story telling using multiple forms of media (text, photos, video).
• JamLegend: Guitar Hero Goes Social. A free online music gaming experience.
• Koofers: Crib Notes For Picking College Classes. A tool to help students build their class schedules with insight into the classes, teachers, and styles as reported by other students. It also provides access to study guides, past exams, etc. to assist students.
• MyGameMug: Match.com For Gamers. A personality test for gamers &emdash find others who like the kinds of games that you do.
• Razume: Resume 2.0. A tool for young (21-35) job seekers. This looks really interesting to me because I had a similar idea several years ago but never executed on it. I’m curious to see how its features overlap with my detailed plan.
• ShareMeme: Evite Meets Twitter. This service is addressing a real issue that is affecting me: being (inadvertently) spammed by my friends as they sign up for social networks and other services. If it succeeds it can be the go to point for distributing information out to your friends and family through the best channels.
• Zadby: Web Video Product Placement. Zadby wants to be the market maker for product placement in web video, through its network of independent web video producers.

Miami-based MyGallons counts a membership of more than 4000 and has offices in Philadelphia and Columbus, Ohio. It launched a pilot program in April. Verona estimates that the testers paid about $3.10 a gallon for gas, saving roughly a dollar over today’s prices. The debit card system measures consumers’ accounts in gallons of regular unleaded fuel. Pumping premium fuel or diesel debits more gallons, but if you find a station with fuel that’s cheaper than your purchased price, your account is credited with additional gallons. Still, there are some potential pitfalls, including the obvious question: What happens if gasoline prices go back down? MyGallons’ suggestion is to stock up; buy low and use it later if prices increase, though it remains to be seen if consumers would adapt to that kind of purchasing behavior.

It seems like an interesting service to help defray higher energy prices.

In related news, I finally received approval from Apple for my iPhone developer cert, so now I can try out my audio application on my iPhone and potentially sell it via the App Store.

Next he demonstrated how to interact with the DOM and JavaScript. The JavaScript integration was in both directions, first he demonstrated calling JavaScript functions from Objective-C and then calling Objective-C from JavaScript. It should be noted that for JavaScript to call Objective-C you need to implement the + (BOOL)isSelectorExcludedFromWebScript:(SEL)aSelector; method and return NO for each of the methods you wish to call.

Finally, Jason walked through various uses of WebKit an demonstrated an application specific, full-screen browser he wrote for a game as well as a del.icio.us Cocoa application he wrote called Delish. Several other webkit based applications were also covered such as Fluid – a freely available site-specific browser creator, PackRat – a commercial site-specific browser for Backpack, and another application which I’ve forgotten (I guess I should’ve taken notes). As an aside, Jason brought up the interesting work that the 280North folks have done for their 280Slides web-based presentation called Objective-J. Objective-J (code named Cappuccino), is an Objective-C like language built on JavaScript.

Matt Wizeman stepped up next to demo a WebKit application he’s developing to wrap a troublesome time tracking web application. It was interesting the machinations he had to go through to know when the page had finished loading and to transition between pages since the application was JavaScript heavy, using tricks like clickable divs.

I learned quite a bit about WebKit. As soon as I got home I was energized to play with it.

Be sure to come to the next meeting on July 10th (day before iPhone 3G day)!

Thoughts on the makeup of the group include obtaining (however you want) a book, working through the book 1 chapter per week on one night of that week with a group of like minded individuals.

To start, the group will work through Joe Armstrong’s Programming Erlang book starting on Monday, June 16th (hopefully I’ll be able to work out a schedule with my wife so I can attend). This is an excellent choice on several levels: every developer should know a functional programming language; single core processors are increasingly rare and the number of cores in commodity hardware should only increase in the coming years; and I already own the book.

If Erlang doesn’t pique your interest, the NSCoderNight DC group is going to work through the 3rd Edition of Aaron Hillegass’s Cocoa Programming for Mac OS X book on Tuesday nights. For several months now, I’ve been playing around with Cocoa and while I’m getting used to the syntax of Objective-C, the XCode IDE and Interface Builder still seem foreign to me (I never liked IDEs having been weened on Emacs).

It’s a great time to be a programmer in Northern Virginia!

Update: It turns out that there’s another Cocoa group right around the corner from me in Reston: CocoaHeads. They meet the second Thursday of each month.

Tonight I had the opportunity to attend a special Refresh DC meeting on the challenges of starting your own business. The format was a panel of folks from the DC startup scene moderated by Jackson Wilkinson of Viget Labs. The panel consisted of Brian Williams of Viget Labs, Andrew Lee of Publi.us, Eric Rupert from Odeo, Eddie Frederickof Hungry Machine, and Sean Greene of LaunchBox Digital.

These were the main points I took away:

1. Be passionate. The most important quality for startup founders is to be passionate about their product or service. Starting a company can be exhilarating but the setbacks can be really difficult. If you’re not passionate about what you’re doing each of these roadblocks will be an excuse for you to quit. Figure out what you’re passionate about and work on that.
2. Focus on your product. Don’t focus on the pie-in-the-sky potential valuations of your company or worry about leasing office space, hiring an attorney, finding an accountant, etc. Instead you should focus on the product or service you’re going to sell. Get something out there quickly — pick the most important feature and get it out there in front of your customers quickly.
3. Have your customers influence your product. If you follow the previous advice and get your product out there quickly you can use customer feedback to iterate your product. You may not know all of the difference ways customers will use your product, so this feedback is critical in charting your product roadmap.
4. Equity is control, don’t surrender it easily. One of the attendees asked about using equity to pay for services if money was tight. Brian said to avoid giving out equity as you make that person a partner in your business. While they may not have a controlling interest, they are still an owner and have some influence.
5. Don’t overfund. All of the panelists warned of the dangers of taking too much money as it does more than dilute the founders’ ownership. Venture capitalists are looking to make a large return o their capital for themselves and their limited partners. If you need to pay a vendor or contractor find another way — defer payment, use credit, etc.
6. Hire slow, fire fast. Early employees can make or break your company. You will be working long hours aside these folks so you must ensure they’re a good fit. If you make a bad hire, you need to resolve the situation quickly, don’t let emotion get in the way.

Some of the books recommended were: Art of the Start, Getting to Yes, and Founders at Work. Andrew also recommended Startup School run by Y Combinator (YC). I’ve been reading Paul Graham’s essays for around seven years now and following YC’s investments. I’m glad to see an early stage investor like YC in the DC area (LaunchBox Digital) and hope that the startup scene in DC and suburbs becomes more vibrant.

Thanks to Strategic Analysis for hosting this. It was a great venue and hope they’ll offer to host Refresh DC again.

This application is very specialized in that the two machines are connected directly through an ethernet switch. This means that the MTU could easily be determined from each end of the link and the extra work to determine the maximum segment size for the transit network (see RFC 1191) was unnecessary. This also made it very easy to watch the traffic between the two hosts as well as the system calls they were using to transfer and receive the data.

Before I get into the steps I took to tune the service, I’d like to introduce the tools used:

• Truss: a tracing utility which displays system calls, dynamically loaded user level function calls, received signals, and incurred machine faults. This is available for many platforms, but I use it most on AIX.
• DTrace/DTruss: a dynamic tracing compiler and tracing utility. This is an amazingly powerful tool from Sun, originally for Solaris but slowly spreading to other platforms. See Sun’s How To Guide.
• strace: a dynamic tracing utility which displays systems calls and received signals under Linux.
• mpstat: collects and displays performance statistics for all logical CPUs in the system.
• prstat: iteratively examines all active processes on the system and reports statistics based on the selected output mode and sort order.
• tcpdump: a utility for capturing network traffic.
• Wireshark: a network protocol analyzer. It replaces the venerable Ethereal tool and allows you to either capture network traffic on demand or load a captured session for analysis. Find out more here.
• gprof: a tool for profiling your code to determine where the performance bottle-necks are. See the manual for more information.
• c++filt: a tool for demangling C++ method names. It is part of the GNU binutils package.

Since I already had the service up and running, I simply ran the two components and captured the traffic between them using tcpdump. While the processes were running, I also used dtruss, truss, or strace (depending on the platform) to capture the system calls being made. Since this is a network service, I focused on calls to select, send, and recv.

13455/15:   2143177    2994      4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
13455/15:   2143180       5      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 1 0
13455/15:   2143185       8      4 recvfrom(0x11, 0xB384A0, 0x10000)                    = 1416 0
13455/15:   2143253       5      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 0 0
13455/15:   2143262      12      8 send(0x11, 0xB084D0, 0x14B8)                         = 5304 0
13455/15:   2143268     365      4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0
13455/15:   2143270       4      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 1 0
13455/15:   2143275       8      4 recvfrom(0x11, 0xB384A0, 0x10000)                    = 1416 0
13455/15:   2143343       5      0 pollsys(0xFFFFFD7EBADDB8D0, 0x1, 0xFFFFFD7EBADDB9F0) = 0 0
13455/15:   2143348       9      4 send(0x11, 0xB084D0, 0x14B8)                         = 5304 0
13455/15:   2143353    1000      4 pollsys(0xFFFFFD7EBADDB910, 0x1, 0xFFFFFD7EBADDBA30) = 1 0

Looking at the results above you can see that select (pollsys) is being called each time we need to send or receive data over the network. Since the socket is non-blocking we can rely on the immediate return when the outgoing socket buffer is full as well as when there is no data available to read. By selecting at the very top of the receive loop we can bundle multiple receive calls together, increasing the application’s throughput. Now the output looks like this:

16712/9:     16202    1560      6 pollsys(0xFFFFFD7EBB9DB940, 0x1, 0xFFFFFD7EBB9DBA30) = 1 0
16712/9:     16217      10      6 recv(0xB, 0x8A6450, 0x10000)                         = 1416 0
16712/9:     16246       9      5 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16267       7      3 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16285       5      1 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16680      10      5 recv(0xB, 0x8A6450, 0x10000)                         = 1416 0
16712/9:     16712      11      7 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16733       7      3 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16753       6      2 send(0xB, 0x876480, 0x540)                           = 1344 0
16712/9:     16768       4      0 recv(0xB, 0x8A6450, 0x10000)                         = -1 Err#11

You’ll notice that now we are able to process two requests and send out six responses in the time that it previously took to call select and receive a single request. When there is nothing left to read, the call to recv returns errno 11 (EAGAIN). This change made the single biggest performance impact on the code. I also changed the calls recvfrom to recv since the application did not make use of the foreign address.

At this point the performance was much better but I noticed that under heavy load the sending socket would block as the ratio of requests to responses was 1:3. As this was a UDP application, having the sending buffers fill up seemed strange as we assumed that additional packets would simply be dropped on the floor.

On the server, I checked the UDP socket buffer size using ndd (this was under Solaris. For AIX the command is no and for Linux the command is sysctl).

The following code was added to the socket initialize (minus the error handling) to ensure that the socket buffers were large enough.

unsigned size = 1024 * 1024; // 1MB
int ret = setsockopt(desc, SOL_SOCKET, SO_SNDBUF, &size, sizeof(size));
    ret = setsockopt(desc, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size));

Now that the application was performing acceptably I decided to run it under the profiler. This turned up the function which was adding responses to the in-memory packet. It turned out that as responses were being added to the packet, the headers were being recalculated each time. I removed this unnecessary work and only made the calculations right before the packet was sent. This improved performance a few percentage points more.

By binding the network interrupts to a particular core and keeping the sending thread off of that core we were able to eek out additional performance from the application. To accomplish this, the application allows the operator to specify which core(s) it should bind to using sched_setaffinity (Linux) and processor_bind (Solaris). You can also accomplish this using taskset (Linux) and pbind (Solaris) if you don’t wish to modify your application.

Looking at the network traffic with tcpdump, I saw that I could fit an additional response in the response bundle packet if I reduced or removed some of the items in the packet header. At this point the analysis and tuning had gone on for a few weeks and we had a schedule to meet. Since the performance was where we needed it, the application was wrapped up and sent to quality assurance.

The single most important lesson I learned from this exercise was to use non-blocking sockets to their fullest by continually calling recv/send until the call would block and then using select to idle the process until there is work to do.

Using AJAX with REST

Problem:

You have finally mastered REST (Representational State Transfer) and wish to enhance your application with JavaScript but don’t know which of the resource URLs to call.

Ingredients:

This recipe requires Rails 1.2 or later as it relies on RJS for the Ajax portion and REST for the controller design.

Solution:

In this example we need users to be able to register (and unregister) for meetings so that we can properly report attendance. So we have three models: User, Meeting and Registration.

Since Registrations are associated with a specific meeting, we’ve created them as a nested resource, like so:

~~~~ {lang=“ruby”} map.resources :meetings do |meetings|

meetings.resources :registrations

end ~~~~

To register for a meeting we want to create a new Registration instance tied to the Meeting and User. Unregistering from a meeting should destroy the Registration object. So our RegistrationsController looks like this:

~~~~ {lang=“ruby”} # POST /registrations def create

@meeting = Meeting.find(params[:meeting_id])
current_user.register_for(@meeting)
respond_to do |format| 
  format.html { redirect_to meetings_url }
  format.js   # create.rjs
  format.xml  { head :ok }
end

end

# DELETE /registrations/1 def destroy

@registration = Registration.find(params[:id])
@registration.destroy
respond_to do |format|
  format.html { redirect_to meetings_url }
  format.js   # destroy.rjs
  format.xml  { head :ok }
end

end ~~~~

In the view we want to use Ajax to allow the user to modify their registration, so we need to use link_to_remote:

~~~~ {lang=“ruby”} <% if !current_user.registered_for(meeting) –%>

<%= link_to_remote "Register!", :url => registrations_path(meeting), 
                              :meeting_id => meeting.id, :method => :post %>

<% else –%>

<%= link_to_remote "Unregister", 
                              :url => registration_path(meeting, 
                                         current_user.registration_for(meeting)),  
                              :confirm => 'Are you sure?', :method => :delete %>

<% end –%> ~~~~

The RJS to update the page looks like this:

~~~~ {lang=“ruby”} page[“ meeting_#{@meeting.id}”.to_sym].replace_html

   :partial => 'shared/meeting', :object => @meeting

page[“ meeting_#{@meeting.id}”.to_sym].visual_effect :highlight, :duration => 5 ~~~~

Discussion:

The resource URLs provided by Rails are extremely useful, but they can be confusing. Tools like FireBug make it easy to see what is going on behind the scenes. When developing Rails applications, it is usually easier to develop without Ajax and then go back and add Ajax to those portions of the application where it would make sense. With this approach, you can gracefully degrade your service back to the original mode for those who don’t have or don’t enable JavaScript. It also allows you to click through your application with FireBug enabled and watch which URLs are hit and with which methods (GET, POST, PUT, or DELETE).

The table below outlines which resource URLs to use with your remote helpers:

Intent Action Resource URL Method

Creating a new instance create plural form of model name (i.e. registrations_path) POST Retrieving an instance show singular form of model name with id (i.e. registration_path(registration) GET Retrieving all instances index plural form of model name (i.e. registrations_path) GET Modifying an existing instance update singular form of model name with id (i.e. registration_path(registration)) PUT Deleting an existing instance destroy singular form of model name with id (i.e. registration_path(registration)) DELETE

Once you are comfortable with the resource URLs, adding Ajax to your RESTful is much easier.

Further Reading:

If you are unfamiliar with REST, read Chapter 20 of Agile Web Development with Rails, 2nd Edition by Dave Thomas and David Heinemeier Hansson (Pragmatic Programmers, 2006). There is also a nice screencast on the subject from PeepCode.

For more on Ajax in Rails, you can check out Ajax on Rails by Scott Raymond (O’Reilly & Associates, 2007).

I was shocked when Jose explained that the Core Audio and AudioQueues tools are all Carbon (i.e. plain C) instead of Cocoa. Mistakenly, I was under the impression that Carbon had been officially deprecated by Apple and that only Cocoa would be supported going forward. The AudioQueueTools (/Developer/Examples/CoreAudio/SimpleSDK/AudioQueueTools) example which comes with the Apple Developer Tools shows how to create command-line tools for recording and playing back sounds, but trying to integrate this with you Cocoa application is a different story. Luckily, we found this post on Mark Darlymple’s blog which explains how to use C callbacks in Objective-C. I’m still playing around with this and also looking at the QuickTime API support for recording sound.

Quartz Composer is an amazingly cool app, that appears deceptively simply to throw something together with. I was blown away by Jose’s demo of it, having never seen it before. The visual programming environment, Quartz Composer Editor, has a fairly straight-forward GUI and allows you to easily ‘compose’ multiple processing units (a.k.a. patches) together into composition while the view window displays what the composition will look like in real-time. I’m definitely going to spend some time to go through the examples (/Developer/Examples/Quartz Composer/Applications).

If you are free on a Tuesday night, be sure to stop by the Tysons Corner Panera&sll=38.878131,-77.278595&sspn=0.009689,0.0156&ie=UTF8&om=1&ll=38.931038,-77.230368&spn=0.043267,0.079479&z=14&iwloc=addr&source=embed) from 7pm-9pm. I’m going to make every effort to attend regularly.

The idea of such a classification system both interests and appalls me quite a bit, so I’ve done some research into the 70 segments and 21 life-stage groups.

Even though this data is proprietary, I’ve found the following information through publicly available sources (ordered by cluster number):

1. Summit Estates: The wealthiest of all the clusters. In every sense, these families are enjoying the good life—luxury travel, entertainment and consumption of every kind are within easy reach. Middle-aged (36-56). (Group 11B – Boomer Barons)
2. Established Elite: Late middle-aged (46-65) married home owners with no children living with them. They are wealthy and live in suburban and urban locations. (Group 15M – Mature Wealth)
3. Corporate Clout: This exceedingly well educated group defines themselves as workaholics. Savvy investors who like to read Forbes and travel magazines, they are also heavy users of air travel and corporate credit cards. (Group 15M – Mature Wealth)
4. Skyboxes & Suburbans: One of the best-educated and wealthiest clusters. They shop at upscale stores, spend time feathering their nest and adhere to regular fitness programs. Middle-aged (36-55). (Group 11B – Boomer Barons)
5. Sitting Pretty: Late middle-aged (46-65), married home owners with no children in the home. They are wealthy, living in the outer-suburbs and towns. (Group 19M – Golden Years)
6. Shooting Stars: Still relatively young at a mean age of 36, and with top rankings for income, college education, home value and net worth, these consumers have the world by the tail. (Group 7X – Cash & Careers)
7. Leveraged Lifestyles: Middle-aged (36-55), married home owners with school-aged kids in the household. Primarily upper-middle income, living in the outer-suburbs and towns. (Group 11B – Boomer Barons)
8. Full Steaming: Late middle-aged (55-64), both single and married home owners with no kids in the household. Primarily upper-middle income, living in the outer-suburbs and towns. (Group 19M – Golden Years)
9. Platinum Oldies: Seniors (aged 66+), both single and married, who own their own homes with no kids in the household. They are wealthy, living in the outer-suburbs and towns. (Group 19M – Golden Years)
10. Hard Chargers: This affluent group of single men and women between the ages of 30 and 45 primarily own their own homes in the suburbs. (Group 7X – Cash & Careers)
11. Kids & Clout: Middle-aged (36-45) married home owners with school-aged children. They are affluent, living in the outer suburbs and towns. (Group 8X – Jumbo Families)
12. Tots & Toys: Work and family consume these middle-aged (30-45) couples. They’re putting their college degrees into action with lucrative careers, while saving for their children’s education. (Group 8X – Jumbo Families)
13. Solid Single Parents: Middle-aged (36-54) single parents who own their own homes with school-aged children. They are affluent and live in urban and suburban areas. (Group 12B – Flush Families)
14. Career Centered Singles: The affluent members of this segment are late middle-aged (46-65) and single. They own their own homes in urban and suburban areas and have no kids. (Group 16M – Aging Upscales)
15. Country Ways: Late middle-aged (45-65), married home owners with no kids in the household. Primarily upper-middle income in a rural location. (Group 16M – Aging Upscales)
16. Country Single: Middle-aged (36-55) single home owners with no kids. They are upper-middle income, living in rural areas. (Group 14B – Our Turn)
17. Apple Pie Families: Late middle-aged (46-55) married home owners with school-aged kids. Members of this group are upper-middle income, living in urban and suburban areas. (Group 12B – Flush Families)
18. Married Sophisticates: Recently married young couples who are well educated and enjoy health upper-middle range incomes. They are almost all homeowners in upscale suburban neighborhoods. (*[Group 2Y
    • Taking Hold](http://www.acxiom.com/subimages/1126200385918taking_hold_group_2y.pdf)*)
19. Country Comfort: Middle-aged (36-55), married home-owners who have school-aged children in the household (big families, family-oriented) make up this group. Primarily upper-middle income in a rural location. (Group 8X – Jumbo Families)
20. Dynamic Duos: Made-up of upper-middle income married couples who own their own home but don’t have children, the members of this cluster are between 36 and 45. (Group 7X – Cash & Careers)
21. Children First: At a mean age of 25, they are already raising an average of 1.2 kids. Despite a nearly even split between married and single households, 100% show the presence of children. They earn upper-middle incomes and live in suburban areas. (Group 2Y – Taking Hold)
22. Fun & Games: The upper-middle income members of this cluster are between the ages of 46 and 55, married with no kids, and own their own homes. (Group 14B – Our Turn)
23. Acred Couples: Late middle-aged (56-65) married home owners with no kids. These upper-middle income earners live in the outer suburbs and towns. (Group 16M – Aging Upscales)
24. Career Building: This cluster is made up of young, childless singles. They are a mix of mobile renters and first-time homeowners, living in condos and single-family houses. (Group 2Y – Taking Hold)
25. Clubs & Causes: Consisting of both married and single upper-middle income folks, the members of this cluster are aged 66 to 75 with no children who own their own homes. (Group 20S – Active Elders)
26. Savvy Singles: Young middle-aged (30-45) single renters and home owners with no kids. Primarily upper-middle income, living in urban and suburban areas. (Group 7X – Cash & Careers)
27. Soccer & SUVs: This cluster consists of upper-middle income married couples between the ages of 36 and 45 who own their own home and have school-age kids. (Group 8X – Jumbo Families)
28. Suburban Seniors: These aged seniors (76+) own their own homes and have no kids. They are a mix of single and married upper-middle income earners living in urban and suburban areas. (Group 20S – Active Elders)
29. City Mixers: Middle-aged (36-45) single renters and home owners with no kids. Primarily middle income, they live in downtown metro areas. (Group 9B – Boomer Singles)
30. Spouses & Houses: This cluster is dominated by middle-income, childless couples in their mid-20s. They are mainly high school grads who own their homes and tend to live in smaller second cities around the country. (Group 2Y – Taking Hold)
31. Mid Americana: Married suburbanites. They are middle of the road in terms of education and income, although long tenure and high equity versus home values result in high net worth. (Group 14B – Our Turn)
32. Downtown Boomer Couples: Late middle-aged (46-65) married home owners with no kids. They are middle income and live in downtown metro areas. (Group 14B – Our Turn)
33. Urban Tenants: Late middle-aged (46-55), single renters with no kids. Located in downtown metro areas, they are middle income. (Group 14B – Our Turn)
34. Outward Bound: Middle-aged (30-45), married home owners with no kids. Primarily middle income, living in rural locations. (*Group 3X
    • Transition Blues*)
35. Solo & Stable: Middle-aged (36-45) single home owners with no kids. Primarily middle income, living in urban and suburban areas. (Group 9B – Boomer Singles)
36. Raisin’ Grandkids: Seniors (aged 66+), both single and married home owners with kids of mixed ages in the household. Primarily middle income, living in suburband and urban areas. (Group 20S – Active Elders)
37. Cartoons & Carpools: Middle-aged (30-45), married home owners with school-aged kids. Primarily middle income, living in urban and suburban areas. (Group 5X – Gen X Parents)
38. Blue Collar Bunch: Late middle-aged (46-65) married home owners with school-aged kids. Primarily lower-middle income, living in urban and suburban areas. (Group 13B – True Blues)
39. Early Parents: Young (18-29) single and married parents who either rent or own their residence with kids (mixed ages) at home. Primarily lower income, living in urban and suburban areas. (Group 1Y – Beginnings)
40. The Great Outdoors: Late middle-aged (45-65), married home owners with no children in the household. Primarily lower-middle income in a rural location. (Group 18M – Mature Rustics)
41. Trucks & Trailers: Middle-aged (30-45), both single and married renters and home owners with no kids. Primarily lower-middle income, living in rural areas. (Group 3X – Transition Blues)
42. First Mortgage: Young middle-aged (30-35), single home owners with no kids. Primarily lower-middle income, living in urban and suburban areas. (Group 4X – Gen X Singles)
43. Work & Causes: These lower-middle income singles own their own homes. Living in urban and suburban areas, they are between 46 and 55 and have no children. (Group 17M – Modest Means)
44. Community Singles: Late middle-aged (56-65) single home owners with no kids. With a lower-middle income, they live in urban and suburban areas. (Group 17M – Modest Means)
45. First Digs: This group consists of young (24-29) singles with no kids who either rent or own. Primarily lower-middle income, living in urban and suburban areas. (Group 1Y – Beginnings)
46. Home Cooking: Middle-aged (30-45) married home owners with no kids. Primarily lower-middle income, living in urban and suburban areas. (Group 3X – Transition Blues)
47. Rural Parents: This group consists of middle-aged (36-55) single parents who own their own homes and have children of mixed ages. Primarily lower-middle income in a rural location. (Group 10B – Mixed Boomers)
48. Farmland Families: Middle-aged (36-55), married home owners who have school-aged children in the household. Primarily lower-middle income in a rural location. (Group 13B – True Blues)
49. Sedentarians: These married seniors (76+) own their own homes and have no children. Living in urban and suburban areas, this group consists of lower-middle income earners. (Group 21S – Leisure Buffs)
50. The Greatest Generation: This group consists of married seniors (66+) who own their own homes where the kids have moved out. Primarily lower-middle income in a rural location. (Group 18M – Mature Rustics)
51. Family Matters: These seniors (66-75) are married home owners with no kids at home. Living in the outer suburbs and towns, they are in the lower-middle income bracket. (Group 21S – Leisure Buffs)
52. Still Landlorded: Middle-aged (36-45) singles renting their residence with no kids. Primarily lower-middle income, living in urban and suburban areas. (Group 4X – Gen X Singles)
53. Metro Parents: Middle-aged (36-55) single parents who own their own homes and have school-aged children. Primarily lower-middle income, living in urban and suburban locations. (Group 10B – Mixed Boomers)
54. Still Truckin’: This group consists of late middle-aged (46-65), single home owners with no kids in the household. Primarily lower-middle income in a rural location. (Group 18M – Mature Rustics)
55. Humble Homes: These married, late middle-aged (46-65) home owners don’t have any kids. They live in the outer suburbs and towns and are lower income earners. (Group 17M – Modest Means)
56. Modest Wages: Middle-aged (36-45) single home owners with no kids. Primarily low income, living in urban and suburban areas. (Group 9B – Boomer Singles)
57. Collegiate Crowd: This group consists of young (18-23) singles with no kids who rent or own. Primarily lower income, living in urban and suburban areas. (Group 1Y – Beginnings)
58. Young Workboots: Young (18-29) singles with no kids who rent or own. Primarily blue-collar workers (low income), living in rural areas. (Group 1Y – Beginnings)
59. Low-Rent Digs: Young middle-aged (30-35) single renters with no kids. They are in the lowest income bracket, living in urban and suburban areas. (Group 4X – Gen X Singles)
60. Rural Rovers: Middle-aged (36-45) single renters with no kids, they live in rural areas and are lower-middle income. (Group 10B – Mixed Boomers)
61. Urban Scramble: Young (24-35) single renters with no kids. Primarily lower income, living in down town metro areas. (*Group 6X
    • Mixed Singles*)
62. Kids & Rent: Middle-aged (30-45) single and married renters with kids (mixed-ages). Primarily lower income, living in the outer suburbs and towns. (Group 5X – Gen X Parents)
63. Single City Struggles: These late middle-aged (46-65) single renters live in urban and suburban areas. They have no kids and are low income earners. (Group 17M – Modest Means)
64. Rural Antiques: This group consists of single renters and owners aged 76 and older with no children in their household. Primarily lower income in a rural location. (Group 21S – Leisure Buffs)
65. Thrifty Elders: These single seniors (66-75) own their own homes in the outer suburbs and towns. They are among the lowest income earners. (Group 21S – Leisure Buffs)
66. Timeless Elders: In the lowest income bracket, these single aged seniors (76+) own their own homes in urban and suburban areas. (Group 21S – Leisure Buffs)
67. Rolling Stones: This group consists of young (24-29) singles with no kids who rent or own. They are the in the lowest income bracket, living in urban and suburban areas. (Group 1Y – Beginnings)
68. Penny Pinchers: These late middle-aged (46-65) single home owners are among the lowest income earners. They live in urban and suburban areas with no children. (Group 17M – Modest Means)
69. Mortgage Woes: Middle-aged (30-45) single home owners with no kids. They are in the lowest income bracket, living in urban and suburban areas. (Group 6X – Mixed Singles)
70. On the Edge: Middle-aged (36-45) single renters with no kids. They are in the lowest income bracket, living in urban and suburban areas. (Group 6X – Mixed Singles)

Some the cluster identifiers are funny while others are pretty disheartening—I’m sure the intention was to make them memorable and in that Acxiom has succeeded. You’ll notice that the majority of information available online focuses on the top thirty or so segments, as these are apparently the most lucrative (consisting of the upper and upper-middle classes).

It’s amazing how much data Acxiom collects and the detail of information they can extract from that data. For instance, for each segment they can give you an idea of what activities they enjoy, their opinions/beliefs, how they vote, where they shop, what they read, what they eat, what their finances look like, etc.

Acxiom provides a document on their website which shows how they can predict the migration of consumers from one segment to the next. They need to predict well as these segments are so specific that nearly one-third of Americans change their segment each year.

If you can help me find more freely-available, detailed information (like this) for any of these segments, please leave a comment below or email me.

References

• http://www.acxiom.com/default.aspx?ID=1766&DisplayID=18
• http://www.wired.com/culture/lifestyle/news/2004/11/65640
• http://www.informationweek.com/story/showArticle.jhtml?articleID=51000186
• http://www.acxiom.com/subimages/1126200385918taking_hold_group_2y.pdf
• http://www.acxiom.com/subimages/715200351943closer_look_–_america_s_wealthy.pdf
• http://www.relevance-x.com/images/Relevance-X.pdf
• http://www.hoythighland.com/presentation.pdf
• http://www.astech-intermedia.com/resources/downloads/Roundtable_2007_Thornquist.ppt
• http://sustainingfamilyforests.org/pdfs/report_intro.pdf
• http://www.acxiom.com/subimages/26200352112wp_meth_103.pdf

This actually happened back in January, but since I was never notified, I researched it myself after the recent news of Cisco opening up IOS to 3rd party developers.

The secret is to encrypt your Ruby source, store it in a database and then modify the Ruby interpreter to look for your code in the database and decrypt it on the fly. I should note that there is no way to completely protect a product which is distributed to your customers—with enough diligence any security measures can be broken.

These instructions are for Unix operating systems (include MacOS X). Unfortunately (or fortunately, for me), I don’t own a Windows machine.

If you don’t already have Berkeley DB (BDB) installed on your system you can download it here. You will need to follow the instructions which come with BDB which explain how to build and install it. Originally, I was going to use OpenSSL to encrypt our Ruby source code before inserting it into the database and decrypt it after retrieving it from the database. Thankfully, the latest version of BDB includes AES support which allows you to maintain an encrypted database very easily.

You will also need to obtain a fresh copy of the Ruby source code so that we can build our own private version which knows how to pull classes out of our BDB instance. So let’s start by creating a directory for us to work in:

mkdir -p ~/RubyProject/deploy

You should unpack the Ruby source in the \~/RubyProject directory (this should create directory which looks like \~/RubyProject/ruby-1.8.6-p111). Go into that directory and run the configure script as follows:

./configure --prefix=~/RubyProject/deploy --with-static-linked-ext

This will enable us to build a stand alone Ruby interpreter which has everything it needs statically linked in.

To find out where we need to hook in our handler which loads missing classes from the database, you can grep through the source code looking for const_missing like this:

grep const_missing *.c

This turned up two files object.c and variable.c. Looking at the output, I could see that the const_missing function is actually defined in variable.c. Jumping in, I searched for the code executed when a constant cannot be found. This led me to the rb_const_get_0 function which walks up the class hierarchy looking for the specified constant. If it cannot be found, it returns the result of the const_missing function. So right before that final return, I added the following hook:

    /* At this point we haven't found the class so we must look
       inside of the BerkeleyDB for it, see dbloader.c */
    value = load_from_db(id);
    if (value != Qundef) return value;

Now I just had to implement the load_from_db function which should look inside of the encrypted BDB instance for the file. Looking at how the rb_const_get_0 function works, I knew my new method had to return a Ruby VALUE:

VALUE
load_from_db(id)
     ID id;
{
    DBT k;
    DBT d;

    memset(&k, 0, sizeof(DBT));
    memset(&d, 0, sizeof(DBT));

    verify_database_state();

    k.data = rb_id2name(id);
    k.size = strlen(k.data);

    d.flags = DB_DBT_MALLOC;

    if (bdb->get(bdb, NULL, &k, &d, 0) == 0) {
      rb_eval_string((const char *)d.data);
      free(d.data);
      return rb_eval_string(rb_id2name(id));
    }
    return Qundef;
}

You’ll note that if we find the class in the database, we first evaluate it using rb_eval_string. Since this returns nil, we need evaluate the class name so that we can pass back a Ruby VALUE. If the class doesn’t exist in the database, we return Qundef and const_missing gets called as usual.

The verify_database_state function ensures that the password for the BDB instance was passed and the database opened so that the get call can access it. It is implemented like so:

/* a simple way to obfuscate the password in memory */
#define A(c)             (c) - 0x1d
#define ENCRYPT_PWD(str) do { char * p = str; while (*p) *p++ -= 0x1d; } while (0)
#define DECRYPT_PWD(str) do { char * p = str; while (*p) *p++ += 0x1d; } while (0)

static void
verify_database_state()
{
  /* don't forget to NULL terminate this array! */
  static char info[] =
    { A('1'), A('2'), A('3'), A('4'), A('5'),
       A('6'), A('7'), A('8'), A('9'), A('0'),
       0 };

  const char * database_file = "data.db";

  if (bdb == NULL) {
    db_env_create(&bdbenv, 0);
    db_create(&bdb, bdbenv, 0);

    DECRYPT_PWD(info);
    bdbenv->set_encrypt(bdbenv, info, DB_ENCRYPT_AES);
    ENCRYPT_PWD(info);
    bdbenv->open(bdbenv, ".", DB_INIT_MPOOL | DB_CREATE | DB_PRIVATE, 0600);
    bdb->set_flags(bdb, DB_ENCRYPT);
    bdb->open(bdb, NULL, database_file, NULL, DB_BTREE, 0, 0644);
  }
}

You’ll notice that the password is hard coded into the binary along with the location of the BDB instance. If you are going use this code, you’ll want to change the default password (which needs to match the password you used when storing your Ruby source in the database) and perhaps the location of the database. To deter those looking to decrypt our Ruby source, I’ve written C macros which performs a transformation on the password. It is fairly simple to circumvent, so you should research other methods of securely storing passwords within applications.

Finally, you’ll need to remember to close the BDB instance before the Ruby interpreter exits. To do this we simply open up main.c and add the following line immediately following the call to ruby_run:

close_database();

The implementation of close_database is trivial, so you can just look in the patch I’ve provided for it.

The patch to the Ruby source code is included in this zipped attachment along with another utility I wrote to load all of your Ruby source into the encrypted BDB instance (along with its Makefile). Here is a transcript which shows how to use the tools and your new Ruby interpreter:

~RubyProject/tools> ./rb_store data.db *.rb
Enter database password:
Stored Example (131) bytes

~RubyProject/tools> ./rb_load data.db
Enter database password:
Hit CTRL+C to quit.
Enter a class name: Example
Found: 

class Example
  def initialize
    puts "Hello from the Example class"
  end

  def aMethod
    puts "Called aMethod"
  end
end

Enter a class name: Foo
No such class
Enter a class name: ^C

~RubyProject/ruby-1.8.6-p111> ./ruby -e "e = Example.new"
Hello from the Example class

~RubyProject/ruby-1.8.6-p111> ./ruby -e "f = Foo.new"
-e:1: uninitialized constant Foo (NameError)

This implementation doesn’t deal with multiple classes of the same name residing in different directories. Also, the way in which the rb_load tool determines the name of the class is rather naive and can screw it up. I’ve wanted to write this for over a year now after hearing Rich Kilmer talk about the way in which InfoEther distributes its Ruby applications. So this is more of a proof-of-concept for myself than anything else. Hopefully you learned from this tutorial and can put it to good use.