This blog post gives instructions for how to get Firmata working with both Processing and MaxMSP on Mac OSX 10.6.3. The information is current as of 8 July 2010, but is naturally certain to change. The context for this blog post is that I’m working with my university’s Computing Services staff to develop an administrator’s disk image for some new Macs that have just arrived. I prepared the information below for this crew, who are talented IT professionals that happen not to have domain expertise in arts-computing platforms. To the best of my knowledge, this information has not yet been compiled in any one place, but is rather distributed around the Internet in too many pages, some of which are out of date. I welcome comments and corrections.

————————————————————————————-

The specific list of 8 installation packages are as follows.
Detailed instructions for installing and testing these packages
can be found in the sections below.

1. Arduino IDE version 0018: arduino-0018.dmg
2. Drivers for Arduino boards: FTDIUSBSerialDriver_10_4_10_5_10_6
3. Firmata version 2.1+ for Arduino: firmata.tar.gz
4. Max/MSP/Jitter 5.1.3 or later: e.g. Max5_42462.dmg
5. OSC-route external library for Max: OSC-route_1.17.1.tar.gz
6. Maxuino library for Max v.009: maxuino-009.zip
7. Processing version 0186, processing-0186.dmg.
8. Arduino for Processing library, processing-arduino-0017.zip.

Overview
In January 2010 the F.A.T. Lab released a suite of XML specifications and open-source tools for recording and displaying handwritten marks, which they named Graffiti Markup Language (GML). Their flagship system is the Graffiti Analysis software by Chris Sugrue and Evan Roth, which allows GML drawings to be recorded with an iPhone, uploaded to an online database, and freely shared at the #000000book.com website. Users of Graffiti Analysis have contributed some 17,500 tags to date. As with our Robotagger project, the GMLStenciler software presented here offers a means by which these virtual handwritten marks can be given physical form, made multiple, and economically disseminated at a variety of scales.

Operation
The Graffiti Markup Language specification defines a “tag” as a sequence of linear “strokes”, each consisting of a sequence of timestamped (x,y) points. These strokes lack a width or thickness; as a result, it is not possible to stencil raw GML tags directly. GMLStenciler solves this by providing a tool in which:

● The user imports a GML file, such as this one from 000000book.com:

● An interface is provided by which the user can select an adjustable thickness for rendering the tag’s strokes. Using OpenCV, the system then automatically detects “holes” in the rendered tag (such as would be produced by the closed counters within characters like A, B, D, O, P, Q, R, etc.). These holes are problematic for stencils because they form loose “islands” that, left unbridged, would disappear from the final stenciled design.

● The user can select one of several methods by which these holes will be automatically bridged. Options for this operation include: bridging at the top point of each counter, bridging at the nearest point to the counter’s enclosing stroke, etc. Because GMLStenciler bridges holes recursively, strokes with complex nested structures or holes-inside-of-holes (as in the symbols ®, ◎, ©, ʘ, etc.) are still bridged successfully:

● The user can export the stencil design as an EPS (Encapsulated PostScript) vector file. (Here’s a raw example output file created by GMLStenciler.) Optionally, this EPS file can be loaded into CAD programs or vector art editors like Illustrator or CorelDraw for further editing:

● The EPS file can then be used as the pattern for cutting a stencil on a laser cutter, vinyl cutter, water-jet cutter, CNC router, or other computer-controlled cutting machine. Laser cutting services are widely available in many cities, and are also increasingly common in regional hacker spaces and university art, architecture and design departments. Below, the stencil has just been cut from 1/8″ MDF (20″x16″, about $3) on a 40-watt Epilog laser cutter; cutting time was under 2 minutes. For very large stencils, I recommend using a CNC router, many of which can accommodate materials up to 5′x10′.

These stencils can then be used in a variety of ways. One interesting potential use for such stencils is so-called cleanfitti or reverse graffiti (also known as clean tagging, dust tagging, or grime writing), in which the stencil is used as a frisket for dirt removal.

Downloads

The GMLStenciler can be downloaded here:

• GMLStenciler_1.00_mac.zip [20MB] Includes app and source, as an openFrameworks v0.061 project for MacOSX 10.6.3 / XCode. This zip includes all necessary (ofx) addons, but developers wishing to recompile this will still need to download the openFrameworks v0.061 library. This slimmer 11MB zip omits standard ofx addons and assumes you already have OF v0.061 working.
• GMLStenciler_0.99_win.zip [21MB] Includes exe and source, as an openFrameworks v0.061 project for WinXP+ with MS VisualStudio 2008. This build currently lags the Mac version very slightly.
• As a Github repository. Developers, note: The most recent version of the source code can be found here.

License and Disclaimer
The GMLStenciler software is licensed under the CC-GNU GPL version 2.0 or later. All Graffiti Analysis related media is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License. Users of GMLStenciler are solely responsible for their activities while using this tool; this site and its author(s) do not condone illegal uses of the GMLStenciler software.

Credits and Acknowledgments
The GMLStenciler software was entirely developed by Charlie, my 15-year-old intern, during June 2010. Charlie (a rising Junior in a Pittsburgh-area high school) has a solid background in Java programming and server-side web scripting, but hadn’t previously worked with C++, OpenGL, OpenCV, or openFrameworks, so this proved to be an ideal learning project. Here’s a photo of Charlie.

Charlie developed GMLStenciler in openFrameworks under the auspices of Golan Levin. GMLStenciler makes use of the Graffiti Markup Language (GML) specification released by the F.A.T. Lab, and includes code developed by Chris Sugrue and Evan Roth for their Graffiti Analysis software. The GMLStenciler also uses the OpenCV open-source computer vision library. Thanks to Kyle McDonald, Jacob Tonski and Solomon Bisker for supporting Charlie with ad hoc coding advice during his internship. Thanks to the STUDIO for Creative Inquiry at Carnegie Mellon University for hosting Charlie, and to the CMU School of Art for the use of its laser cutter. GML “em@il” tag by Evan Roth. Documentation photos by Kyle McDonald.

References

• “Semi-Automatic Stencil Creation through Error Minimization“. Jonathan Bronson, Penny Rheingans, and Marc Olano. Proceedings of the 6th International Symposium on Non-Photorealistic Animation and Rendering, 2008.
• “A Drawing Editor for Stencil Design” [ステンシルデザインのためのドローエディタ]. Yuki Igarashi and Takeo Igarashi. 17th Workshop on Interactive Systems and Software (WISS), 2009.

Keywords
FAT, GML, GRL, #OF, Free Art and Technology, Graffiti Markup Language, openFrameworks, graffiti research, GML-Stenciler, GMLStenciler, GML-Stenciller, GMLStenciller, free open-source software, source code, software tools, app, laser cutting, laser cutter, rapid prototyping, digital manufacture, pochoir, frisket, stencil, stenciller, stenciler, cut patterns, vector art, vector output, XML, EPS, PDF, CAD, C++, OpenCV, graf, graffiti, grafitti, cleanffiti, cleanfitti, inverse graffiti, reverse graffiti, clean tagging, dust tagging, grime writing, computational design, computer-aided stenciling, automatic stencil creation, automatic stencil generation, hacking, Robotagger, graffiti and automation.

A complete set of about two dozen such panoramas can be seen in this Flickr set.

A modified version of the software — the FlowerUnwarper — was optimized for unwarping flower images, and is available here (including source code). The software is built in Processing and uses the ControlP5 library. All of the source photos used in this project were distributed with Creative Commons Attribution licenses. Our software is free in the Public Domain and our own re-processed images are released under the Creative Commons Attribution-ShareAlike 3.0 Unported License.

Update (July 10): The folks at NotCot have since used our software to create these fascinating views of unwarped chandeliers!

Keywords: polar to cartesian, flowers, warping, unwarping, unwrapping, dewarping, rectification, inverse warping, correction, panorama, macro, macrophotography, Processing, surrealism, flower landscapes, petals, Processing, Java, free source code, free software, polar transformation of flowers.

In my previous blog post about panoramic imaging with the Bloggie, I complained about how the Bloggie automatically lowers its resolution from 1920×1080 down to 1280×720 when the panoramic lens accessory is fitted onto the camera. Happily, Jan Martin from DIY-streetview.org helpfully pointed me to this short post by onezebra1 on the PanoTools discussion board, which describes how to modify the accessory to support full-resolution video recording with the Bloggie. As we had suspected, the Bloggie uses a magnet to trip the resolution switch. Removing this magnet is easy, and produces the desired result — annular panoramic video at the device’s highest possible resolution — with some caveats.

Here’s the Bloggie’s panoramic lens accessory, Sony Part Number X25467051, labeled VCL-BPP1 on its body. On the underside of the accessory are 4 very small cross-head screws (not pictured):

Removing these four screws allows the optics to be separated from the mounting base. Once separated, you will find two small rare-earth magnets inset into the the inner (upper) surface of the base. These magnets (each about 3x4mm) can be extracted with a small flat screwdriver. The photo below shows the inner surface of the base. At the bottom right of the photo is the optics half of the assembly.

It turns out that only the left magnet needs to be removed for the hack to work. I recommend doing this to preserve maximum flexibility for your optics.

The optics are reassembled and fit onto the camera as before, and voilà — panoramic video at 1080p resolution on the Bloggie:

There is, however, an important caveat. The panoramic donut is now truncated on the top and bottom, as you can see above. Here is a still frame from the 1920×1080 video recorded in this way.

For comparison, below is a still from a video recorded in 1280×720 resolution; as you can see, the entire donut is visible. I had left one of the magnets installed, so I just flipped the same lens around to produce this. But it’s also possible to recover the Bloggie’s factory-standard 720p panoramic video behavior by selecting 1280×720 resolution from its menu system.

Using the 1080p-resolution annular videos with my BloggieUnwarper software is straightforward. (The MAXR_FACTOR and MINR_FACTOR properties must be set to approximately 1.2648 and 0.3018, respectively.) Below is a screenshot, which shows how the truncated regions are interpreted by the rectification process. For what it’s worth, these missing regions account for 6.8% of the area of the unwarped panorama.

By the way, it’s worth clarifying that the Bloggie’s still images remain unaffected by this modification; the operation described above only affects the camera’s video recording capabilities. Below is a panoramic still shot at the Bloggie’s highest resolution (2592×1944). The annulus has the same dimensions as the 1920x1o80 video, but without the cropping. Both unmodified and modified optics will produce such still images on the Bloggie. The image quality is much clearer, probably because of a longer exposure time and different compression configurations.

After making some more careful measurements, I’ve determined that the maximum vertical resolution obtainable from the Bloggie’s standard 1280×720 video is ~261 pixels, while the maximum obtainable from the 1920×1080 video is ~518 pixels. So it may be worth making the mod, if you can tolerate missing about 6.8% of your pixels in a couple of places. [Photography: Kyle McDonald]

Obviously, this “donut vision” (annular) video needs to be transformed into something more useful, such as a long panoramic strip — a process variously called unwarping, dewarping, inverse warping, remapping, reprojection, rectilinearization, or simply distortion correction. Although Sony released unwarping software for Microsoft Windows, they — inexplicably — didn’t release Bloggie unwarping software for Mac OSX. Well, that’s problematic. Perhaps because panoramic imaging is considered a “niche” market, commercial tools for panoramic unwarping can be quite pricey. And although Christophe Maillot has lately released ThreeSixZero, a free OSX tool for viewing Bloggie panoramas, his project is not open-source and doesn’t export converted panoramic video! That’s the motivation for the releases presented here. Fortunately, the mathematics for polar-to-cartesian geometry conversion are relatively straightforward. Behold, a 360-degree panoramic video, remapped to a rectilinear perspective with our BloggieUnwarper software:

Projects and Source Code

Below are links to BloggieUnwarper, my open-source code projects for unwarping the panoramic videos produced by the Sony Bloggie. These projects are created in openFrameworks (a C++ library for creative coding) and Processing (a Java-based programming environment for people who want to create images, animations, and interactions). Both OpenFrameworks and Processing are free, open-source, cross-platform toolkits, optimized for visual computing, which take most of the headache out of creating graphically-oriented software. Both programming environments work across multiple operating systems (Mac, Windows, and Linux), and multiple compilers (e.g., openFrameworks compiles in XCode, Visual Studio, Code::Blocks, and more). If you’re interested in interactive art or computational design, these environments are a great way to start making projects fast.

Here’s a screenshot of the openFrameworks BloggieUnwarper software, showing the original (annular) video at upper left; the reprojected panoramic strip at the bottom; and (in the upper right) a view of the panorama from within a texture-mapped 3D cylinder:

Complete source code can be found here:

• BloggieStillUnwarp.zip (for Processing; 1.3Mb download) — This is a single-page Processing sketch which performs inverse warping on a single panoramic still image stored in the sketch’s data folder. The program supports both nearest-neighbor and bicubic interpolation, and can save out the unwarped panorama to disk. To get started, download the Processing environment here; unzip this project into your “Sketchbook” folder (in “Documents”); open the sketch in the Processing development environment, and press play.
• BloggieVideoUnwarp.zip (for Processing; 20.2Mb download) –This is a single-page Processing sketch which unwarps a Quicktime video stored in the sketch’s data folder, and exports an unwarped version of the video — either as a new Quicktime movie, or as a folder full of still images. Parameters for the unwarping (such as the optic center) are loaded from a settings file. The audio from the original file is (regrettably) not maintained; see my notes below about audio. (Please note that the bulk of this download is a 20Mb demonstration video.)
• BloggieUnwarpOF.zip (for OSX & openFrameworks, 52.7Mb download) — This is a complete openFrameworks project for viewing, projecting, dewarping, and exporting panoramic videos shot with a catadioptric reflector like the Sony Bloggie’s. This project is built for Mac OSX 10.6.3, using XCode, and openFrameworks v.0.061, which is available here; be sure to download the FAT version. The project makes use of the ofxOpenCv and ofxXmlSettings addons which are included in the download. To avoid compilation errors, install the project at the same hierarchical directory level as the various “example” apps which come with the FAT download. There is some additional information about getting OF set up on Snow Leopard here. This project is available two ways:
  • as a project on Github: http://github.com/golanlevin/BloggieUnwarp, or
  • download BloggieUnwarp for OSX here (52.7MB) as a zip file.

More About the BloggieUnwarper

Here’s some more information about panoramic video on the Sony Bloggie, our app, and its implementation.

Sourcing the Bloggie and its Panoramic Lens Attachment.
The Sony Bloggie MHS-PM5K (about $160-180) comes with an included 360-degree panoramic lens attachment. You may hear this accessory erroneously referred to as a “fisheye lens“, but it’s actually a hyperbolic or parabolic mirror (with an optical path folded by a flat mirror, as it turns out) known as a catadioptric reflector. This panoramic lens attachment, which easily clips on and off the body of the camera, is the black piece shown above. The attachment is Sony Part Number X25467051, a “360 VIDEO ACCESSORY”, labeled VCL-BPP1 on its body. The attachment really does come with the camera! But as an isolated component, this accessory is also available here through Sony’s services site (Tel. 1-800-488-7669) for $82.95 USD. Now given that this is, like, a $5 piece of plastic, this is still a minor bummer, but oh well: competing mirror-based optics for one-shot 360-degree imaging, like the Palnon parabolic C-mount panoramic lenses from JBC, the Go Pano from EyeSee360, the OmniAlert360 from RemoteReality, the PentaOne P360 series ($~1000), the EggSolution EGG Lens (~$2000), or the 0-360 Panoramic Optic ($495), can cost hundreds or even thousands of dollars; there’s simply nothing else quite like the Bloggie at the low end. In any case, it’s worth pointing out that you can duct-tape this Sony pano accessory onto just about any micro-format camera if your application demands a different hardware scenario. I made relatively decent panoramic images by simply holding it in front of the cameras on my laptop and my mobile phone, and I also had good luck affixing the mirror module onto standard 12mm “micro-video” lenses such as those used in many web-cams. If you’re interested in doing something low-budget like DIY-Streetview, the Bloggie and/or this lens attachment is the camera for you.

Limits on the Resolution of Resulting Panoramas.
The Sony Bloggie is capable of recording non-panoramic HD video at resolutions up to 1920×1080 (30fps). Unfortunately (and this isn’t well-advertised), the Bloggie only records panoramic digital video at 1280×720. In video mode, the camera auto-detects the panoramic lens attachment (magnetically, it turns out) and switches to the lower resolution. Moreover, by the time the unwarping process has removed the wasted black pixels (inside and around the warped video annulus), it becomes clear that the maximum effective vertical resolution of the Bloggie’s panoramic video is only about 256 pixels, which may or may not suit your purposes. Used in still-picture mode, the Bloggie is able to create panoramic source images with its full (5 megapixel, 2592×1944) resolution; the effective vertical resolution of its panoramic stills is then 512 pixels. Since the Bloggie lens accessory has a vertical FOV of approximately 54.5 degrees (according to our measurements; others have measured the vertical field of view at 52.5°), panoramas produced with these optics should have aspect ratios close to (360/54.5) ≈ 6.6:1. For panoramas produced with the BloggieUnwarper, I therefore recommend dimensions like 1280×196, 1680×256, or (for 5MP stills) 3360×512. (Note that the Bloggie has a 1/2.5” CMOS imaging sensor and F3.6 optics. This means: expect relatively poor low-light performance, and some very exotic rolling-shutter artifacts on fast-moving subjects after your source video is unwarped.) Update: in a new blog post, “Modding the Bloggie Panoramic Lens Accessory“, I detail a simple modification to the Bloggie accessory which enables 1920×1080 panoramic video recording.

An Opportunity for Real-Time Panoramic Video
The BloggieUnwarper software can compute unwarped panoramas in real-time or faster. And happily, the Bloggie camera can also produce real-time panoramic video, through its analog NTSC/PAL composite video output, using the proprietary Sony HandyCam Audio/Video Cable (VMC-15FS A/V) which comes with the camera. This interlaced analog video signal produced by the Bloggie is then suitable for 720×480 capture via standard analog capture cards such as the Dazzle. An opportunity thus exists for (e.g.) interactive computer-vision based applications that use 360-degree sensing. To be perfectly clear, the Bloggie does not support use as a real-time USB webcam; you’ll need to run its analog video output through a capture card (or fit the Bloggie lens attachment onto some other camera) if you intend to build apps that incorporate real-time panoramic video.

The code projects provided here are configured for unwarping Bloggie videos stored on disk — but modifying these apps to use real-time video is straightforward; you’ll simply need to swap out the source of input pixels from the pre-stored video to the live video input:

• If you’re working in openFrameworks, integrate the ofVideoGrabber class into the project, as demonstrated by the movieGrabberExample example included in the FAT download, to understand how to access the live video stream and its pixels.
• For Processing, on the other hand, integrate the Capture class within the Video library, taking a careful look at the included GettingStartedCapture and Mirror examples to understand how to access the live video and its pixels.

Special Considerations, Features, and Known Bugs
Unwarping panoramic video from the Bloggie has a few special considerations, mentioned here.

• Handling Inexact Centering of the Panoramic Optics.
  Sad to say, the Bloggie’s removable panoramic attachment is a wee jiggly. As a result, it’s unwise to assume that the optical center of the video donut is located precisely at the location [width/2, height/2]; indeed, the optical center has a movement range of approximately ±2% of the image dimensions. This has significant implications for an unwarping tool; even small differences of a quarter pixel can cause an undesirable sinusoidal “wobble” that runs the length of the panorama. The apps provided here allow for the user to adjust, set and save the center point of the video annulus, with sub-pixel accuracy.
• Handling Corrections for Radial/Vertical Nonlinearity
  If the Bloggie had a ‘perfect’ catadioptric mirror, there would be no vertical distortion in the unwarped panoramas. Unfortunately, our careful measurements of rectilinear grids photographed with the Bloggie’s panoramic lens attachment proved otherwise; we discovered a minor nonlinear distortion in the radial/vertical dimension of the Bloggie lens. We also discovered that the Bloggie panoramic lens has an asymmetrical vertical field of view, with approximately 80% of the image falling above the horizon, and 20% below. Taking both perspectival distortion and the Bloggie’s off-center vertical optics into account, we computed this distortion, modeled it with a 2nd-order polynomial, and have corrected for this in our unwarping procedure. You’ll need to recalculate these fitting parameters if you wish to use this software with a different catadioptric mirror.
• Providing Azimuthal/Horizontal Shifting
  When unwarping an annular video to a horizontal panoramic strip, the placement of the “zero angle” becomes an important artistic choice. Modifying this angle has the effect of determining the horizontal placement of objects in the scene. BloggieUnwarper allows for user-determined shifting of this kind.
• Bugs with Preserving Source Audio
  As of this writing, BloggieUnwarper doesn’t handle audio nearly as well as I would like. The Processing version does not preserve the source audio at all; if you want to keep it, you’ll need to use a tool like FinalCut or AfterEffects to recombine the original source audio with the corrected panorama video. The openFrameworks version is a little more strange. It currently produces “non-standard” videos, which contain the original audio alongside the unwarped video, and are playable in Quicktime7+ on the Mac, but which fail to play in Windows, and are rejected by Vimeo for unspecified incompatibilities. I have successfully circumvented this by re-compressing the results using Quicktime and/or MPEG Streamclip.

A brief note about speed and library dependencies.
To accomplish fast (better than real-time) inverse warping, we compared four methods: a well-optimized, hand-coded inverse lookup table; the cvRemap() function available within the free OpenCV library; the fwiRemap() function in AMD’s free FrameWave signal processing library; and the ippiRemap() function from the commercial Intel Performance Primitives (IPP) 6.0 libraries. That’s right — we had a shootout. All of our implementations were coded for 3-channel (RGB) images, and for each method we tested both Nearest Neighbor and Bicubic interpolation methods. We set compiler optimizations to level -O3 in XCode and compiled for 10.6|Release. Under these conditions, the quick summary of the libraries’ relative performance was as follows:

• IPP ippiRemap: 1 time unit
• OpenCV cvRemap: 2 time units
• Framewave fwiRemap: 4 time units
• Hand-coded version: 2.5-10 time units, depending on the extent of optimization

In other words: IPP was the fastest; OpenCV was twice as slow as IPP; Framewave was four times as slow; and (short of getting all medieval with assembly code), my own best hand-coded algorithm was about 25% slower than OpenCV. Based on the above shootout, we elected to use OpenCV with our openFrameworks build (see the ofxOpenCV addon example, which comes with the download), and a hand-coded unwarping algorithm for the Processing version. Researchers who are serious about speed may wish to consider the (non-free, non-open-source) IPP libraries. As an aside, OpenCV is documented as being able to “take advantage of IPP if it is available“, but this feature of OpenCV is being phased out.

Acknowledgments
Ivan Poupyrev first alerted me to the existence of the Sony Bloggie. Solomon Bisker helped port my original Processing sketch to openFrameworks, and also set up the four-way shootout between IPP, Framewave, OpenCV, and my hand-coded algorithms. Amisha Gadani tracked down purchasing details for the X25467051 lens accessory. Kyle McDonald patiently walked me through learning Git, and collaborated in many stimulating discussions and experimentation related to low-cost panoramic video. This (side-)project was supported by the STUDIO for Creative Inquiry at Carnegie Mellon University, with the help of grants from the Fine Foundation and the National Endowment for the Arts. This project has been featured at Hack-a-Day.

Keywords
Golan Levin, Flong, Processing, openFrameworks, #Processing, #openFrameworks, Sony Bloggie camera, 360° video, 360-degree pano, panoramic video, catadioptric reflector, hyperbolic reflector, parabolic mirror, unwarp, dewarp, inverse warp, remap, reprojection, rectilinearization, distortion correction, free open-source code, Java, C++, software, free video tools, optics, lens, attachment, one-shot panoramic imaging, download, Bloggie software for Mac OSX, free Bloggie for MaxOS X, Mac OS X.

References
“A Theory of Single-Viewpoint Catadioptric Image Formation“, S. Baker and S.K. Nayar. International Journal on Computer Vision, Vol.35, No.2, pp.175-196, Nov, 1999. [PDF]

“Image Warping / Distortion“, P. Bourke. http://local.wasp.uwa.edu.au/~pbourke/miscellaneous/imagewarp/, Dec. 2002.

Licence
This software is licensed under the CC-GNU LGPL version 2.1 or later.

TykeType! is a free, open-source application I created for my toddler to type at the computer keyboard. The app is very minimal, though it does have a few features (like automatically separating numbers from letters) that make reading text aloud easier. You can see a quick demonstration video below.

TykeType! is built for Mac OSX and uses openFrameworks, an open-source toolkit for creative coding. You’re invited to download the application (and all source code) here. The software and code are free for non-commercial purposes, and are provided without any support.

TykeType! by Golan Levin is licensed under a Creative Commons Attribution-Noncommercial 3.0 Unported License. Keywords: children’s software, software for children, free toddler typing game, free typing program for kids, toddler typing software, pressing keys, pushing buttons, keyboarding, introductory keyboard skills, basic reading skills, keyboard interaction, typing software, computer keyboard play. Also see related software: TykeWriter, ToddlerType.

Many computer vision projects require a camera which blocks visible light and exclusively sees infrared light. Unless you have money to spend on optical glass units, such filters generally take the form of a thin sliver of black gelatine or polyester sheet, inserted between your camera’s sensor chip and your lens. Representative products include the Kodak Wratten series (about $0.80 per square cm), and the Lee filter series (~$0.25 per square cm). These filters each have several varieties with different spectral properties: Type 87 passes IR longer than 750 nanometers, while type 87C passes IR light longer than 800nm. In the photo above, I’m holding a sample of each of these filters. (The camera can see through these, but they’re both optically black to humans.) Here you can see the filters’ spectral responses:

Meanwhile, most CCD sensors have spectral responses which taper off in the infrared regions above ~700nm:

Thus, the photo at the top of this post confirms what you would logically expect: as the sensitivity of the CCD diminishes for longer wavelengths of IR light, IR-pass filters with higher cutoff frequencies will allow less and less light through to the camera. How much less? To judge from my casual test, I’d say about 10%, 25 grayscale levels, or roughly ½ of an F-stop. No big surprises here, but it’s nice to see the theory confirmed. The quick upshot: Choose an 87 filter over an 87C if you need to get the most out of your infrared illumination.

For more information about IR imaging, I recommend the resources:

• The Infra-Red Photography FAQ
• Don Klipstein’s page.
• Infrared basics for digital photographers
• Reflected infrared photography: Filters
• Infrared Photography with a Digital Camera

(Please note that this article refers to cameras and filters for “near IR” [close to red, with wavelengths around 700-900 nanometers], not the “deep IR” thermal-imaging cameras [sensitive to wavelengths in the 7-14 µm range], such as those made by FLIR.)

• Overview of the ACM multimedia 2009 workshop on multimedia for cooking and eating activities (CEA’09)

Computation and Agriculture:

• Robotic manipulation of food products – a review (P.Y. Chua)
• Non-destructive technologies for fruit and vegetable size determination – A review (G.P. Moreda)
• Inspection and grading of agricultural and food products by computer vision systems—a review (T. Brosnan)
• Learning techniques used in computer vision for food quality evaluation: a review (C-J Du)

Recipes and Planning:

• Ontology-based semantic querying of the web with respect to food recipes (L. Gutiérrez Villarías)
• Social navigation of food recipes (M. Svensson et al.)

Computation and Food Preparation Technologies:

• Star Trek replicators and diatom nanotechnology (R. Drum et al.)
• Computational simulation of kitchen airflows with commercial hoods (R.M. Kelso et al.)
• Increasing rate of water temperature due to an Ekman layer flow in a heated pot (fluid dynamics in the kitchen) (K. Ida et al.)

Analysis of Eating Behavior:

• Fast Food Recognition from Videos of Eating for Calorie Estimation (W. Wu & J. Yang)
• Drinking activity analysis from fast food eating video using generative models (Q. Wang & J. Yang)
• Computerized collection and analysis of dietary intake information (D. Feskanich et al.)
• Automatic dietary assessment from fast food categorization (L. Yang et al.)
• Pittsburgh Fast-Food Image Dataset (M. Chen et al.)

Food Interactions and Gustatory Displays – Proposals:

• Edible Bits: Seamless Interfaces between People, Data and Food (D. Maynes-Aminzade)
• Food simulator: A haptic interface for biting (H. Iwata et al.) *recommended
• Straw-like user interface: virtual experience of the sensation of drinking using a straw (Y. Hashimoto et al.)

The Star Trek replicator, a computational gastronomy system.

From XKCD (#720): A speculation on genetic algorithms and recipe generation. (Thx @zzkt)

Golan LEVIN (Associate Professor of Electronic Art) and Patrick Gage KELLEY (Teaching Assistant and PhD Student, CS), are proud to present capstone projects and other treats from our interdisciplinary Spring 2010 course, Special Topics in Interactive Art & Computational Design, in which 22 students from seven departments — Art, Architecture, Design, Computer Science, Human-Computer Interaction, Electrical Engineering, and Entertainment Technology — spanning the full range from freshmen to doctoral students — have explored computation as a medium for curiosity-driven experimentation!

COME SEE information visualizations … augmented realities … interactive robotics … generative form fabrication … contestational retrosurveillance …mobile software … speech-driven gaming … computational inflatables … crowd-sourced psychoanalytics … 2D and 3D game design … culture analysis machines … useful web services … and freestyle computing hacks … all developed in arts-oriented computing environments like PROCESSING, OPENFRAMEWORKS, FLASH and ARDUINO.

Featuring new works by: Caryn AUDENRIED, Solomon BISKER, Amanda BURRIDGE, Richard CAMERON, Justin EDMUND, Mehrdad GHODS, Max HAWKINS, Michael HILL, Xiaoyuan JIANG, Nara KASBERGEN, Jessica MENG, Jon MILLER, Alyssa REUTER, Matthew SANDLER, Paul SHEN, Jordan SINCLAIR, Gabriela URIBE, Karl D.D. WILLIS, Kuan Ju WU, Cheng XU, David YEN, & Raymond YUN. This exhibition was made possible with generous support from the Fine Foundation; the STUDIO for Creative Inquiry at Carnegie Mellon, and the CMU School of Art.

Please consider seeing the show by if you’re in Pittsburgh! The show runs through April 24th at the Philip Chosky Theater in the Purnell Center at CMU [map]. Performances are at 8pm, Tuesdays through Saturdays, with an additional matinee performance at 2pm on Saturdays.