Kyle Machulis

Talking Bluetooth LE on Desktop in 2016

2016-11-07T17:50:27-08:00

I am sorry. I am so, so sorry.

Every so often, I decide "Gosh, I'd really like to write code for some Bluetooth LE devices, but I don't really do much on mobile. Maybe things have gotten better on desktop!" So far, I have been disappointed every time. Now is no exception, but I've decided to actually write down that disappointment as a form of therapy.

This post will go over how different desktop OSes, libraries, and hardware deal with Bluetooth LE. I'm sticking to desktop here because product manufacturers assume BTLE devices will usually be used with phones. Mobile certainly isn't a solved problem either, but it's better than desktop right now.

This article isn't an introduction to BTLE itself. I'm going to assume readers know the basic terms and differences between, say, pairing and connection. If you're not familiar, I recommend checking out Adafruit's BTLE Intro. Apple's CoreBluetooth Overview has some nice explanation also, though the examples are obviously platform specific.

Operating System Support

OS X

Starting off easy. OS X has had support as a BTLE Central Node since 10.6, and as a peripheral node since 10.9. Done!

Linux

tl;dr: Bluez < 5.38 or so, use gattlib. Bluez >= 5.38 or so, use dbus.

And then right on up the difficulty curve to Linux, where we have bluez. I've yet to ever hear anyone say "yay bluez!"

Bluez got BTLE support in 4.93 or so. As of this writing (November 2016), we're at 5.43. That's a full major version and a ton of minor versions difference.

Between bluez 4 and 5, APIs moved from direct access to dbus. Then, within DBUS 5, the methods have changed multiple times. I spent part of last weekend trying to write some dbus code for accessing BTLE devices with no luck, as I couldn't seem to identify services on the device. It turns out that I'm on debian Jessie, which comes with dbus 5.23 (released September 2014) . After looking around a bit, it seems most current bluez supporting libraries expect users to have at least 5.38 or higher, and sure enough, those expose different methods.

I was pretty confused by this, as I'd been using pygattlib with no problems on the same linux box to write some BTLE test scripts. Turns out, pygattlib is similar to the C-based gattlib. Both of these use the GATT functions from gatttool in the bluez 4 line to talk to BTLE devices without having to go through dbus, to let older machines/kernels talk BTLE. That's why things worked, because it was just bypassing the dbus interface.

Windows

tl;dr: Either hope your device requires pairing, or your code can deal with WinRT APIs and will only run on updated Windows 10. Otherwise, do something crazy.

Windows started supporting communicating with BTLE as of Windows 8. However, this didn't mean you could just go talking willy-nilly to any device you pleased. You actually had to pair with devices before they were available to functions.

The problem is that device manufacturers are lazy and cheap, and pairing is an optional part of the BTLE handshake process. It's also the part that's vaguely secure, but a lot of products don't really care about that. For many devices, you just connect, query for services, and off you go. This is possible on both OS X and Linux, but on windows, it was a no-go up until a set of Windows 10 updates.

As of updates leading up to and including the Anniversary update in August 2016 (thanks @kraln, WinRT started providing functions to connect without pairing, which is how some of the platforms listed in the next section are going to tackle this. Microsoft has apparently stated that there will never be connection APIs for Win32 (thanks to @vincent_scheib for info).

There is one other workaround for BTLE on windows, even for pre-windows 10 platforms, but it's not pretty. Check out the section below on Noble for more information.

Cross-Platform ways to access BTLE currently

Given those warnings, if you still want to access bluetooth in a pre-written, cross-platform way, here's a few choices. This is by no means an even partially complete list of bluetooth wrappers/libraries, it's just what I looked up while figuring all this out..

Qt (C++)

Qt was actually one of the first places I went to check for this, as they're usually pretty good about supporting as much functionality as possible across platforms.

Qt has had support for BTLE on OS X, Linux, Android, and iOS since 5.7. Notice the lack of Windows there? Looks like they'll get WinRT in 5.8, "platform" support "later".

Web Bluetooth

There is currently a proposed spec being implemented by Google in Blink (the browser engine that backs both Chrome and Opera) that will allow webpages to interact with BTLE devices. This is slated to ship in Chrome 56. It will support Linux, OS X, Android, and ChromeOS. Windows is apparently coming with WinRT later.

Currently, Edge has no stated plans to implement Web Bluetooth but it is "Under Consideration". Mozilla is pushing back on spec implementation in Firefox due to privacy concerns,, though work is happening in Servo.

So while this API may show up in Chrome, it may also ONLY show up in Chrome. This situation certainly hasn't stopped anyone from using APIs before, though.

And for anyone that is saying "Wait, Kyle, didn't you implement one of the Bluetooth stacks for FirefoxOS? What about that?", my reply is "DON'T MENTION THE WAR." (Translation: That was an early version of a non-standardized API that has since been removed from Gecko, Mozilla's browser engine)

Noble (Node.js)

Noble is a node.js BTLE library that supports OS X, Linux, and Windows.

Yes, Windows. Without WinRT. Crazy, right?

Well, yes. It actually is crazy. To use Noble on windows (which many IoT/Maker programs do), you have to install WinUSB drivers over the standard Bluetooth Dongle drivers. Noble then handles the full bluetooth stack for you, bypassing the connection/scanning APIs missing from regular old windows. While a clever way to do that, it's not exactly something you'd want to ship to non-savvy end-users.

BGAPI

Some people aren't happy to just bitbang bluetooth to a dongle though. Instead, they go all the way and implement a specialized API specifically for their dongle. The BlueGiga BLED112 BTLE Dongle comes with a special, proprietary API that allows users to connect to BTLE devices on Windows (and other platforms), also routing around the lack of OS API functionality. So, as long as your platform can talk USB, it can also talk BTLE.

(Found this dongle via an article on the PyGATT library, which supports both Linux dbus and BGAPI for cross platform python bluetooth support).

Conclusion

Well, that's the state of things for the moment. Those are some of the reasons there's no libusb-equivilent for bluetooth yet. Hopefully we'll see Microsoft fill out the Windows API surface soon and make this article a little less sad, 'cause the WinRT stuff is kinda painful. Until then, though, this is what we get to deal with.

Thanks to Sandeep Mistry for filling in some of the details on the Windows situation.

PS Oh, yeah, almost forgot FreeBSD

Using A Firewire Phantom Omni on Windows 8/8.1/10

2016-10-03T19:31:28-07:00

Last weekend I decided to dust off my Sensable Phantom Omni (now the 3D Systems Geomagic Touch, but I bought mine before Geomagic bought Sensable and 3D Systems bought Geomagic), and see if it was still usable. I had to order a PCIe 1394b card, but other than that, I hooked it up, and the Phantom drivers seemed to install correctly on Windows 10. However, when running the Phantom demo software, anytime a program tried to access the motors, the program would crash. Sensor reading seemed ok, but I couldn't get any feedback.

A quick call to the Geomagic Freeform support line turned up the solution. As with many video products that used 1394b, the Phantom Omni requires the "Legacy" firewire drivers. These were included with Windows up to Windows 7, but as of Windows 8 and above, are no longer included with the operating system.

The legacy 1394 drivers can now be retrieved from the Microsoft Support Site.

After installing the drivers and changing the 1394 PCIe interface to use the Legacy drivers (process documented here), I rebooted and the demos worked fine, with force feedback and all!

Along the way, I also found some information on repairing internal cable breakage in the Omni, by a research team at John Hopkins University. The original site had died, but the instructions and images were still available at this link via the Internet Archive.

Anyways, hope this helps others that still want to get some life out of their haptic controllers!

Autodesk Pier 9 Residency Project Lecture: Industrial ASMR

2015-09-19T17:58:10-07:00

I was an Artist In Residence at Autodesk Pier 9 from July 2014 to January 2015, concentrating on a sound art project to extract new and interesting sounds from the prototyping machines around the workshop. There's a video of the lecture I gave covering my time at Pier 9:

OpenTranscripts was also nice enough to provide a transcription of the talk. The project was covered by media outlets such as CNet and re/code.

Upcoming Speaking Engagements and Project Updates

2012-10-21T16:42:10-07:00

Time for more speaking updates!

I'll be part of a panel on using biometrics for generative game audio at AES, on Friday, October 26th, at 2pm. Should be fun!

Also, I've just moved a lot of the health driver projects I currently maintain to the OpenYou organization on github. The hope is to get more developers working on these projects, versus having the world waiting on me to have time to work on things. There's more information available at the post on openyou.org.

Keepon Control via Kinect

2011-11-16T00:12:09-08:00

And the hits just keep on comin'.

I threw this together this evening, in about 4 hours from top to bottom (code + video). Yay open source.

Project uses:

Arduino + Keepoff http://www.github.com/qdot/keepoff - Control of Keepon Robot
Processing http://www.processing.org - tying the whole thing together
OSCP5 http://www.sojamo.de/oscP5 - talk to python script that's controlling arduino, already had that written so didn't write serial controls in Processing
GSVideo http://gsvideo.sourceforge.net - for webcam (filming keepon)
SimpleOpenNI http://code.google.com/p/simple-openni - kinect recording and skeleton tracking in processing
libfreenect http://www.openkinect.org - Cross platform kinect access

Processing running on Linux, X Forwarded to OS X because apparently it's impossible to get good screencast software on linux.

It's missing the side to side bend sensor because I'm still not quite sure how that motor message works yet, but this is good enough for a first demo.

Code is, as usual, available at

http://www.github.com/qdot/keepoff

So, why goth dance? This is a running joke we had at the Art and Code conference at CMU last month. For those not familiar with the youtube meme:

I figured that I might as well turn it into a way to demo new hardware. Other A&&C people, consider the (velvet, tear stained) gauntlet thrown down.

Keepon Hacking Proof of Concept

2011-11-14T00:12:09-08:00

We have Keepon control!

Yay! Thanks to mAngO on the comment thread for my last keepon post, we now know that grounding out the bus during keepon's powerup allows you to act as the master to the bus! This means we can now control the motors and sound, as can be seen in the video above. I'm just controlling motors there, using the Control Program for Android to send OSC messages to a python script I wrote. The python talks to the USB serial port, and the arduino turns the commands coming over serial into I2C to go to keepon.

All the source code for this is available in completely raw, uncommented form at

http://www.github.com/qdot/keepoff

So, that's the first part finished. Now it's on to polishing things out and figuring out the rest of the parts of the hardware we don't have access to yet. I'm keeping the github issues list updated with things we have left to do.

Everything You Ever Wanted to Know about MyKeepon Except for the Parts I Don't Know About Yet

2011-11-09T17:15:09-08:00

UPDATE 2013-06-01:

While this post still has relevant information, the engineers at BeatBots have created a far more stable firmware. I highly recommend using their MyKeepon firmware, as it fixes a lot of the timing issues the KeepOff firmware had. The MyKeepon firmware is available at:

https://github.com/beatbots/MyKeepon

UPDATE 2011-11-14:

Keepon hacking has made a major step! Thanks to mAngO on the comment thread for my last keepon post, we now know that grounding out the bus during keepon's powerup allows you to act as the master to the bus! There's a Proof of Concept video posted on youtube now.. I'm leaving the rest of this post as it was when I first wrote it for history sake, but the information in it plus knowing that you just need to hold down the I2C lines for a second when the keepon powers up are enough to actually get control going. The reverse engineering document and code in the keepoff repository will be updated to reflect this information.

I'm really not sure I've never spent so much time cursing at something so adorable. The past week has been yelling, crying, and generally losing my emotional shit toward a few servos wrapped in a weird, sticky plasticy skin, better known as the MyKeepon Dancing Robot.

How better to atone for my sin of the vivisection of the most adorable christmas toy this year, than writing up what I found. That way, future generations can avoid the pain inflicted on it, and the pain it inflicted on me.

But good lord, it's so fucking CUTE.

Usually I wouldn't write this up until after I had things completely finished, but I gave myself a week deadline for that, and that deadline passed 2 days ago. I'm still in the middle of a few different ideas for reversing it, but those could take a while (stupid real life getting in the way of toy hacking), so I figured I'd dump what information I do have now.

Resources

Before I get into technical descriptions, the current resources available for updates on keepon hacking are:

Default Interfaces (i.e. What It's Supposed to Do)

The MyKeepon robot isn't the most complicated toy in the world. The main selling points of the toy are that it's cute, it's interactive, it can dance to your music, and it's cute.

To establish the interactive cuteness, the following interfaces are available to users:

2 Front Buttons, for switching between "Touch" and "Music" mode
5 Body Buttons, inside the skin of the keepon. 4 around the sides, one on the top of the head.
A Microphone in the nose

There's 4 degrees of freedom in movement for the MyKeepon (X-Axis horizontal, Y-Axis up, Z-axis thru):

Base rotation, that rotates the whole bot around the Y-Axis.
X-Axis bend, for bending the head/body left/right
Z-Axis bend, for bending the head/body forward/back
Y-Axis compression, for 'squatting' (action when top of head button is tapped)

A speaker is also available for playing sounds. There is no volume knob, sounds are always on, and always loud. The secondary market for keepons with volume knobs installed is going to be killer.

Keepon States

MyKeepon has 3 states when powered on:

Touch - Pays attention to buttons, looks around
Music - Listens for music, or dances to rhythm tapped out using head button
Sleep - Processor(s) in low power state, can be brought out of sleep by hitting music button or tapping head, which causes reset line to be pulled.

Mechanisms and Power

Keepon runs on a 12v supply, using either 8AA batteries in what has to be the worst enclosure I've ever had the displeasure of jamming batteries into, or using a 12v plug, not sold with the device (though they give a TON of info about the plug sizes needed in the manual, a surprising, rarely seen bit of information). The 12V runs to the motors, and is stepped down to 3.3v for the internal circuit.

Each of the motors has an encoding mechanism on it, but I haven't really done much work figuring out exactly what it is yet. I believe the largest motor for turning the base is a regular gear motor (see encoding conjecture below), but the bend motors may be small servos.

On the circuit board, there are 3 pins (see picture below) that come in contact with this piece that's mounted on top of the battery pack. It's used to recenter the bot on boot, as no state may be available for motors to know their postion at time of last power off. The bot being centered is the only position where the middle pin won't have an electrical contact with one of the outer pins on the circuit board. This accounts for part of the "startup time" mentioned in the keepon manual.

Circuits and Chips

There's one main circuit board in the MyKeepon, with 2 very odd processors.

The microprocessors are

Padauk P234CS20 (P234 chip in a SOP20 package)
Padauk P232CS14 (P232 chip in a SOP14 package)

Their data sheets describe them as FPPAs, or "Field Programmable Processor Arrays". I have no idea how this makes them special. They're dual core, hence the "processor array", but other than that, they look like they're One Time Programmable, hence the Field Programmable part being... questionable at best. Also, as marcan found, they tend to straight up lift figures and paragraphs from the PIC datasheets.

The two processors talk to each other via I2C. The PS232 deals with sound and encoders and is the slave node on the I2C bus, while the PS234 handles driving the h-bridges, main processing (including handling button presses), and is the master node on the I2C bus. There's more info on this communication in the next section.

There's 3 H-Bridges, running to the motors listed earlier.

I2C Bus and Protocol

The MyKeepon developers have been nice enough to provide pads to access the I2C bus between the processors. Looking at the board with the solder mask facing up, it's on the lower left hand side of the board, marked with a gigantic smilie face. So they were damn well aware of what they were doing.

For those not familiar with the I2C Bus, there's a good tutorial at http://www.i2c-bus.org/.

The pads exist on the bus between the processors, meaning that you can see how the master node addresses the slave node. I2C addresses are for "devices", and there can be up to 127 devices on the bus, that can be written to/read from. As of this writing, only 2 devices have been found:

0x52 - Sound
0x55 - Motors

To trigger a sound, an message is written by the master node to the bus in the format "0x01 0xWW", where WW is the index of a sound. The following indexes are known so far (though more sounds are certainly available, they just haven't been mapped yet):

Motor messages are 3 bytes, of the format "0xUU 0xWW 0xVV", where:

0xUU: Motor Index
0xWW: Motor Position?
0xVV: Unknown

To retreive information about the motors, a read request for 12 bytes is sent to device 0x55. In normal communications between the processors, this request is sent every ~15ms by the master node. Format of the returned information is currenly unknown.

Information about button presses (either body or base buttons) and input from the microphone do not seem to be relayed across the I2C bus.

Accessing the I2C Bus

Since the protocol of the I2C bus is (mostly) known, the problem becomes talking to the devices on the bus. The tap exists in between the chips, and the master node does not seem to have the capabilities to work on a multimaster bus (not to mention, the default animations being timing based means we can't stop moves we don't want from being sent). When the device is asleep, the data and clock lines are pulled to ground, meaning we can't actually communicate to any of the other devices.

The only current solution seems to be to lift the I2C pins on one of the chips. This seems like the wrong thing to do, since the pads are obviously available and there's a smilie face above them, letting on that there's some way to easily access the hardware on the board.

Simple Modifications

I've only made a couple of small modications so far. The first is running a wire from the I2C pads out of the bot. I dremeled a small hole in the back near the plug outlet, and made sure the wire had a LOT of play inside the bot, since turning pulls it around inside.

The second is putting a switch in between the speaker wires. This allows for the sound to be turned off, which is a huge blessing when you're just trying to watch the bus. A rheostat could also be installed to turn the volume down. There's a good amount of room for installation of the rheostat inside the device.

What's Left to Do

The protocol is known for the motors and sound banks, outside of what the 12 byte return from the motor device is. I've actually got a board where I've lifted the I2C pins on the slave processor, and am going to see whether I can communicate directly with it. Everything I'm using that talks I2C doesn't deal with arbitration of a multi-master bus, so it could very well be that the master node on-board will arbitrate correctly if another node interrupts, but I haven't figured out whether or not that is true yet. It really does seem like there should be an easier, non-pin-removing way to speak to the chips on the bus, and I'm really hoping no one follows my lead on pin lifting before we figure it out.

However, once all that's done, we should have a completely USB controllable keepon, which I then have all sorts of ideas for, except NO NOT THAT you pervert.

Console Controls Usage and the Kinect SDK

2011-06-16T14:15:09-07:00

Oh frabjous fucking day. The Microsoft Kinect SDK is out. Along with a license that takes a very, very nasty FAQ to explain.

On this big day in UI development, let's take a look over the current console controls landscape, and what it means to non-game developers.

Why focus on game consoles controls? They've driven down sensor prices like crazy, due to mass manufacturing and required price points for game sales. They've established more than a few careers of non-game-developers now. Uses of the kinect and the wiimote for projects not pertaining to their original console have been all over the media lately. Keeping a forecast of where development for these technologies is going means we have a better idea of how to ride the wave when it comes.

Disclaimers

In terms of licensing issues, I am not a lawyer. I do not play one on TV. However, I do have a lawyer fursona.
While I am part of the OpenKinect project, I do not speak for others involved in the project. All opinions expressed here are my own, and all cursing is far fucking better than anyone else on the project could turn out, so while I may share my source code, I'm not giving them rights to that.
I strive to keep all the information as correct as possible, but, well, I've been drinking.
I am not a game developer. I am a reverse engineer that specializes in controls and interface devices. My view of this hardware is purely from the driver and capabilities side.
I have not directly used the Move SDK or Kinect SDK. But I have read some articles and created very strong opinions, which means they are valid for internet consumption.
This article is only about reversing/using alternative console controllers, not about reversing consoles themselves. There's a completely different history to that which would take much more than a blog post to cover, though I will admit that it does have some influence on the information here..

Nintendo Wiimote

The WiiMote was first out of the gate, so let's start with it.

You get:

100hz update for 4 IR points with 4 bits of depth @ 1024x768
3-axis Accelerometer
Bluetooth Communication
3-axis Gyro with WiiMotion Plus
Extensible Control Port

What Nintendo provide in terms of software

Nothing.

What the community provides in terms of software

The Wiimote protocols have been reversed and known for years, and there's availability in pretty much every language you can think of.

What Nintendo thinks of non-game developers

Are you Zelda? Are you Mario? No. No you are not. Therefore, go fuck yourself.

Nintendo hasn't put out anything in terms of press releases about the DIY community during the lifetime of the Wiimote. They're happy to let them live seperately, and that's a fine strategy. Nintendo has to put zero into support, they aren't actively stopping people from using the wiimote, and developers can survive on their own.

Where Nintendo is going with it

Honestly? Not real sure here. With the upcoming release of the WiiU controller, there wasn't a lot of talk about Nintendo's flaily controls strategy.

Where the community is going with it

Where haven't they? There's Johnny Lee's demos, there's sex toys, there's more "generative art" than you could shake a wiimote at. The Wiimote is as ubiquitous as alternative controls get these days.

With the WiiMotion Plus, it even turns into a nice IMU. The WiiMote still has some life to it as a cheap, extensible sensor platform, especially with the amount of prior usage it has seen already. Outside of a somewhat flakey bluetooth interface, the "just works"ness of it will keep it alive for a while to come.

Nintendo WiiU Controller

I don't even know if this'll exist in a year, but it's been announced, so might as well talk about it.

You Get:

A 6.2inch screen
No idea on communications method, but enough bandwidth to pipe over video in real time
Joystick
3-axis Accel/Gyro
Front Facing Camera
Probably other stuff that I'm missing or they'll add.

What Nintendo provide in terms of software

Well, it's not gonna come out for another 18 months, so I have no idea. It may not even look the same by then.

Where Nintendo's going with it

I... don't know, and I don't think that's good. It's almost too integrated, when we're already seeing new controllers that provide both physical controls and a detachable screen.

Above: Livid Instruments Controller integrated with iPad

These are using tablets that, by the release time of the WiiU, should be under $100. Between the Android OpenAccessories SDK and whatever Apple decides to do, this will be far beyond replicated by release. Not only that, the rumors are flying about the WiiU only working with one controller per console.

In the end, it's really too far out to make accurate predictions for this, though that's obviously not stopping me from trying. I thought the GBA interface to Gamecube games was great, but, well, it could still be a GBA after that too. This... Who knows. It doesn't seems as mind breaking at the Wiimote did, but then again, you only get to one-up the joystick once.

What Nintendo think of non-game developers

You still won't be Mario or Zelda, so you can still go fuck yourself.

More important this time though, why would it be easier for you to work with Nintendo's hardware than with a tablet and another accessory? Nintendo has already proven time and time again they don't care to support developers outside of console games. Doesn't seem like this will gain community traction in its current state.

Playstation Move

Sony's entry into the gaming market: a wand with a light on it.

You get:

A wand with a light on it
Bluetooth connection
Accelerometer
Gyro
EyeToy Camera to watch wand with light on it

What Sony provides in terms of software

The MoveMe SDK, announced at GDC, not yet released. This allows you to get full real-time 3d positioning from Move controllers.

It requires you to get this information via a network connection to your PS3. The tracking algorithms are locked onto the PS3, and you have to have the console running special server software to use the SDK.

To develop for the Move, you have a minimum investment of around $600, for the controller plus the console. A fanboy would tell you that you also get a Blu-Ray player out of that. I am not a fanboy.

What the community provides in terms of software

There's the MoveOnPC project, which is an effort to create open source drivers for using the Move as a control mechanism. They're not real far along yet, but the project is at least active.

What Sony think of non-game developers

"Absolutely adorable" is about the best thing I can think of. They think the homebrew and DIY community is cute, and they seem to view homebrew devs, much like their customers, as lesser beings. Weirder still, they seem to harbor both fear and disdain at the same time. For instance, as seen through the way they released the SDK. They're making SURE you have to have a console, and that you only get what they want, and getting to their algos will require both hardware and software reversing.

Where Sony's going with it

Same idea as the Wiimote, except more accurate positioning. Build plastic toys around it, use those as controls. Oh boy.

Where the community is going with it

This turns the Nintendo view around... Does the community really care? I haven't seen bounties out for the Move. I haven't seen message boards filled with people working on it. This time, it seems like the company would be insulting the community with the way they released their SDK, but the community doesn't even care in the first place.

Microsoft Kinect

For the Kinect, we've got so many different solutions out there now that I'm actually splitting them into their own sections.

You get:

640x480 RGB @ 30hz
320x240 Depth @ 30hz, .5-4m depth range
USB 2.0 High Speed Connection
Microphone Array
LED, Accelerometer

Microsoft Kinect - OpenKinect

OpenKinect was the project that sprung up around the OpenKinect Bounty hosted by Adafruit Industries in November 2010. If you've like a more in-depth history, check out my presentation on it at Maker Faire 2011.

As I said in the intro, while I am part of the OpenKinect project, what I am stating about the project here is my own opinion, and does not speak for other members of the project.

What OpenKinect provides

An open-source, cross-platform method for accessing raw data coming off of the kinect. Nothing more, nothing less. It was the first to publically provide access to images, leds, and accelerometers on all major platforms. Audio support is in the works, and has been taken to the proof of concept stage.

There's been talk of including processing algorithms created by the open source community, in a seperate library. This part of the project has not yet taken shape, though, and most concentration lies in finishing the driver.

What OpenKinect think of non-game developers

Not to sound harsh, but they just don't. They don't really think about any kind of specific developer. The driver exists to be just the driver, and this is the simplest way to serve that goal. You take it and do whatever you want with it. Have fun.

What this means is that, unlike OpenNI and Microsoft's SDK, OpenKinect is the easiest way to get the raw data from the camera. If you are interested in doing something other than skeleton tracking, this makes it the lowest barrier to entry.

Where OpenKinect is going with it

At this point, it's all about finishing providing the features of the camera. This mostly has to do with the audio core, as the camera features are fairly well covered.

In terms of what will happen with OpenKinect now that the Microsoft SDK is out, I think the answer is "not much". There's a fork of OpenNI's sensor library that uses OpenKinect to talk to the kinect and provide the rest of the OpenNI capabilities for that camera. MS has no interest in supporting non-windows 7 platforms, so there will certainly still be a place for OpenKinect in the Kinect ecosystem.

Not only that, Windows has been by far the hardest platform to deal with for development and support for OpenKinect. I actually hope that we can build an API wrapper around the MS SDK, to make it fit with OpenKinect without having to switch out drivers. However, having not read much of the MS SDK documentation as of yet, this remains to be seen.

Also, with the licensing terms as they currently are, OpenKinect remains commercially viable on all platforms, while Microsoft's SDK specifically prohibits that kind of usage.

Microsoft Kinect - PrimeSense/OpenNI

A few weeks after the OpenKinect project put out the source for their library, depth camera chip manufacturer PrimeSense followed suit with their own SDK, known as OpenNI.

OpenNI is both a library and an initiative. The library is the implementation of a standard that PrimeSense is pushing to be a standard SDK for future depth sensors. The initiative part includes multiple companies on board with this standard.

What PrimeSense provides

PrimeSense provides 3 different libraries, all cross-platform, with varying open/closed source policies:

Sensor - Hardware access library. Their version only supports their PrimeSense SDK camera, but there's a version that's build on top of OpenKinect for kinect access. Open Source.
OpenNI - This is supposed to be an "abstraction library for depth cameras". Which I guess translates into "huge C++ beast". It's a way to abstract depth camera information so you can use anything providing depth data with any program that will take depth data. Seeing the only consumer depth cameras are PrimeSense's, this works out well for them. Open Source.
NITE - This is PrimeSense's body/skeleton tracking library. Unlike Microsoft's algorithms, NITE requires a calibration pose to find skeletons, meaning you have to stand in front of the camera and hold your arms in a certain way for it to find you in the scene. This can be flakey sometimes. Unlike Microsoft's libraries, NITE is available for commercial use. It is closed source, but the binaries are free.

What they think of non-game developers

Developers, via OpenNI, are marketing for PrimeSense.

Funny enough, PrimeSense doesn't actually care about game developers. It's not really even their domain. Primesense wants to own the home theatre remote control market, hence their focus on things like the Asus Wavi Xtion. The battle for being able to control your media consumption is far larger than the battle to control how you flail in front of your TV and call it control. More people watch TV and movies than play video games, and the amount of hours logged on watching activities is exponentially higher. TV Remotes are getting unwieldy now, so the next step that is apparently logical to someone with money is that we now wave our hands around to start and stop our movies, or change our channels.

On top of this, PrimeSense is not a camera company, they are a chip company. They need to be able to sell their chip in large quantities to people who will manufacture cameras with it. Therefore, if developers make applications with OpenNI/NITE, and OpenNI/NITE will "just work" with any PrimeSense camera, then PrimeSense gets to claim they are "open", and anyone that manufactures a camera with a PrimeSense chip will have access to all the applications that've already been written by other developers.

Where they're going with it

OpenNI/NITE is another weapon in the battle for the home theatre as home information hub. That's why PrimeSense wants the chip and software everywhere, not just in kinects. So, they're betting both for and against Microsoft at the same time, which is a very interesting position to be in.

If NITE doesn't support non-calibration poses very soon, it's going to lose out to SDKs that do. MS has proven you can track bodies without calibration, even if it does take thousands of hours of video to analyze through advanced algorithms. Now that MS's SDK is out and does not require calibration poses, people's expectations are quickly going to change. NITE still has a foothold on the commercial licensing side, but that doesn't help much for consumer expectations of product.

Microsoft Kinect - Microsoft's SDK

On June 16th 2011, Microsoft released their own SDK for the kinect, much to the surprise of just about everyone, since there had been no communication since March about it.

As of this writing, the SDK is still considered "beta", and is only available for non-commercial use.

What Microsoft provides

Seamless skeleton tracking, which is a huge deal. While the algorithms have been released and replicated in open source, the advantage that MS has here is big data. Thousands of hours of training sets to send through the algorithm so that it can find any body type, meaning that people can walk in and out of the scene and instantly be recognized and tracked. Seriously fucking BIG deal.

There is also access to the audio system, which no project has gotten a complete hold on yet. OpenKinect has some access to the raw audio stream, but Microsoft provides a voice SDK that allows developers to identify and position users based on sound.

What they think of non-game developers

With the beta licensing terms, developers are somewhere between fanboys and marketing. Open Source developers are back on the bad side, too.

The Beta SDK FAQ is chock full of interesting issues, including:

Not being able to distribute applications standalone, users must also download SDK to get the runtimes
No commercial use, and on top of that, due to the fact that MS cannot predict the usage of the kinect SDK, all SDK derived applications should not be considered "allowed under the SDK".
Microsoft owns the right to say what software you can use the hardware with, and using the kinect with anything outside of the SDK is not allowed. Even with this wording in place, the MSDN Channel 9 launch video lauded all of the open source applications currently available for the kinect.
The SDK will not run on Virtual Machines
Refusal for any SDK derived application code to be released under copyleft licenses (GPL, etc...)

In other words, you can write kinect apps for yourself, then you can upload videos of those apps to show off how awesome the SDK is, but redistribution is not allowed, and neither is selling. There's no word on when a commercial SDK will be available or how much it will cost.

Of course, MS can't really track most of what they claim in the FAQ, but it seems to be worded in a very predatory way. It's still better than Sony's "lock down the algorithms on the console" idea, and it's not really enforcable for a lot of projects. Whether anyone but nerds like me will give a shit about that is a completely different question that we're not going to answer because it's my blog and I'm the center of the universe here.

Where they're going with it

Everywhere, and into everything, as fast as possible. Which, for a company the size of Microsoft, will not be all that quick. There's already talk of Windows 8 shipping with kinect drivers.

With the algorithms and samples Microsoft has put out in their SDK, they've jumped way ahead of the other solutions in the NUI game. Without the calibration pose requirement, MS SDK programs should "just work" for people coming in to/out of the scene. It adds access to the audio core that no other solution can get anywhere near at the moment. For the time being, MS is now winning the NUI game in terms of capabilities available to developers.

At least, for developers want to make non-distributable demos. Since the license is still non-commercial only, that's all they're going to win. This could end up pissing off the industry enough that they find some way to replicate it without Microsoft's terms. At that point, we have a very fun war on our hands.

Where the community is going with it

There's not much of a community to speak of since it's been out all of 10 hours as I write this. While I know CodePlex is huge, I've never really dealt with the MS Open Source Community, or, well, any MS community period in the past decade or so. I guess we'll see what happens. I'm certainly interested to see how the cultural philosophies of OpenKinect versus MS SDK influence the projects that come out of them.

For a better perspective on this, I defer to Josh Blake, who I am signing up to write an article on it before even asking him. Hi Josh!

Conclusion ##

So, that's how I see things going for creative outside development on console controls for the time being. Take it with a variable sized grain of salt.

For the wiimote, it's gonna keep dropping from an already fairly cheap price, and the fact that you don't have to solder things to it makes it very handy for prototyping. It'll live for a while longer.

I wouldn't be surprised if the move never sees the light of day in the maker community. It's just not getting any sort of traction from either side, and it seems easy enough to just replicate at some point.

For the Kinect, MS has a good grip on the NUI market now. However, there are many other uses for depth cameras that don't require a body to be in front of it. 3d modeling, robotics, art, etc... Yes, the kinect was made to track bodies, and that's what the camera range and other properties of the camera reflect. While the MS SDK will make access the raw data easier on windows, it won't completely overtake the Kinect development world.

It'll also be interesting to see if the open source community can figure out a way to source enough data to train their own algorithms, and provide MS with some competition. We've thrown this idea around at the SF 3D Vision Meetup and in the OpenKinect IRC channel, but that's a huge undertaking. It signals the next big move for the open source community though. We've proven out big code, and see huge projects released as open source. Now it's time to start playing more with open big data.

Upcoming Speaking Engagements

2011-05-13T20:12:12-07:00

This summer is shaping up to be a busy one...

I'm speaking at Maker Faire. Twice, even!

2011-05-21 3:30pm - OpenYou.org Presentation, Health 2.0 Stage
2011-05-21 6:00pm - OpenKinect Presentation, Main Stage

Then there's the Quantified Self Conference on May 28-29th, 2011, at the Computer History Museum in Mountain View, CA. There's no central presentation, but honestly, I probably won't stop talking at any point during the 2 days, as I have a table at the expo, plus will be helping out with the health hardware session and the hackathon.

On June 23rd, I'll be doing a presentation on the OpenKinect project at NetExplorateur Zoom 2011 in Paris.

XIO - Novint's Gaming Exoskeleton

2011-04-24T22:12:21-07:00

I'm not really used to writing about hardware that's hardly to prototype stage yet, but damn, I cannot wait 'til this comes out so I can start reverse engineering it.

Novint, the company that manufacturers the Falcon haptic device (that I wrote/maintain the cross-platform libnifalcon for - if you aren't familiar with the falcon, check out this rather exhaustive article I wrote on it a couple of years ago), recently announced a merger with another company.

The other company in the deal, Forcetek Enterprises, doesn't even seem to exist outside of the PR about this merger (UPDATE: Ok, I actually found their old website finally. Apparently this was shown at E3?). Successful stealth mode.

What came out of the merger...

is a partial exoskeleton for gaming.

The XIO is a sleeve exoskeleton that allows you to feel forces throughout the arm, versus just through the hand like the Novint Falcon. There is actuation along the arm and elbow, meaning much larger force distribution and a much more immersive feel. In applied terms, this means that you'll be able to feel things like gun kickback all the way through your shoulder, versus just your hand. You can also do interesting things like simulating weight and fatigue in the arms by restricting certain movements.

This combined with depth cameras like the kinect could be HUGE. Players would have both full body tracking AND at least partial body actuation, which is better than the "flail without feedback" option we've had for years with the Power Glove/P5/Wiimote/Kinect/Move.

The demo video below shows a full VR rig built from consumer hardware, using

Vuzix HMD
XIO
TNGames Third Space Vest (Which I also write drivers for!)
Some new gun controller

The XIO is supposed to integrate with Novint's F-Gen drivers. F-Gen is an abstract programmer layer (similar to GlovePIE) made to implement haptics on top of arbitrary games, instead of doing direct game integration (which Novint has with things like Source Engine games and the Penumbra series). This allows users to possibly script haptics to whatever game they want. How well this works, I have no idea, but it means that any game the falcon supports should also be supported by the XIO on release.

Novint's yet again done a horrible job of the PR with this one, as it looks like no one has really picked up the news yet even though the press release happened over a week ago, and now all of the images on the front page of their site are broken. Novint's CEO even changed his twitter account on the launch for reasons I'm not real sure of.

That said, I've been incredibly happy with Novint's quality of engineering on the Falcon. Novint knows how to make extensible hardware, as they've shown with the grip and firmware system on the falcon. It was a joy to reverse engineer, and I'm hoping that follows onto this as well. I can't wait to get my hands on (and in) the XIO.