Deconstructing the NUI

Windows 8 on Microsoft Surface 1.0 (video)

2011-09-15T05:33:00.000-04:00

I've been neglecting this blog a bit since I'm trying to focus on finishing my book, but I had to break my self-imposed radio silence to break this news:

I installed Windows 8 on Microsoft Surface 1.0 hardware, and it works! By "it works", I mean the Microsoft Surface 1.0 shell and applications run as expected on Windows 8, and immersive-mode applications and regular desktop applications can be controlled using the Surface vision system touch input.

Windows 8 Start screen working on Microsoft Surface 1.0

I tweeted the above photo while video taping proof of this technical hackery. The video is embedded below:

Yes, this is the Windows 8 that was released publicly only a little over a day ago. I did not have early access, so this is another 24 hour hack for me like Kinductor from the Kinect Code Camp, though it didn't take me a full 24 hours of straight effort.

How did I do it?
Since Surface 1.0 is intended to run on Vista, the fact that any of this works is a testament to the backwards compatibility story of Windows 8. In short, I shrunk the Surface's Vista partition then installed Win 8 from USB key to the new partition (make it 40 GB minimum if you want to develop on it). The installer automatically set up dual booting, so I can boot back to Vista for the "correct" Surface 1.0 environment if I like.

The tricky part was getting the Surface SDK (full version that comes on Surface, not the workstation edition) to install (edit the msi launch conditions using Orca) and then getting the inputs from the Surface vision inputs to drive the Windows 8 UIs. To do that I used MultiTouchVista, which has a component that can connect to SurfaceInput.exe as a client as well as push touch data to an HID touch input driver. Windows 8 doesn't know it's not a capacitive touchscreen at all. (This technique also works for combining Windows 7 with Surface.) Win8 UI also won't be aware of tags or blobs but if you want to write a Surface application that uses those, just use the Surface SDK! Some other time I'll post a deep technical walkthrough, but this thread should be sufficient to get you started.

All of the Surface hardware drivers and Surface applications run great. The immersive mode touch responsiveness is a bit flaky as you can see in the video, but this could have been any number of things from the MultiTouchVista plug-ins to the Win8 touch stack to the Start screen. The Win8 touch visualizations (trails and auras) were always working fine, so I'm leaning towards the problem being in the Win8 UI layer. This is a pre-beta build though, so entirely understandable.

I do need to say that this configuration is completely unsupported and would probably violate the warranty if it wasn't already expired for this particular machine. It also has nothing to do with Surface 2.0. You won't ever see Microsoft create a configuration like this for general use since running three shells (Surface, Immersive, and Desktop) side-by-side would be entirely confusing for real users. In the video I was basically in the equivalent of Surface administrator mode, manually running SurfaceShell.exe, and then switching ad-hoc into immersive mode, which is like a full screen start menu. There are too many competing metaphors and modes for this configuration to be used by the general public.

Does Immersive mode make the Surface shell obsolete?
No way. It makes the Surface shell even more important. My initial impressions with actually using the Win8 immersive mode on Surface is that it really doesn't work for a large, horizontal, multi-user form factor. For one, everything is oriented in one direction, and two, I need to reach too far to get to certain interface elements across the table. That's the opposite of the ergonomic intent of the immersive mode, which was design for smaller screens and slates. It might do better on a vertical wall-mounted form factor. There is definitely a place here for both Immersive mode as well as the Surface shell since they are each designed to do different things in different scenarios. This is especially true if we account for the improved Surface shell that is coming soon with Surface 2.0 hardware.

I think we'll see that there will is a large difference between the design of an immersive mode application to run on a Slate or PC and used by one person, and the design of Microsoft Surface applications to be used by multiple people in a 360 degree interface.

What Surface and Win8 can learn from each other
Even though the designs for each of these modalities need to be different, there is a lot that they can learn from each other. I really like swiping in from the edge to open the charms and app bars, and could see that being incorporated into the version of Surface after 2.0. Windows 8 isn't perfect though, since as soon as there is an extra finger somewhere in immersive mode, the whole interaction breaks down.

It may just be the developer bits, but many or most of the Win8 sample games were almost unusable as a Surface application. My 3 year old tried to play a few with me but we kept getting in each others way since the games were designed for a single orientation. Worse than that, my touches would block her from interacting with the game, and vice versa. Some of the interns that developed those samples apparently did not account for multiple types of input (mouse and touch) or even multi-touch at all, indicating they were still thinking about coding interactions from a single-point mouse point-of-view.

I know that on slates and PCs with a single user they may not need to plan for a full 360 degree interface, but Windows 8 developers need to learn the lessons that Microsoft Surface developers have learned years ago about allowing multiple inputs, multiple touches, and letting the user recover from mental mistakes and accidental touches.

Watching the Windows 8 unveiling during the Build Day 1 keynote was somewhat amazing for me because all of a sudden NUI design and development skills (my focus since 2008) has gone from a small niche to a core requirement for the Windows platform. Anyone interested in building truly great Windows 8 applications needs take an active interest in learning from Surface developers. I'm expecting that some of my colleagues in the Surface community will end up coming up with the most amazing Windows 8 applications in no time at all. (Feel free to send those apps to me for testing on my FrankenSurface Window 8 machine!)

Natural User Interfaces in .NET
While at first I was nervous that the Windows 8 announcements would make my almost-complete book a waste, now that I think about it things are still pretty good. If you want to use the premier technology for creating Natural User Intefaces, then .NET and WPF is still the way to go. You can't target Surface or Kinect in immersive mode, and the immersive mode interface isn't well suited for multiple people. You also have to consider that Windows 8 is still six months to a year away, but you can develop NUIs for Windows 7 multitouch, Surface, and Kinect today with WPF. You can even give them a metro look-and-feel.

If this interests you and you haven't already, take a look at the free sample chapter of my book, and if you like it pre-order it and get the early access PDF today with all of the completed chapters.

Official Kinect SDK to be Open Source

2011-04-18T13:54:00.001-04:00

Microsoft had previously announced its plans to make the Kinect for Windows SDK available this Spring, initially for non-commercial use and later for commercial use. At the MIX11 conference this past week during the day 2 keynote (starting at the 1:40:10 mark of the video) they reiterated this plan and gave a few more details, including support for C++, C#, and VB development. They did not mention it explicitly but they used WPF during the live coding demonstration.

On day 3, several Kinect and Microsoft Research (MSR) presented a panel Q&A session. At 26:05 of the Kinect panel video, I introduced myself as the founder of the OpenKinect community and asked whether they had considered embracing the open source Kinect movement started by this community, which has lead to hundreds of Kinect Hacks around the world, by open sourcing their code.

To my surprise, Dr. Kristin Tolle, Director of MSR’s Natural User Interface team, responded that they will be publishing the Kinect SDK as open source! She referenced earlier discussion about embracing and enabling the community to take what Microsoft has created and using it in new ways and mentioned that MSR has a history of releasing open source SDKs.

When I returned to my seat, another MSR researcher approached me and suggested that Kristin may have misunderstood my question or had misspoken; however, I followed up with Kristin afterwards and reconfirmed that she meant open source as in the public can modify and recompile the sources as well as share the modifications. She also emphasized that they want to enable people to share modifications and enhancements.

This is great news!

It is not clear yet what type of license they will use. If the license allows modification and redistribution but is, at least initially, limited to non-commercial use, then it would not be a standard open source license according to the Open Source Definition. Looking at other projects published by MSR, I could see them choosing the MSR-LA as used by the Singularity experimental OS.

I’ll restate here that Microsoft has on multiple occasions confirmed that a commercial license will be available at a date after the initial non-commercial release. The ideal scenario would be if they switched from MSR-LA to a true open source license and set up a collaborative project between MSR, Kinect team, and the open community styled after the successful Nuget project, which uses the Apache 2 license and includes Microsoft and non-Microsoft contributors.

Considering the reaction by the other MSR researcher to this open source statement, there may be some confusion internally about whether this will happen like this. I think the best thing that we as a community can do is to express support for Microsoft open sourcing the Kinect for Windows SDK (including the skeleton tracking and training components), even if they use a non-commercial license initially, and keep the pressure on them to create a better relationship with the open community and transition to a Nuget style collaboration with a liberal license such as Apache 2.

Update 4/18 7:34pm: Mary Jo Foley picks up this story, but the Microsoft spokesperson she talked to denied that the Kinect SDK will be open source. As she notes, Microsoft has pulled 180’s before regarding Kinect. After spokespeople initially were hostile to the idea of Kinect hacking, Xbox executives later embraced the idea that people are using Kinect for non-gaming purposes on the PC. Let’s hope Microsoft stays open to this idea.

NUI at MIX11

2011-04-04T21:55:00.001-04:00

It's only a week away from MIX11 and I'm very excited. NUI will be represented even more than last year. Here’s a run down of all the NUI events I’m aware of:

Monday, April 11 6-9pm Ballroom D	Open Source Fest A showcase of 52 open source projects, including InfoStrat.MotionFx, our project for integrating Kinect with WPF 4 Touch.
Tuesday, April 12 2-3pm Lagoon H	A Whole NUI World: Microsoft Surface 2 and Windows Touch by Luis Cabrera of the Surface team. Luis will get everyone up to speed about developing with Surface SDK for Surface 2 and Windows 7 touch devices.
Tuesday, April 12 3-4pm Lagoon H	The Microsoft Surface MVPs Present: Natural User Interfaces, Today and Tomorrow; An Interactive Discussion and Demonstration by me and the rest of the Surface MVPs. We will have several cool demos for Surface 2 and Kinect. (This is the open call session I submitted and voted in by the community. Thanks!)
Tuesday, April 12 7-9pm Lobby in front of The Commons	NUI MIXup at MIX11 NUI MIXup is back! RSVP at the link above. Last year’s NUI MIXup was a great success. Frank La Vigne, a Microsoft Public Sector Developer Evangelist, will be sponsoring drinks and appetizers for the group. I hope to see you there!
Wednesday, April 13 11:30-12:30pm Lagoon H	Audio for Kinect: From Idea to "Xbox, Play!" by Ivan Tashev of Microsoft Research, the man behind Kinect’s audio processing algorithms.
Thursday, April 14 12-1pm Lagoon H	Interactive Panel: Kinect and Natural User Interfaces (NUI) A panel featuring luminaries from Microsoft Research talking about Kinect
April 12-14 Breakers L	UX Lightning Sessions Three UX Lightning sessions, each with four speakers. Not purely NUI, but probably still of interest to most NUI enthusiasts.
April 12-14 Shorelines A	The Connect Lounge Microsoft Research and Coding4Fun will be showing off some cool NUI and Kinect projects.

I can’t say where yet, but InfoStrat will also have a presence with some of our demos. I’m also hearing rumors of some things that might be announced during the keynotes, but we’ll just have to wait and see.

If you’re looking for me at MIX11, the best way is to check my twitter stream @joshblake or send me a mention. You can also email me joshblake atgmail com.

MIX11 Open Call Surface and NUI Sessions

2011-01-27T03:31:00.001-05:00

MIX is a great conference that focuses on development, design, and upcoming technology trends. Last year, MIX started an “Open Call for Content” where the general public was invited to submit sessions and the general public voted to decide which sessions would be given actual spots during the conference. The Open Call sessions can be about any topic the community desires to hear and they augment the Microsoft sessions that make up the meat of the conference.

This year Open Call is back and there are 228% increase in proposals! Out of 384 entries, they posted 207 sessions. I read through every single one yesterday and expected to see some great sessions and some mediocre sessions, but to my surprise almost every single session sounds fantastic. MIX11 is going to have some great content.

Last year, I was fortunate enough to be one of 12 MIX10 Open Call sessions. I presented “Developing Natural User Interfaces with Microsoft Silverlight and WPF 4 Touch [and Windows Phone 7]”. (Read my MIX10 recap.) My session was one of three that focused on NUI, multi-touch, or Surface topics.

This year there are 8 9 entries about Surface or NUI and I’m hoping several will be picked. I’m organizing a panel session featuring the Microsoft Surface MVPs, some interesting discussion, and some cool demos. You can vote for up to 10 sessions, so please consider including the Surface MVP panel and any of the other NUI sessions that sound interesting to you in your voting.

The Microsoft Surface MVPs present: Natural User Interfaces, Today and Tomorrow; an interactive discussion and demonstrationJoshua Blake; Neil Roodyn; Dennis Vroegop; Rick Barraza; Bart Roozendaal; Josh Santangelo; Nicolas Calvi

The Natural User Interface (NUI) is a hot topic that generates a lot of excitement, but there are only a handful of companies doing real innovation with NUIs and most of the practical experience in the NUI style of design and development is limited to a small number of experts. The Microsoft Surface MVPs are a subset of these experts that have extensive real-world experience with Microsoft Surface and other NUI devices.This session is a panel featuring the Microsoft Surface MVPs and an unfiltered discussion with each other and the audience about the state of the art in NUI design and development. We will share our experiences and ideas, discuss what we think NUI will look like in the near future, and back up our statements with cutting-edge demonstrations prepared by the panelists involving combinations of Microsoft Surface 2.0, Kinect, and Windows Phone 7.

[Vote for this session here!]

I also want to highlight my InfoStrat colleague Josh Wall’s entry, which should be very interesting.

Microsoft Surface at the Smithsonian – 30 seconds to MagicJosh Wall

How do you design a Microsoft Surface application that creates a magical experience in less than thirty seconds, uses real-world physical objects, and is both fun and educational for kids? That is quite a task, but oh yeah – you need to build seven of these magical applications that are distinct yet connected by a common theme. This was the challenge presented to InfoStrat, a Surface Strategic Partner in Washington DC, by the Smithsonian Institution. This session will explore the design and development process that went into building the innovative physical object interactions for a suite of Microsoft Surface applications used in the “Wonder of Light: Touch and Learn!” exhibit at the Smithsonian Institution in Washington, DC. Josh Wall, director of the InfoStrat Advanced Technology Group, will demonstrate and analyze five different types of physical object interactions that take advantage of the unique vision system in Microsoft Surface. He will also discuss lessons learned from the experience, including designing Surface applications for kids, when to use a linear or non-linear task path, and how to apply Natural User Interface (NUI) concepts to interactive museum exhibits.

[Vote for this session here!]

Here are the other NUI sessions:

Microsoft Surface v2 – designing for the new form factor
Josh Santangelo

Building Really Social Software
Neil Roodyn

Wave, Touch, Pen, Speech, Mouse and Keyboard
Neil Roodyn

How to build a great Microsoft Surface application
Neil Roodyn

How NUI can help to perform complex tasks, like flying a helicopter
Bart Roozendaal

Understanding the Metro design language – from Zune to mobile to Microsoft Surface and beyond
Nathan Moody

[Update 1/27 9:40am: I overlooked the following session! Sorry Davide!]

Silverlight, Windows Phone 7, Multi-touch and Natural User Interfaces
Davide Zordan

There are also many other interesting topics, but I hope you’ll help vote in the NUI and Surface sessions. The MIX content team takes into account the popularity of various topics. (For example, last year they accepted 12 Open Call sessions when they were originally planning only 10.)

Voting is open until midnight PST on February 4th, so lock in your votes now!

P.S. Here is a really nifty pivot viewer for visualizing and filtering the Open Call entries: http://www.xpert360.net/MIX11/

Natural User Interfaces in .NET book update

2011-01-20T00:27:00.001-05:00

Today Manning pushed out a new update to my book’s MEAP (early access Ebook). This is the first update in quite a while, and that’s because I’ve been secretly updating chapters and writing new chapters that include NDA content.

Note: please read through to the end of this post for a 50% off coupon for my book good until February 20, 2011.

The secret NDA content is about the future of the Surface platform and Surface SDK. Now that Surface 2.0 has been announced, I can start talking about it and pushing out the non-NDA content. You’ll notice in this update an updated table of contents and a new title, Natural User Interfaces in .NET, to reflect the reorganized content and focus. Here is the new cover:

Based upon reader and reviewer feedback and the announcement of Surface 2.0 and Kinect, we decided to focus the book on the WPF 4 platform with Surface SDK 2.0. The combination of WPF 4 and Surface SDK 2.0 makes the most advanced NUI platform available that targets a wide variety of devices, form factors, and NUI input technologies. Surface SDK 2.0 is the successor to both Surface Toolkit and Surface SDK 1.0 and will run on any Windows 7 machine, including desktops, tablets, slates, and Surface 2.0 units.

I’ve also decided to include Kinect content because I have determined that we can use the same WPF 4 and Surface SDK 2.0 platform to target depth sensing camera interfaces as well. In early experiments, I have injected hand tracking and hand pose information into the WPF 4 touch stack and can control any WPF 4 touch interface with my hands. Using this unified NUI platform, you can write a single interface that users can use with all of the Windows 7 multi-touch devices, Surface 2.0, as well as Kinect and similar depth sensors.

Of course, you are going to want to design the interface differently based upon which input device your application is using. For example, Surface lets you detect tagged objects and finger orientation and hand tracking depth sensors can give you extra information on hand poses and gestures. That’s why I’ve organized the book into the three parts:

Part 1: Introducing NUI Concepts
What developers need to know about NUI design

Part 2: Learning WPF Touch and Surface SDK
A unified NUI platform

Part 3: Building great experiences
How to take advantage of the unique capabilities of your input devices

I still have a few more chapters to complete, but the content complete milestone is in sight. I’ve been getting great feedback from my MEAP subscribers and continue to make improvements based upon feedback. (Anyone who pre-orders the eBook or print book is also subscribed to MEAP updates and can read the completed chapters as they become ready.)

If you haven’t pre-ordered my book yet, you can buy the eBook or print book (and get the MEAP for free) for 50% off until February 20, 2011. All you need to do is enter in coupon code multinet50 during checkout. If you buy it directly after clicking this link, Manning will give me a couple extra bucks and I would appreciate it. You can also download a free sample chapter and the sample code from that link.

OpenKinect: Creating next-generation interfaces, Jan 20, Arlington, VA

2011-01-18T23:55:00.000-05:00

This Thursday, January 20 at 7pm I'll be presenting at the CapArea.NET Silverlight SIG in Arlington, VA. If you're in the area, please come by.

Navy League Building
First Floor Conference Room
2300 Wilson Blvd
Arlington, VA
[Bing Maps | Google Maps] and is Metro accessible (the Court House station).

Free street parking after 6pm.

Presentation Title:
OpenKinect: Creating next-generation interfaces

Presentation Description:
Kinect is awesome. Kinect on your PC is even more awesome. An international community of developers creating free, open source drivers and software for using Kinect with your PC is the most awesomest of all. That is what the OpenKinect Community is about. In this session, Joshua Blake, founder of the OpenKinect Community, will tell the story of how the community worked together to open up Kinect, demonstrate Kinect working on a PC and some videos of what other people have created so far, and show you how you can start programming with Kinect and create next-generation natural user interfaces.

Kinect hand tracking with WPF 4 and Bing Maps 3-D

2011-01-14T02:05:00.001-05:00

I’ve been playing around with Kinect and PrimeSensor in WPF for a bit and have achieved a small technical success that I wanted to share in a new video from InfoStrat. Using the same techniques that allow us to use WPF 4 on Surface 1.0, we can now use depth camera hand tracking to control multi-touch applications.

Here is a very rough proof-of-concept where I’m controlling the InfoStrat.VE WPF 4 multi-touch control using a depth camera.

Bing Maps controlled with Kinect 3D-sensing technology

This is just a multi-touch application and I have added my one line of code to enable hand tracking to feed the WPF 4 touch stack. I also display outlines of the tracked hands to provide better feedback about what is going on. In this video I also used OpenNI and NITE from PrimeSense.

The tracked hands can participate in all of the multi-touch manipulations and gestures that you’ve already written for your touch application. You can even interact using hand tracking and touch at the same time in the same window. The code that enables this is part of our internal InfoStrat.MotionFx project and will eventually be open sourced, but it needs a bit more work to be practical.

One enhancement we’re planning is adding hand pose extraction, including palm orientation and finger positions. This would allow you can use hand poses and hand gestures to control whether a hand is “touching” the screen or not instead of the current technique. Currently, it determines that a hand is “touching” if it is over a certain distance from the shoulder. Knowing the hand pose would also enable new types of interactions, just like finger orientation, blob size, and tagged objects on Surface enables new interactions.

In the end, we will want to design interfaces that use motion tracking to take advantage of the unique capabilities of that modality to create truly natural interactions. (Pinch-to-zoom with large arm movements is not the most natural interaction.) What I’ve shown above, though, is that it is feasible to use the WPF 4 Touch stack and Surface SDK as the unified platform for both multi-touch and motion tracking modalities.

Microsoft Surface 2.0 Data Visualization Controls by InfoStrat

2011-01-06T21:48:00.001-05:00

On Wednesday at CES2011, Microsoft finally announced the next version of Microsoft Surface. It takes advantage of a new technology called PixelSense where IR sensing pixels are interwoven with display pixels right in the LCD display itself. The computer specifications were updated and form factor went through a major overhaul, making Surface 2.0 one sexy table computer:

My employer, InfoStrat, is a member of the Surface Technology Adoption Program (TAP), so we’ve had access to early SDK builds and have been giving feedback to the Surface team since May 2010. (Serious feedback… at the last TAP event I won a backpack full of swag for finding and submitting the most bugs during the TAP. Don’t worry, the bugs were all addressed.)

InfoStrat and other Surface TAP members have also been working on applications using Surface SDK 2.0. If you were watching the Microsoft CES2011 keynote (skip to 44:15), then you saw one of those applications. Unfortunately there wasn’t time to feature all of the partner applications during the keynote, but that doesn’t mean that we can’t show them to you anyway!

Today InfoStrat announced that we will be open sourcing several of our WPF 4 data visualization controls that we designed for Surface SDK 2.0. This includes the DeepZoom and Pivot controls I’ve been hinting about on twitter as well as a few others. Unfortunately, we cannot publish the controls or code quite yet. We have to wait until Surface SDK 2.0 becomes available to the public.

We’ve been working on these controls for several months and so I’m very excited to share with you a video of these controls used in our Surface 2.0 application, InfoStrat Show for Microsoft Surface. This application is designed to take advantage of the unique capabilities of Microsoft Surface to enable natural and engaging presentations and small group collaboration. This video is just a sneak peak at the controls. We’ll be sharing more as we get closer to the code release. Below the video is InfoStrat’s press release about the controls.

Microsoft Surface 2.0 Data Visualization Controls by InfoStrat. Also shown are the updated visual styles of the Surface SDK 2.0 controls.

For Immediate Release

9 a.m. PST

January 6, 2010

InfoStrat Releases Next-Generation Data Visualization Controls for Microsoft Surface 2.0

Washington DC – January 6, 2010 – InfoStrat today announced plans to support Microsoft Surface 2.0 by releasing a control suite that accelerates the development of next-generation multi-touch data visualizations. The controls will be made available as open source software at no charge on CodePlex.com in the first half of 2011.

This data visualization control suite provides multi-touch versions of the following controls:

§ Deep Zoom multi-resolution image control that allows high performance display of very high-resolution imagery

§ PowerPoint Viewer which enables slide decks to be arranged and presented using multi-touch

§ Pivot Viewer chart control that allows dynamic sorting and categorization of data

§ Physics Canvas which provides an infinite, dynamic canvas for viewing and organizing content

Other features of the controls:

§ Works on both Microsoft Surface and Microsoft Windows 7 with touch

§ A single application built with the data visualization framework can support multiple hardware form factors including: horizontal multi-touch tables, tablets, and large format vertical touch screens

§ Innovative object recognition to enable rapid data manipulations (only on Microsoft Surface)

Watch a sneak preview of the control suite on YouTube: http://www.youtube.com/watch?v=lEVtjHlrf4I

InfoStrat is a member of Microsoft’s Technology Adoption Program (TAP) for Microsoft Surface. As a Microsoft Surface 2.0 TAP member, InfoStrat receives early access to hardware and software, allowing InfoStrat to gain expertise and influence the development of the product before it was released to the public.

In 2008, InfoStrat solved the problem of using Bing Maps 3D on Microsoft Surface in a way that performed well and was WPF-friendly. InfoStrat open-sourced the solution as a reusable control for the WPF and Surface community. Since then, the control has received over 120,000 page views and has over 8200 downloads, and has also been featured in many of our own applications. This control, known as InfoStrat.VE, has become one of the most popular controls for building mapping applications on Microsoft Surface: http://bingmapswpf.codeplex.com

“We are proud to be part of the Microsoft Surface development community,” according to Jim Townsend, president of InfoStrat, “and excited about the possibilities of Microsoft’s new version of Surface.”

Microsoft Surface provides a new way to experience and use information and digital content, engaging the senses, improving collaboration and empowering people to interact. Microsoft Surface is at the forefront of developing software and hardware that uses vision-based technology to fundamentally change the way people use computing devices. More information can be found at http://www.surface.com.

Information Strategies ("InfoStrat") is an award-winning Microsoft Gold Certified Partner and a Microsoft Surface Strategic Partner and member of the Technology Adopter Program.

For more information, press only:

Josh Wall, InfoStrat, (202) 364-8822 ext. 202, joshw@infostrat.com

Interview: Tamir Berliner of PrimeSense

2010-12-09T08:00:00.001-05:00

The natural user interface space has been very busy recently. Last month after Microsoft launched Kinect for Xbox, I organized people interested in using Kinect on PC and founded the OpenKinect community. Since then the community has grown significantly and we’ve seen some amazing tech demos and applications using the libfreenect, the open source Kinect for PC drivers developed by OpenKinect community members.

A few weeks ago, I was contacted by PrimeSense co-founder Tamir Berliner, who wanted to discuss their plans and ways PrimeSense could engage with the community. (Microsoft licensed PrimeSense’s technology and based Kinect on the PrimeSense reference design.) After seeing the dozens of OpenKinect videos and the community enthusiasm, PrimeSense decided to move up their plans and yesterday announced the OpenNI initiative and have open sourced their middleware for natural interaction and drivers for their depth cameras.

Tamir graciously agreed to let me interview him for my blog.

Joshua Blake: Thanks for agreeing to answer my questions. First, could you please give us a brief history of PrimeSense and its role in the natural interaction space?

Tamir Berliner: Thanks for having me :)

PrimeSense started from the founders’ understanding that we are bored with standard gameplay. We felt that computer games didn’t innovate any more but rather mainly used a different story to differentiate. We felt it was time for a change, and understanding where natural interaction is headed to we decided to take the first step towards that vision in the games domain, with a 3D camera that was built for that purpose and at the right price point

Josh: What is your role within PrimeSense?

Tamir: It’s been changing all along in the past 5 years, the main idea is to do whatever it takes to make sure the vision is being fulfilled and that we keep on striving towards Natural Interaction. In the past 5 years I’ve been the head of an R&D team, the first software product manager, I created our partners program and now I’m running this amazing effort of contributing our best and latest to the developers community so you guys can do what we didn’t even start to imagine

Josh: I know that any technology takes a long time to mature. For example, Microsoft Research’s Bill Buxton has doing multi-touch research since before the Apple Macintosh came out in 1984, and there were others before him. Multi-touch has only become mainstream in the last few years. How long has research been going on in the motion tracking and pose estimation space?

Tamir: Over 20 years if you count the first time image processing was done in order to extract a human skeleton from an image, or the usage of two cameras to reconstruct a 3D image.

Josh: How long has PrimeSense been involved in this field? What have you guys been up to that has prepared for today when are seeing mass adoption of motion tracking and natural user interfaces in the home?

Tamir: We started ‘running’ in Sept 2005. It took us 5 years to get the first product that uses our technology to hit the shelves. From day one, our goal was computer games. We actually had first trials of skeleton running in early 2006 which helped us both sell the technology and explain what’s possible, and set up all the different parameters so that they will fit best to the usage in the living room domain and even took some tradeoffs to tailor make it to computer gaming.

Josh: I have heard several people make the comment that what can be done with depth sensors has been done before with single RGB cameras and computer vision techniques. Many of those approaches require controlled environments and are not robust enough to use in any random living room, for example. How do depth sensors and PrimeSense technology change the game as far as the interfaces and interactions that are possible now? How does hardware cost play in to the equation?

Tamir: The main issue that PrimeSense has solved is a robust, real time, low-cost depth-map generating device that can be configured to create a skeleton on a low end PC. Our middleware takes less than 1 ATOM CPU and provides a great skeleton model for two people in real time. It also allows you to control a fully fledged media center from 15ft away with mere hand gestures that we keep on making sure passes user testing so everyone can use those at home

Josh: We touched on pose estimation before. When thinking about human-computer interaction, what can we do with a skeleton model of humans in a real space that we cannot do with just raw depth data or point clouds?

Tamir: The raw data is great for testing and rapid prototyping of basic interaction. The main issue is that it usually requires the user to be too aware of the algorithms – take for example the virtual touch screens that we’ve seen - you have to know exactly where to stand and how far your hand needs to reach. These cannot work in the living room or in standard home environments as you will just walk through the virtual screen, interacting with the TV without wanting to. Also imagine that you need to get up every time you want to flip channels or change the volume?. Another example are all the different ‘trainers’. Whether it’s titles like Dance Central or Your Shape, those cannot be made with just raw data, you have to know where the user is and what exactly he/she is doing

Josh: You have described PrimeSense technology as enabling Natural Interaction. What does the phrase mean and where is it heading?

Tamir: The goal of Natural Interaction from my point of view would be to be able to take any person (even those who have never used a technological system) and have them be able to use it without any training. I think that Kinect and iPhone are both a great step towards that direction, but we’ve got a lot to do both on the hardware side and on the software side. Merging the interfaces between all the different devices that you have and making them “know you” is a key step and require tons of work. I believe this revolution will take at least 10 to 15 years till we can look back and say ‘what were we thinking with those mice and keyboards?!’

Josh: Is the future of natural interaction and 3D cameras limited to games and entertainment or will it see adoption in other fields as well?

Tamir: Natural Interaction is already coming from all different directions at us, you can see it with Wii, iPhone, and even in public restrooms where the water tap understands you want to wash your hands and starts the water flow automatically. The more we see Natural Interaction being introduced into the market, the more you will see everyone using technology. The keyboard and mouse seem ‘normal’ today, but if you come to think of it, they are as natural to a human as leash is to a dog.

3D cameras are of course one of the key enablers of Natural Interaction, just as speech and touch are, and as such I foresee this technology being incorporated in many more technological devices in different forms and as different input paradigms. We can see great examples from the community with regards to what’s possible and I believe this is merely the beginning. Another very important area where 3D cameras are coming into play is providing technology with “eyes”. Just like providing phones with accelerometers and GPS was a big step, so will this be and some of the main targets will be technologies that need to move and navigate in the real world

Josh: We’re already seeing a lot of early videos of these types of applications come out of the OpenKinect community. What was the reaction inside PrimeSense like to the really creative and diverse ways that people are using the Kinect depth sensor? I heard it caught you by surprise.

Tamir: The reaction was ‘Wow’ after 5 years of meeting with people and trying to explain what is possible, you finally see people who not only do things you’ve thought of, but do much more and create amazing applications that we didn’t even consider! It’s mind blowing to see that creativity is far greater when it’s in the hands of a community, and it’s one of the main reasons for why PrimeSense has decided to provide the community with the work that it has invested a lot into. I don’t know if the people in the community will like what we’ve got to offer, but what led us to make the decision was that we’ve seen people starting to tackle issues that we’ve been working on for years now so it would almost be a ‘sin’ not to contribute from our experience.

Josh: Clearly there is a quickly growing community that is extremely enthusiastic about the possibilities unlocked by this type of technology. (The OpenKinect mailing list has grown to over 1250 members join in just four weeks.) Even so, Natural Interaction is still new and we are still at the beginning of the Natural User Interface revolution. What are the biggest challenges that we will see to adoption of Natural Interaction across the industry?

Tamir: I believe that the value of Natural Interaction is great, but only when it’s implemented properly. Only a small set of companies have been able to really create devices that both are worth the money and have provided a real Natural Interaction leap. As I mentioned I see companies like Microsoft with the Kinect, Apple with the iPhone and Nintendo with the Wii as being the first and I can only hope (as a consumer) to see everyone in the world understanding the this is merely the beginning. The number one issue for all of the devices is to simulate the interfaces that we already know from the ‘real world’. Wii did it with motion, iPhone did it with touch, and Kinect has taken it to a whole new level of full body integration into games and media control with the great adder of speech detection.

Josh: This leads us to the OpenNI initiative. Tell us about Open Natural Interaction, who is involved, and the role you see it playing.

Tamir: I believe OpenNI to be a major key player in Natural Interaction. As I’ve mentioned, Natural Interaction consists of hardware devices that we’ll see coming from a variety of CE [Consumer Electronics] vendors, and software modules that will be developed by many different software providers. Having all of those comply with one standard ensure that any application that is developed can deploy on a large number of systems with an install base that is not fragmented. As the first step we’ve worked with some of our partners to create OpenNI and at this point we’ve decided to launch it a bit earlier than expected due to the large amount of developers already working. Our initial goal was to launch the OpenNI standard a few months from now. Many industry leaders are interested and when they are ready they will join us in the management of the OpenNI consortium.

Josh: Along with OpenNI, PrimeSense released a whole suite of software, including some open source. What can the public do with this software, and how would you like to see the community engage with OpenNI? Are you open to people contributing to the software projects or extending it with new capabilities?

Tamir: The software is divided into two main parts, the first being the OpenNI standard which is mainly the APIs to devices and image processing algorithms. Extending OpenNI and submitting changes is welcomed. In the next versions, the OpenNI consortium will publish what will be included prior to closing up the version so everyone can suggest introducing their changes to the official release.

The second part is NITE which is PrimeSense’s middleware for its SoC (System on Chip) and includes UI framework for controlling the TV from 15ft away without the need to start searching for the remote. This framework takes a bit time to learn to use but provides great power to the developers, as it provides them with exactly what they need – events that are generated according to the user’s motions but that are still tailor made for their applications.

The other part NITE includes is the skeleton image processing module that enables the application developer to receive the joints of the user that initiated interaction (the way to start interaction is by standing with your hand to the sides in a ‘freeze’ position). After that you have a skeleton that sticks to you. In order to contribute even further we’ve also provided the “scene analyzer” which segments the users from the background so you actually can enumerate the depth map and get the number of users. You can see both the Skeleton and Scene Analyzer in these videos on Youtube

Josh: Those videos are awesome. I’m just imagining what I can do with those capabilities. Is there anything else you’d like to share or tell us about?

Tamir: Sure – I believe that till today the devices we’ve been using, made us learn greatly lot about them before we could use them and gain their value. I’m pretty sure everyone who is reading this has got at least 3 remotes sitting on his living room table, and at least once a week needs to help someone use their computer/media center/phone/etc. It’s time for that to change and it’s up to us, the technologists to make this revolution happen, it’s time for the devices to take the step of understanding what we want and making sure we get that, even without asking if it’s a trivial task as opening a door when we approach, closing the lights when we leave the room, even making sure we have hot water to shower with when we return from work or wake up in the morning, depends on what we normally do.

Having said that, we need to remember that now localization is not only changing the written language, it’s becoming the gestures that people are used to, the accent in speech detection. Even the difference between when a user wants to control and when they don’t is not so trivial anymore as it’s not just “press the button”… it’s up to us to make sure that the users enjoy a new set of very compelling experiences that don’t break. It’s a new huge growing market, and I’d suggest anyone with technological skills to jump on and enjoy the creation of this new era.

Josh: I’m very excited about all of this. Thanks again for answering these questions and I’m looking forward to working with you to make some amazing natural user interfaces and natural interactions.

InfoStrat.VE and the Bing Maps 3D end-of-life

2010-11-04T12:49:00.001-04:00

On Tuesday, the Bing team announced that the Bing Maps 3D control (the ActiveX plug-in, not Silverlight) is being end-of-lifed. The 3D control will be removed from the website in a few weeks, but the data feeds that support it will remain live until December 2011. I've gotten some questions on twitter and the InfoStrat.VE forum about what this announcement means for InfoStrat.VE so I wanted to post an update about this.

Based upon this announcement, the current version of InfoStrat.VE will still work until December 2011. After that date, any applications using the current version will essentially stop working. We are pursuing several options that will replace as much of the functionality of the Bing Maps 3D backend to InfoStrat.VE as possible.

I'm really happy to see the support from people on twitter and comments on the Bing blog for InfoStrat.VE. Unfortunately the decision to end-of-life Bing Maps 3D was made much earlier this year. We had been working closely with some of the team members so they did give us a heads up back then, but we couldn't discuss it publicly of course.

At that time we provided a lot of feedback and as much push-back as we could, including several of the ideas that people posted in the comments on the Bing blog. This is also why you haven't seen much new lately for InfoStrat.VE. Of course, Microsoft and this project has always said using this control is unsupported with no guarantees.

We are disappointed that Microsoft is bowing out of the 3D mapping market, which is important to commercial and government customers. They've chosen to focus on 2D mapping, which primarily targets consumers. The 2D solution does have Birds Eye and some other cool features, though.

That said, this is not the end of mapping on Surface. InfoStrat.VE should still work in your existing projects until December 2011. We're hoping to have news of a transition path well before then in order to give time to switch over current and future projects. This might end up using 2D only, or possibly extending the backend to support other 2D or 3D mapping providers. We don't have anything solid at this point but we will definitely post some information when we can.

If you have any preferences about where you'd like to see InfoStrat.VE go, please chime in on the comments or send me an email: joshblake at gmail.com.

Thanks for your support!

Speaking about NUI at CapArea.NET Silverlight SIG

2010-10-19T01:48:00.000-04:00

Tonight (10/19/2010) I'll be speaking at the CapArea.NET Silverlight SIG about NUI development with WPF and Silverlight. It is at 7pm at the Navy League Building at 2300 Wilson Blvd, Arlington VA 22201 and is easily accessible via the Court House Metro Station.

Abstract
The Natural User Interface (NUI) is the next revolution of human-computer interaction. Microsoft Surface has shown the potential of multi-touch NUIs to uniquely engage users, and multi-touch tablets and displays are becoming more and more common. In this presentation, Joshua will cover fundamental NUI and multi-touch concepts and discuss multi-touch development options with WPF 4 and Silverlight. You will learn about raw touch input and the manipulation API and see the differences between WPF and Silverlight multi-touch development. If you are interested in creating rich, engaging multi-touch interfaces for the web or client, then you won't want to miss this presentation!

If you are in the area, come by and say hi afterwards. See you there!

Recent NUI videos and links

2010-10-19T01:36:00.002-04:00

In the last week or two there have been several interesting new links about NUI research and innovations. I thought I'd share them all together in hopes that they are interesting to you as well.

PhoneTouch
http://www.technologyreview.com/computing/26458/
Researchers at University of Lancaster are coordinating cell phone accelerator data timing with touch data on a touch table to uniquely identify users. When a user uses the phone to touch surface they can create user specific reactions, such as pulling data from that phone, as opposed to another person's phone. Of course, the phone needs to run some software and already be paired via bluetooth or Wifi but this is an interesting idea about sensor fusion.

Video:

LightSpace
http://news.cnet.com/8301-13860_3-20018451-56.html
Ina Fried has a good write up about a new Microsoft Research project by Andy Wilson, one of the founders of the Surface Computing group at Microsoft, and his team. They've taken Surface Computing a step further and using ceiling-mounted projectors and depth cameras (perhaps borrowing a bit from Kinect technology), have turned an entire room into a Surface. A standard wood table-top is now touch-sensitive Surface.

The interesting thing with this is that it can detect presence and coordinate interactions between the interactive zones. The video shows them picking up a video from the table, carrying a representative dot across the room and putting it onto a wall to play the video. They also show a menu system created in a column of air where you select an item by hovering your hand in the right place. The menu items are projected onto your hand.

The interface and interaction technique are still clunky and unrefined, but this is still a research project and they're exploring new capabilities. I think about these types of future interactions a lot. We're going to be seeing more and more of this integrated interactions and reacting to presence as well as touch in the future.

Video:

Manual Deskterity
http://www.zdnet.com/blog/microsoft/microsoft-courier-concepts-move-foward-on-the-surface/7570
Mary Jo Foley updates us about another Microsoft Research project, Manual Deskterity. A video of Manual Deskterity was released earlier this year and now the team has published a paper with more details of their findings and user research. The project focuses on multi-modal and bi-manual (two-handed) interactions.

They use an IR light pen and Microsoft Surface but the techniques are applicable to any type of pen and touch system. (Mary points out a connection to the leaked Courier videos last year and the InkSeine project.) Note that most commercial tablet PCs available today that support stylus and touch only allow one at a time. You'll need special drivers to do both at once.

Video:

Bill Buxton, "NUI - What's in a name?"
http://channel9.msdn.com/posts/TechTalk-NUI-Whats-in-a-Name
Bill Buxton recently gave another presentation about NUI, this time at the Microsoft Development Center in Copenhagen. The presentation is about 60 minutes with another 30 minute Q&A. It combines some of his previous speeches this year with a few new things. I'd encourage you to plan some time to watch this, even if you've already seen everything else he's ever presented. He has a great way of breaking down difficult-to-understand topics and making you see things in a new way.

Video:

I’m a Microsoft Surface MVP!

2010-10-04T01:16:00.001-04:00

On Friday, October 1st, I got an email that I was hoping to get for a while. It started:

Dear Joshua Blake,

Congratulations! We are pleased to present you with the 2010 Microsoft® MVP Award! This award is given to exceptional technical community leaders who actively share their high quality, real world expertise with others. We appreciate your outstanding contributions in Surface technical communities during the past year.

The Microsoft MVP Award provides us the unique opportunity to celebrate and honor your significant contributions and say "Thank you for your technical leadership."

The email went on to tell me how to log in to the MVP site and that I should be expecting an award package.

The Microsoft MVP is a year-long award with new and returning MVPs announced every three months. The Surface MVP category was created just before the last cycle of MVP awards that went out July 1st. Dr. Neil from nsquared in Australia holds the honor of being the first Surface MVP since he made it in during the last cycle.

This cycle adds Dennis Vroegop in Netherlands, Rick Barraza from Cynergy in San Diego, CA, and me from InfoStrat in DC, bringing the number of Surface MVPs to four in the world. You can see our MVP profiles here. I’m very excited, proud, and humbled to be recognized among the best of the Surface community.

I have to say thanks for all of my readers and supporters. I plan to continue to contribute to the community and encourage others to do so as well. There are many new and exciting things coming up, both from me and InfoStrat as well as within the Surface community. It’s going to be a great year!

NUI session at Heartland Developers Conference

2010-09-08T19:18:00.000-04:00

I'm here at the Heartland Developers Conference in Omaha, NE. If any of my readers are here, please look me up and we'll chat! You can email me or send me a message on twitter: @joshblake.

I'm presenting a session tomorrow (9/9/2010) at 1:15pm right after lunch:
Deconstructing the Natural User Interface: UX design in the NUI revolution

I'll be using NaturalShow again, and there may just be an announcement. You'll have to come to the session to find out! (Or read my blog when I post a follow-up.)

Multitouch on Windows book update

2010-08-27T15:57:00.000-04:00

Here is a quick update about the progress of my book, Multitouch on Windows. The MEAP (Manning Early Access Program -- pre-order access to the eBook) now includes chapters 1-2 and 4-7. You can get access the MEAP by pre-ordering either the eBook or print+eBook. (Disclosure: that link is affiliate and sends me a few extra dollars if you buy through it.) You can also download the source code samples from the book page at that link.

I'm currently working on chapter 8 about the WPF manipulations API. This is a key chapter so I'm giving it some extra love and care. I'm also balancing out work and family life. It is a challenge sometimes but this stuff motivates me so I push through. I'm also motivated by the comments, feedback, and successes of my readers. If you're doing something cool with multi-touch let me know!

Since I'm closing in on half-way through the book, I posted some survey questions for readers on the Manning forums. I basically need to know how important Silverlight and Surface content is to readers so I know how much effort to put into various areas. I'm also trying to build the case for the book to be printed in color (the eBook is already in color) so the more responses the better case I can present to Manning. Please take a few minutes and answer these four questions, even if you haven't read the book yet:

Multitouch on Windows reader survey

If you don't want to post the answers on that forum or don't want to make an account, you can always just email me directly: joshblake at gmail dotcom. Thank you to those of you who have already answered the survey.

Microsoft Surface and Magical Object Interaction

2010-08-26T23:05:00.000-04:00

Work at InfoStrat has been really exciting recently. We are getting more and more multi-touch projects for both Surface and Windows 7. We've created some really interesting interfaces and applications and soon we'll be able to share more.

For now, I can share this video. We built a suite of Microsoft Surface applications for a well-known museum. These are designed to be used by young children. During user testing we found that it works great for them and also does pretty well at bringing out the inner child from people of all ages. I'll post more details on the findings of our user testing in a future post.

The theme of the applications was "light" and we designed them to really take advantage of the unique capabilities of Surface: bringing people together socially and physical object interaction.

Let me know what you think!

By the way, if you're interested in working for InfoStrat and you know WPF, we're hiring! Take a look at our job posting for WPF experts as well as Microsoft CRM and Microsoft SharePoint experts.

Sharing binaries between Surface and Windows 7

2010-07-16T02:04:00.002-04:00

This post is part of a series about Blake.NUI, my open-source library for multi-touch natural user interface development with WPF 4 and Surface. In this post I discuss how to set up your projects to share 99% of your binaries between Surface and Windows 7. See my Blake.NUI announcement post for an overview of the project as well as a discount code for my book Multitouch on Windows, good until July 23.

In my last post, I showed how to use the SurfaceTouchDevice class to let WPF 4 Touch applications work on Microsoft Surface. In this post, I will discuss how to set up your project to allow you to reuse almost your entire WPF 4 code-base between a Surface SDK application and Win7 Surface Toolkit application. This approach will make your code binary compatible between the two platforms.

Note that as I discuss the technique, I am assuming that you are setting up your projects on a Windows 7 multi-touch machine with Visual Studio 2010 and both Surface SDK SP1 Workstation Edition and Surface Toolkit beta installed. Once the projects are setup, you can compile the solution on a Surface device as long as you create a solution configuration that does not build the projects that reference Surface Toolkit.

1. Project Structure

If you want to target both Win7 and Surface with the same code-base, you'll need to keep the Surface SDK-specific code separate from your reusable code. There are several possible approaches, but here is the one I have been using. Figure 1 shows the architecture of the solution.

Figure 1 Most of the controls and code is located in a common project that is consumed by a Surface project and a Win7 project.

I recommend creating a shell project for Surface and a shell project for Win7. Each of these projects has only a window. In the Surface project, add SurfaceTouchDevice.RegisterEvents(this) to the SurfaceWindow constructor. (I also add MouseTouchDevice.RegisterEvents(this) to both Surface and Win7 projects.) Then in a separate common library I put all of my MVVM code and added the same view user control to the window in each of the shell projects.

Note that the Surface project references the Surface SDK (v1.0) and the Win7 project references Surface Toolkit (v1.5). You will need to check the properties on the assembly references to ensure they are the correct version for each application. The Surface project probably is referencing v1.5 of Microsoft.Surface.Presentation.dll and Microsoft.Surface.Presentation.Generic.dll. Simply remove those reference and in the Add Reference dialog browse and add references to the 1.0 version of the assemblies.

Surface Toolkit contains a subset of controls from the full Surface SDK. This means that you can reference either version 1.0 or 1.5 of Microsoft.Surface.Presentation.dll in your common project and use any of the common controls that exist in both assemblies, such as SurfaceButton and ScatterView. As long as you stick to the subset of common controls, the same binary will work on both Surface and Win7. I would recommend using v1.0 in the common project so you can still compile the project on your Surface device without Surface Toolkit.

2. Loading the correct assemblies

Assuming that you do reference Microsoft.Surface.Presentation.dll v1.0 in the common project and Microsoft.Surface.Presentation.dll v1.5 in the Win7 project, you need to do one more thing to ensure that the application will load correctly. If you do not, then depending upon the order in which components load, you may get a runtime exception like the one shown in figure 2.

Figure 2. If you mix the assembly references, you may get an IOException: Cannot locate resource 'controls/touchvisualizations/touchvisualizationsresources/trailtop.png'. This exception occured on the first touch after the Win7 application loaded.

The solution to this is to add a versioning policy to redirect the v1.0 assembly to v1.5. This blog has a good overview of versioning policies. In the Win7 project, add this to the app.config within the configuration section:

<runtime>

  <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">

    <dependentAssembly>

      <assemblyIdentity name="Microsoft.Surface.Presentation"

          publicKeyToken="31BF3856AD364E35" culture="neutral"/>

      <bindingRedirect oldVersion="0.0.0.0-1.5.0.0"

          newVersion="1.5.0.0"/>

    </dependentAssembly>

  </assemblyBinding>

</runtime>

Just to be safe, you may want to also add the same thing to the Surface project's app.config, except change the newVersion to 1.0.0.0. This ensures that each project will always load the correct version of the assembly, provided it is installed.

I have committed a few updates to the Blake.NUI project, including TestApplication.Common, TestApplication.Surface, and TestApplication.Win7. These three projects illustrate the ideas discussed in this blog. Download the latest code from the Source Code tab on the CodePlex site.

3. Summary

We have learned how to structure a WPF 4 solution to target both Win7 and Surface while reusing almost all of our code in a common library. We also saw how to configure the projects to ensure they load the correct version of the Surface assemblies at runtime.

So far, we know that we can only reference the Surface controls that exist in both Surface SDK and Surface Toolkit. In the next post, I will discuss some techniques that can enable you to use the Surface SDK-specific controls such as TagVisualizer and ElementMenu while maximizing the code reuse.

How to write WPF 4 Surface applications with Blake.NUI

2010-07-14T00:01:00.003-04:00

This post is part of a series about Blake.NUI, my open-source library for multi-touch natural user interface development with WPF 4 and Surface. In this post I discuss the origin of SurfaceTouchDevice and MouseTouchDevice and how to use them in your application. See my Blake.NUI announcement post for details as well as a discount code for my book Multitouch on Windows, good until July 23.

Between InfoStrat projects and writing my book, I've been neck-deep developing WPF 4 Touch and Surface applications for months. If you have used them both then you'll know that the WPF 4 Touch API is a little bit easier to develop for than the Surface SDK it is based upon. This is because WPF 4 touch and manipulation features are integrated directly into the framework. There are also a few other subtle differences, like not needing a separate ManipulationProcessor and InertiaProcessor.

On my current project, I needed to create a Surface and a Win7 version of the same application. I decided that I wanted to use .NET 4 on Surface. This would let me take advantage of the new animation easing functions in .NET 4 and it would let me reuse more code between both versions of the application. First problem: can I install Visual Studio 2010 and .NET 4 on Surface? I tweeted @Surface and Eric Havir replied. Figure 1 shows this twitter conversation.

Figure 1. A twitter conversation between me and Eric Havir from @Surface where he confirms .NET 4 would probably install but could not be used for Surface apps.

That is pretty much what I expected. It's been known since PDC09 that the roadmap for the future of Surface includes the WPF 4 Touch API on Windows 7, as shown in figure 2. This sounds like a great unified platform, but it doesn't help me now. The currently available Surface SDK was designed for .NET 3.5 SP1 and uses a custom Surface Input mechanism that doesn't feed into the WPF 4 touch system.

Figure 2. A slide from the PDC09 Multi-touch on Microsoft Surface and Windows 7 session.

I went ahead and installed Visual Studio 2010 on my Surface anyway because I wanted to use the easing functions. It turns out that Surface applications compile just fine under .NET 4 and even the SDK works, both in the emulator and on a Surface device. I figured I would just create some common controls and branch most of my project to target either the WPF 4 Touch with Surface Toolkit or the Surface SDK.

But wait! WPF 4 uses TouchDevices. TouchDevices are extensible. Anson Tsao, a Principal Program Manager who was responsible for the WPF 4 Touch API, wrote about creating custom touch devices in January.

So I'm thinking maybe if I create a custom touch device that listens to the Surface Contact events, I could process those events into WPF 4 TouchDevices and not even write against the Surface SDK.

Well once I had the idea in my head, I had to do it. I created a class, SurfaceTouchDevice, that derives from TouchDevice and promotes Surface SDK Contact events to WPF 4 TouchDevice events. This means that if you install .NET 4 on your Surface and write against the WPF 4 Touch API, It Just Works™.

Here's the best part. All you need to do to enable this dark magic is add this line of code to the constructor of your SurfaceWindow:

SurfaceTouchDevice.RegisterEvents(this);

That's it. Now the same code that works with WPF 4 Touch will also work on Surface. This also means you can develop with Visual Studio 2010. The SurfaceTouchDevice class will not work on a Win7 machine, though, since the Contacts class is not present in the Surface Toolkit.

In most cases you won't need to do anything else. Surface SDK controls, such as TagVisualizer, will react to and handle Contact events. Unhandled events will be promoted to WPF 4 Touch events by the SurfaceTouchDevice class, and then your code will react to the WPF 4 Touch events.

If you do want to check an actual Surface property, such as whether the TouchDevice represents a tag or a blob, you can get access to the original Contact by casting the TouchDevice to a SurfaceTouchDevice:

   1:  private void TouchDown(object sender, TouchEventArgs e)

   2:  {

   3:      bool isFinger = true;

4:

   5:      SurfaceTouchDevice device = e.TouchDevice as SurfaceTouchDevice;

   6:      if (device != null)

   7:      {

   8:          isFinger = device.Contact.IsFingerRecognized;

   9:      }

  10:      //Do something

  11:  }

You can also filter out touches from your manipulations by iterating through e.Manipulations, casting each IManipulator to SurfaceTouchDevice, and calling Manipulation.RemoveManipulator for undesired IManipulators. I will probably add this feature to SurfaceTouchDevice in an update to Blake.NUI.

I also created a MouseTouchDevice class that operates similarly except promotes mouse input to touch input. This is helpful so I do not need to create MouseDown/Move/Up events when I'm creating a custom touch control with TouchDown/Move/Up. It also means the mouse can participate in the manipulation processor.

Both SurfaceTouchDevice and MouseTouchDevice are available, free and open-source, in Blake.NUI, my library of helpful multi-touch NUI code for WPF 4 and Surface.

Now that SurfaceTouchDevice is available, I cannot think of a reason why you would not want to install Visual Studio 2010 on your Surface and write all applications with WPF 4. In my next post, I'm going to talk about how to set up your application architecture so you can share 99% of your code with no changes or branches for Microsoft Surface and Win7.

Announcing Blake.NUI - a multi-touch NUI library

2010-07-12T23:14:00.001-04:00

Almost all my free time and creative efforts recently have been focused on my book, but I've been able to put together something really cool that I think will be useful for anyone doing multi-touch natural user interface work with WPF. Actually, it's a couple really cool things packaged into one.

I'm happy to announce Blake.NUI, an open-source library for multi-touch NUI development. Blake.NUI is available under the MS-PL license at CodePlex:

http://blakenui.codeplex.com

It is a collection of the most useful reusable code that I have created for my own projects and for my book's code samples. It currently includes code for WPF 4 and Surface, although in the future I may add Silverlight code as well.

The library is still in undergoing changes. I may add or rearrange things, but I will document changes from version to version. Here are some of the things currently included in Blake.NUI:

Controls

ZoomCanvas - an infinite canvas control that is the heart of NaturalShow
CircleText - layout text along the circumference of a circle

Gestures

TapGestureTrigger - a configurable Blend Trigger that fires when the user completes a tap gesture
DoubleTapGestureTrigger - a configurable Blend Trigger that fires when the user completes a double tap gesture

TouchDevices

MouseTouchDevice - promotes mouse input to WPF 4 touch events
SurfaceTouchDevice - promotes Microsoft Surface vision system input to WPF 4 touch events

Common classes and utilities

DisplayMatrix
AnimateUtility and SurfaceAnimateUtility
ScatterMatrixHelper
MathUtility
VisualUtility

NaturalShow

Common models, views, factories, and other classes

The full NaturalShow application is not yet included, but many of the core classes are. I'm updating NaturalShow to use Blake.NUI rather than its older, internal versions of the same code.

I'm particularly excited about SurfaceTouchDevice, since with one line of code it enables you to write multi-touch applications against the WPF 4 Touch API, including the integrated touch events and manipulations, and run it on your existing Microsoft Surface devices without modification. This also means you can, with a simple project structure, target both Win7 Touch and Surface v1 with 99% of the same code base! You do have to install .NET 4 and/or Visual Studio 2010 on your Surface.

I will be celebrating the launch of Blake.NUI all this wee by posting a series of blog posts about how to use the library. In addition, you can buy the ebook edition of Multitouch on Windows until July 23rd for 35% off through this link using coupon code blake35bl. (Disclosure: if you buy after clicking this affiliate link it gives me a few extra dollars from the sale.)

I am asking as a courtesy that if you find Blake.NUI useful, particularly for commercial projects, that you buy a copy of my book, Multitouch on Windows. The coupon code provided should help.

Please let me know what you think about Blake.NUI and whether you think you'll use it in your applications.

NaturalShow and new interviews

2010-06-05T13:45:00.003-04:00

Coincidentally, two different shows featuring me talking about NUI were posted yesterday, even though the interviews were several weeks apart.

Brian Peek interviewed for for his Channel9 Coding4Fun show. We talked about NUI and NaturalShow and I showed some of the actual code and how it works.
http://channel9.msdn.com/shows/Coding4FunTV/The-Coding4Fun-Show-Natural-User-Interfaces-with-Josh-Blake/

I also spoke at Richmond Code Camp about NUI last mont and while I was there I was included in a segment of the The Community Megaphone Podcast hosted by Dane Morgridge and Andrew Duthie. I'm the second to last interview.
http://www.communitymegaphonepodcast.com/

I also found out recently that my session submission for the Heartland Development Conference was accepted! HDC is September 8-10 in Omaha, NE.
Deconstructing the Natural User Interface: UX design in the NUI revolution

Finally, I want to make the announcement on my blog that I will be making NaturalShow open source and it will be made available along with my book. I'm going to integrate it with some of the topics in some of the chapters. MEAP subscribers will get the first access to it, although those chapters are not complete yet. I'll be sure to announce when they're included.

[Update: fixed links, thanks Ben!]

Why won't WPF controls work with touch manipulations?

2010-04-06T03:29:00.003-04:00

I recently tweeted asking for some scenarios using WPF Touch, and I got an email from Josh Santangelo (@endquote) with few interesting touch challenges. I created some sample code and sent him back some solutions to his challenges, but I figured they would be useful for the community.

In this blog, I'll answer the question: "Why won't WPF controls work with manipulations?" (This is my own phrasing, not Josh Sanangelo's.)

Problem:
You have a container with some manipulations (perhaps like a ScatterView) and this container has some standard WPF controls like buttons or checkboxes. You can use touch to manipulate the container, and you can use the mouse to click the controls, but touch mysteriously doesn't affect the controls. (The mouse also can't manipulate the container, but that is another post.)

Reason:
When you touch the screen, WPF generates touch events, such as TouchDown, TouchMove, and TouchUp. These are all routed events, which means that first the PreviewTouchDown event is fired on the root of the visual tree, then the next element down the hierarchy, then the next, all the way down the source element that the touch event occured over, as long as the event is not handled along the way.

Once it reaches the source element, the TouchDown event is fired starting from the source element and proceeding up the visual tree to the root. At any point if any of the visual elements set e.IsHandled = true, then the event propagation stops.

On the other hand, if the event propagation reaches all the way up to the root, then the touch event is promoted to a mouse event. At this point, PreviewMouseDown and MouseDown is fired down and up the visual tree.

In our case, when you use touch and there is no manipulation, the touch events return unhandled, then the mouse events are fired and the button handled that and calls your Click event handler. The WPF controls such as button only listen for mouse events (exception: ScrollViewer when PanningMode is set.) This process is illustrated in figure 1.

Figure 1. (Click for larger size.) The touch event flow with no manipulations. The touch events are unhandled, so WPF promotes the event to the mouse equivalent. The button is listening for mouse events and handles that, then calls your Click event handler.

When you do have manipulations enabled in the visual tree, something different happens, as illustrated in figure 2.

Figure 2. (Click for larger size.) When the border has IsManipulationEnabled = true, the manipulation processor handles the TouchDown event and captures the touch device. All future events go directly to the Border, and the ManipulationProcessor handles the rest.

In the case where the Border has a manipulation, the touch event flow is never promoted to mouse events, so the button doesn't even have any idea what is going on. Button didn't get the memo.

Solution:
Of course, it doesn't have to be this way. Sometime soon, the Surface Toolkit for Windows Touch will be released and you'll be able to use the SurfaceButton and the other controls which are designed for touch and handle the touch events. Figure 3 shows a list of the WPF controls the Surface toolkit optimizes for touch.

Figure 3. (Click for larger size.) Surface Toolkit for Windows Touch offers touch-optimized controls that can replace most of the common WPF controls. This is only a partial list of what the Surface Toolkit offers.

But that doesn't help you now. Suppose you have this XAML inside of a container with manipulations:

   1:  <TextBox Name="txtCounter"

   2:           Text="0"

   3:           Margin="10"

   4:           HorizontalAlignment="Center" />

   5:  <Button Content="Native Button Won't work"

   6:          Margin="10"

   7:          Height="40"

   8:          Click="button_Click" />

and this code in the code behind:

   1:  private void button_Click(object sender, RoutedEventArgs e)

   2:  {

   3:      IncrementCounter();

   4:  }

5:

   6:  private void IncrementCounter()

   7:  {

   8:      int number = int.Parse(txtCounter.Text) + 1;

   9:      txtCounter.Text = number.ToString();

  10:  }

This button will not work and the event flow will look like figure 2. Instead, update the button XAML to this:

   1:  <Button Content="Will work with TouchDown/Up"

   2:          Margin="10"

   3:          Height="40"

   4:          Click="button_Click"

   5:          TouchDown="button_TouchDown"

   6:          TouchUp="button_TouchUp" />

and add these methods in the code behind:

   1:  private void button_TouchDown(object sender, TouchEventArgs e)

   2:  {

   3:      FrameworkElement button = sender as FrameworkElement;

   4:      if (button == null)

   5:          return;

6:

   7:      button.CaptureTouch(e.TouchDevice);

8:

   9:      e.Handled = true;

  10:  }

11:

  12:  private void button_TouchUp(object sender, TouchEventArgs e)

  13:  {

  14:      FrameworkElement button = sender as FrameworkElement;

  15:      if (button == null)

  16:          return;

17:

  18:      TouchPoint tp = e.GetTouchPoint(button);

  19:      Rect bounds = new Rect(new Point(0, 0), button.RenderSize);

  20:      if (bounds.Contains(tp.Position))

  21:      {

  22:          IncrementCounter();

  23:      }

24:

  25:      button.ReleaseTouchCapture(e.TouchDevice);

26:

  27:      e.Handled = true;

  28:  }

Now your button will work inside of the container with both mouse and touch. A little explanation:

Line 7: We capture the touch so that the TouchUp and other touch events will be sent to this button, even if it occurs somewhere else.
Line 9: We must set this event to handled, otherwise the TouchDown event will continue to bubble up and the border's manipulation processor will steal the capture, depriving the button of a TouchUp event.
Lines 18-20: We check to make sure the TouchPoint is still within the bounds of the button. The user could have touched the button, but changed his or her mind and moved outside of the button to release the touch. This is consistent with the mouse behavior of buttons.
Line 22: We just call the same function that the button_Click() event handler called to get the same effect.
Lines 25-27: We release the capture and handle this event as well. Technically we might be able to get away without these lines in this scenario but we should do it anyway. In a more complicated scenario there might be an unintended side-effect if we do not do this.

You can apply this technique to the other non-touch aware WPF controls if necessary, although it may get a little tedious. I almost wrote a Blend behavior (part of the Blend 3 SDK and usable in more than just Blend) for this, but figured that the Surface Toolkit will be out soon enough anyway.*

(* No I don't have a date for the release of Surface Toolkit. Sorry!)

You can download the source code for this project below:

ButtonInManipulation.zip

Thanks to Ryan Lee for Gesturecons, where I got the hand icon used in the figures.

NUIs reuse existing skills (updated NUI definition)

2010-04-06T00:18:00.002-04:00

Last month I posted an excerpt from my book with a section that contained a definition of natural user interface.

A natural user interface is a user interface designed to use natural human behaviors for interacting directly with content.

It sparked some great conversation both in the comments of that post and some other blogs. Richard Monson-Haefel suggested on his blog that I consider changing the "use natural human behaviors" part to match how I talk about natural in terms of innate abilities and learned skills. Those discussions were derived from some of Bill Buxton's thoughts on NUI.

At the time, I was on the fence about this change. I wrote in a comment that I liked using the simpler vocabulary words since it will be understandable by more non-technical folks, and that I liked the term behavior for a rhetorical reasons since in the beginning of chapter two I had an interesting set of phrases that needed it to say "behavior" to make sense. (I know that isn't a very good reason and actually I since deleted those pages with that particular discussion.) So I was leaning towards my initial definition but I was keeping my mind open for more convincing arguments.

Well as it turned out, I was convinced by Bill Buxton at MIX10. He wasn't talking directly to me about this, but in his keynote he described how a saxaphone input device is a natural interface for someone who has years of saxaphone training, and he proceeded to use his electronic saxaphone (a Yamaha WX7 wind controller) to create flute and electric guitar sounds. Below is a screenshot, but if you want to watch his part of the keynote, skip to mark 96:45 of the MIX10 Day 2 Keynote video.

Bill Buxton uses a Yamaha WX-7 wind controller and his existing saxaphone skills to create music that sounds like a variety of non-saxaphone instruments.

When I heard him describe this, I immediately understood that my existing definition ("use natural human behaviors") excluded advanced NUIs designs for people who were already experts with advanced skills. I had already prepared my NUI presentation for day 3 including the original definition, so directly after the keynote I went and changed it. In my session video you can see I used a revised definition of NUI, and in the next revision of chapter 1 (now updated in the MEAP) I also updated the definition.

Here is the updated book excerpt:

There are several different ways to define the natural user interface. The easiest way to understand the natural user interface is to compare it to other type of interfaces such as the graphical user interface (GUI) and the command line interface (CLI). In order to do that, let's reveal the definition of NUI that I like to use.

A natural user interface is a user interface designed to reuse existing skills for interacting directly with content.
There are three important things that this definition tells us about natural user interfaces.

NUIs are designed

First, this definition tells us that natural user interfaces are designed, which means they require forethought and specific planning efforts in advance. Special care is required to make sure NUI interactions are appropriate for the user, the content, and the context. Nothing about NUIs should be thrown together or assembled haphazardly. We should acknowledge the role that designers have to play in creating NUI style interactions and make sure that the design process is given just as much priority as development.

NUIs reuse existing skills

Second, the phrase "reuse existing skills" helps us focus on how to create interfaces that are natural. Your users are experts in many skills that they have gained just because they are human. They have been practicing for years skills for human-human communication, both verbal and non-verbal, and human-environmental interaction. Computing power and input technology has progressed to a point where we can take advantage of these existing non-computing skills. NUIs do this by letting users interact with computers using intuitive actions such as touching, gesturing, and talking, and presenting interfaces that users can understand primarily through metaphors that draw from real-world experiences.

This is in contrast to GUI, which uses artificial interface elements such as windows, menus, and icons for output and pointing device such as a mouse for input, or the CLI, which is described as having text output and text input using a keyboard.

At first glance, the primary difference between these definitions is the input modality -- keyboard verses mouse verses touch. There is another subtle yet important difference: CLI and GUI are defined explicitly in terms of the input device, while NUI is defined in terms of the interaction style. Any type of interface technology can be used with NUI as long as the style of interaction focuses on reusing existing skills.

NUIs have direct interaction with content

Finally, think again about GUI, which by definition uses windows, menus, and icons as the primary interface elements. In contrast, the phrase "interacting directly with content" tells us that the focus of the interactions is on the content and directly interacting with it. This doesn't mean that the interface cannot have controls such as buttons or checkboxes when necessary. It only means that the controls should be secondary to the content, and direct manipulation of the content should be the primary interaction method.

I decided to say just "reuse existing skills" and not include the innate abilities part to keep it simpler and also because the thing that convinced me to change it was the fact that some NUIs may use advanced skills, so innate abilities may not always play into NUI. The core thing that did make it natural was reusing existing skills.

I do need to thank the commenters on my last blog, Ben and Laurent, and particularly Richard, who had also put himself out there trying to define NUI but still had great comments about my thoughts. Richard pointed me in the right direction and with some time and thought I realized that he was correct. It's you, my readers, who will help me make this book totally awesome.

Please keep the feedback coming and feel free to call me out if you think I could improve something.

MIX10 NUI session sample code

2010-03-28T13:48:00.000-04:00

I promised that I would provide the source code for the TouchGrid and ScatterTouch samples for WPF, Silverlight for web, and Silverlight for Windows Phone 7 from my MIX10 session. I provided a zip file with the sources to the people running the MIX website after my session, but it still hasn't been posted yet. So I'll just post it here on my blog.

http://www.handsonnui.com/files/MIX10_NUI_EX18.zip

Important: I cannot re-distribute the Microsoft Surface Manipulations and Inertia Sample for Microsoft Silverlight files so you will have to download that yourself and add a reference to System.Windows.Input.Manipulations to these projects SL_ScatterTouch and WP7_ScatterTouch. See the readme.txt for more details.

Of course if you want to run the Windows Phone 7 samples you'll need the Windows Phone 7 SDK.

Have fun!

MIX10 NUI Recap

2010-03-24T02:37:00.004-04:00

Well I'm pretty much recovered from MIX10 by now (March 15-17 in Las Vegas). The whole experience was a blast. Here are the highlights for me:

Windows Phone 7 Series - Finally we learn the full scoop of the WP7 development story. Very excited about the Silverlight and XNA focus. I'm also relieved to learn that WP7 flavor of Silverlight has manipulation processors built in. It isn't quite the same as the WPF manipulations or even the Silverlight manipulations sample (no rotation support) but it makes multi-touch development feasible.

NUI MIXup - I organized a get-together for NUI and multi-touch enthusiasts Monday night. Marc Schweigert (@devkeydet) was able to get sponsorship from his team for drinks and appetizers for the group. It was well attended, about 25 or 30 people. We even had Dr. Neil and Bill Buxton (yes, keynote favorite Bill Buxton!) hang out! After we chatted for about an hour in one of the conference spaces, we moved down to the Border Grill where Marc treated us to drinks and light dinner. See some pictures below, thanks to fellow presenter and NUI enthusiast Mario Meir-Huber (@mario_mh):

Bill Buxton, Ben Reierson (@seraph_ben), and Dr. Neil chatting about Windows Phone 7 design.

The whole group of us heading towards dinner. Marc leads the charge on the left, and I'm in the middle with the black shirt chatting with James Chittenden (@okayjames).

Just about the highlight of my whole week, I had the opportunity to chat with Bill for over an hour at dinner along with several others that attended. It was a humbling experience.

NUI Development session - I didn't go to many sessions in person (they're available for download) because I spent a lot of time Monday and Tuesday putting the final touches on my presentation for Wednesday. I prepared a custom multi-touch presentation tool called NaturalShow that takes the place of PowerPoint. Even Tuesday evening I was sweating some last minute changes in the application, trying to maximize the performance. I also have to thank Josh Wall for listening to at least two dry runs and giving me great advice and feedback.

You can watch or download my presentation at the MIX10 site.

The presentation itself went pretty smoothly. The session had great attendance (144+) and the audience seemed engaged. My demos worked as expected, and NaturalShow didn't crash or otherwise behave abnormally. I included a hashtag #MIX10NUI so that twitter inclined people could follow the conversation. One posted this photo mid-presentation:

Taken from about 1/4 of the way back. The smudges around the word "design" are auras of my fingers starting to zoom in.

I have to admit that part of the reason I threw out the #MIX10NUI hash tag was so I could easily go back and read real-time feedback. When I did I was blown away by the things people said. There were two fair complaints (I took a shortcut when I described Occam's Razor and I forgot to zoom in on code, even though I had practiced doing that) but overall everyone said great things. At the risk of seeming self-focused for a moment, here are two of the most superlative comments that really made my day:

My session was also mentioned in several other people's conference recap blog posts: Luke Wroblewski, Cory Plotts, Roland Weigelt, Ben Reierson.

I guess the point here is that MIX10 was very much about the networking and making new acquaintences as well as meeting online friends for the first time. I'm still pumped by getting to meet so many awesome people, including the ones mentioned above, Steve Dawson, Pete Brown, Laurent Bugnion, Sean Getery and his crew, and many others I'm know I'm missing. Many times it was just a challenge to associate someone's real name with their twitter handle.

After the conference itself died down, Josh Wall and I went out to dinner at Hard Rock Cafe, checked out how they were using Microsoft Surface and a custom touch wall, and chatted about how we were going to apply the loads of information and ideas we gathered at MIX10 to improve InfoStrat.

This touch wall at Hard Rock let you explore memorabilia and was very popular.

Josh Wall testing the limits of interaction design at the Hard Rock Surface deployment.

This girl played with a bar-embedded Surface for over an hour.

Thank you so much to all the people I met and those who attended my session. I hope that we will continue our conversations throughout the year, and I can't wait for MIX11!

NUI at MIX10

2010-03-13T14:09:00.000-05:00

I'm headed to Las Vegas tomorrow for the MIX10 conference. I'm excited about a lot of the topics and news that will be discussed. It will be great to meet a lot of the people I've only talked to online and meet new people.

If you'll be at MIX10, I'd love to meet you! Here are two places I'll definitely be:

NUI MIXup, Monday the 15th, 7pm, The Commons
I'm organizing a tweetup at MIX10 (hence, MIXup) for NUI and multi-touch enthusiasts. Come hang out and chat with others about our shared interests. Depending upon how many people come, we'll chat for a bit then head to a bar or restaurant. Marc Schweigert will be buying a round or two.

Developing Natural User Interfaces session, Wednesday the 17th, 10:30am, Lagoon F
This is my session! I'm very very excited about it (and thank you again to those who voted for me in the Open Call). I've created a custom multi-touch presentation application that takes the place of PowerPoint so I can walk the walk as I discuss how to create NUIs and multi-touch applications.

I'm available on twitter @joshblake or by email joshblake at gmail dot com. Send me a note if you'll be coming.

See you there!