Kodierer [Coder]

Kinect and Leap Motion in Love

2013-03-19T18:15:00.000+01:00

Natural User Interfaces are a trend since a few years and the development of new hardware technology for affordable prices helps to push that movement into the consumer market. The Kinect is one example, the soon-to-be-available Leap Motion is another one or the announced MYO device.

I'm one of the lucky people who applied and actually got a Leap Motion developer device before its release. I also have a Kinect for Windows. Both devices use computer vision with various sensors combined with artificial intelligence like machine learning for the object recognition. The Kinect for Windows recognizes skeletons with joints in a distance of 80 - 400 cm or 40 - 300 cm (Near Mode) which means the Kinect is perfect for the mid range distance but not for close PC interaction. The Leap Motion on the other hand is more for the close range and detects hands, fingers and pen-like objects very precisely. That's why I wanted to combine them and build a proof of concept hybrid solution that leverages both Kinect and Leap Motion to get the best of both worlds (close and mid distance).

In the video below I demo my proof of concept which is based on the Kinect for Windows SDK 1.7 and its Interactions and Controls components, plus the Leap Motion developer device and its beta SDK.
By the way, the Kinect for Windows SDK 1.7 is now publicly available and the bits are nothing less than awesome.

In case you are wondering about the infrared interference of the Kinect and Leap Motion. There's some, but I actually didn't notice any serious effect on the recognition capabilities of the Kinect nor the Leap.
Below is a picture of the Leap Motion as seen through the Kinect's infrared stream. The Leap Motion has three light sources but those are not very bright.

Some might say the Leap Motion is a competitor to the Kinect for Windows, I see it more like a nice addition to our developer toolset and the beginning of a story: Kinect ♥ Leap Motion.

Robot Work - Reducing the Windows Phone Localization Friction a bit

2012-12-12T21:15:00.000+01:00

Windows Phone 8 now supports 3 different screen resolutions which is very nice since there are some great high resolution devices out there and the user has the choice and isn't forced into a one fits all model. However there's also a downside for Windows Phone developers since now also 3 times the amount of screenshots have to be submitted to the Store. If an app has localized versions this adds up very quickly. For example, Pictures Lab supports 11 languages and the soon-to-released v5 update for WP8 also all 3 resolutions: 11 x 3 x 8 screenshots = 264! That's a crazy amount of mindless click-through work to take all those screenshots, not to mention the Dev Center submission process.
Unfortunately there is no automated way or better support by the tools, so one has to do this ridiculous amount of manual work. In a future version of the tools I'd really love to see some kind of macro recorder which records certain steps including screenshot actions and can then use the recorded steps to play this for all supported app languages and generate the remaining screenshots automatically.

In this quick post I want to share some practices I used in order to reduce the work at least a bit for the Pictures Lab v5 update:

Only create screenshots for the 720p (720 x 1280) and the WXGA (768 x 1280) resolution. The WVGA (480 x 800) screenshots can be generated from the WXGA images by scaling those down by 1.6. In contrast to upscaling, are the downscaling artifacts hardly visible if a good algorithm is used. I used my favorite image viewer IrfanView and its batch processing feature which can scale down by the longest side, rename by a pattern, convert, etc.
Add some code to easily switch between UI languages in your app so you don't only rely on the system language and need to reboot the emulator to switch languages. An app's UI language can be set globally by using
```
System.Threading.Thread.CurrentThread.CurrentUICulture
```
For example:
```
System.Threading.Thread.CurrentThread.CurrentUICulture = new CultureInfo("de-DE");
```
in the App.xaml.cs Launching event.
Use a macro recorder / scripting to save a bit time with the repetitive tasks of taking screenshots only with different UI languages. I used a little free tool called Mouse Recorder Pro. This tool records your mouse and keyboard actions and can then play it back also with up to 40% faster playback speed. Of course this doesn't work for all apps and only when using the emulators, esp. uncontrollable live content would likely not work, but it worked for my scenario quite well and saved me from creating all 264 screenshots manually. In fact I watched the magic ghost mouse do its job most of the time and worked on other stuff.
A few gotchas when using the Mouse Recorder Pro with the WP emulators:

Don't click on Pivot headers and rather use the swipe gesture to change between Pivot items. The header texts will have different lengths in different languages.
Try to avoid the bounce effect of the ScrollViewer, it would result in random offsets and the Mouse Recorder playback will then have non-matching offsets recorded.
Do a dummy save in the destination folder for the screenshots before each playback so the playback will save the files to the right folder. This will avoid file name collisions which would need manual input.

Of course there's a lot of room for improvements, but maybe the described practices help to reduce the friction that comes with localization at least a bit until a better solution is known.

WACK ARM

2012-11-01T22:06:00.004+01:00

If you are developing Windows 8 Store apps you should really test those on low-powered x86 and ARM devices. The difference to your high-end x64 developer machine can be dramatically, esp. if you use a non-default design.
Testing on ARM is important and Tim Heuer wrote a nice post which shows how to remotely deploy, debug and profile on ARM devices right from Visual Studio 2012. Another nice thing which performs a couple of automated tests is the Windows App Certification Kit (WACK). Unfortunately is the WACK tool for ARM hidden inside the Windows 8 SDK, but here's how to find it:

Download the Windows 8 SDK web setup from here and run it.
Choose to download the SDK files for offline usage.
On the "Select the features you want to download" screen it's enough if you only select the "Windows App Certification Kit" checkbox.
After the download is finished, you will find an "Installers" subfolder in the download target folder.
Copy the "Windows App Certification Kit arm-arm_en-us.msi" from the "Installers" folder to an USB key or SkyDrive.
Open the "Windows App Certification Kit arm-arm_en-us.msi" on your ARM device and install the WACK on it.
Search for "Windows App Cert Kit" on your ARM device using the Search Charm and run the WACK.

Faster! - WriteableBitmapEx for Windows Phone 8 and WinRT Updated

2012-10-30T19:53:00.000+01:00

The Windows Phone 8 SDK is now available and with that WriteableBitmapEx was updated too. The performance of the WinRT XAML version was also improved dramatically and an update is highly recommended.
WriteableBitmapEx is available for 4 platforms: Windows Phone 8 and 7, WPF, Silverlight and Windows Store WinRT .NET XAML.
You can download the binaries here or via the NuGet package. The packages contain the WriteableBitmapEx binaries. As usual all samples and the source code can be found in the repository.

Know Your Users - Windows 8 Store App System Info

2012-10-28T19:57:00.000+01:00

This is just a quick a post to provide a little class for Windows 8 Store app development. The SystemInformation class gathers some useful information about the current system and can dump those as a string. I usually have such a class for all the platforms I work on and it's very helpful for logging and error analysis, but it also serves as a list of all available system information which are scattered in different APIs. The info can also be used to dynamically adapt the app, etc.

Here's the code I have so far, you can also download the complete class here.

public static async Task Dump(bool shouldDumpCompleteDeviceInfos = false)
{
   var builder = new StringBuilder();
   var packageId = Windows.ApplicationModel.Package.Current.Id;
   var clientDeviceInformation = new EasClientDeviceInformation();
    
   // Get hardware Id
   var token = HardwareIdentification.GetPackageSpecificToken(null);
   var stream = token.Id.AsStream();
   string hardwareId;
   using (var reader = new BinaryReader(stream))
   {
      var bytes = reader.ReadBytes((int)stream.Length);
      hardwareId = BitConverter.ToString(bytes);
   }

   builder.AppendLine("***** System Infos *****");
   builder.AppendLine();
#if DEBUG
   builder.AppendLine("DEBUG");
   builder.AppendLine();
#endif
   builder.AppendFormat("Time: {0}", DateTime.Now.ToUniversalTime().ToString("r"));
   builder.AppendLine();
   builder.AppendFormat("App Name: {0}", packageId.Name);
   builder.AppendLine();
   builder.AppendFormat("App Version: {0}.{1}.{2}.{3}", packageId.Version.Major, packageId.Version.Minor, packageId.Version.Build, packageId.Version.Revision);
   builder.AppendLine();
   builder.AppendFormat("App Publisher: {0}", packageId.Publisher);
   builder.AppendLine();
   builder.AppendFormat("Supported Package Architecture: {0}", packageId.Architecture);
   builder.AppendLine();
   builder.AppendFormat("Installed Location: {0}", Windows.ApplicationModel.Package.Current.InstalledLocation.Path);
   builder.AppendLine();
   builder.AppendFormat("Store App Id: {0}", CurrentApp.AppId);
   builder.AppendLine();
   if (CurrentApp.LicenseInformation.IsActive)
   {
      var listingInformation = await CurrentApp.LoadListingInformationAsync();
      builder.AppendFormat("Store Current Market: {0}", listingInformation.CurrentMarket);
      builder.AppendLine();
   }
   builder.AppendFormat("Culture: {0}", CultureInfo.CurrentCulture);
   builder.AppendLine();
   builder.AppendFormat("OS: {0}", clientDeviceInformation.OperatingSystem);
   builder.AppendLine(); 
   builder.AppendFormat("System Manufacturer: {0}", clientDeviceInformation.SystemManufacturer);
   builder.AppendLine();
   builder.AppendFormat("System Product Name: {0}", clientDeviceInformation.SystemProductName);
   builder.AppendLine();
   builder.AppendFormat("System Sku: {0}", clientDeviceInformation.SystemSku);
   builder.AppendLine();
   builder.AppendFormat("System Name: {0}", clientDeviceInformation.FriendlyName);
   builder.AppendLine();
   builder.AppendFormat("System ID: {0}", clientDeviceInformation.Id);
   builder.AppendLine();
   builder.AppendFormat("Hardware ID: {0}", hardwareId);
   builder.AppendLine();
   builder.AppendFormat("User Display Name: {0}", await UserInformation.GetDisplayNameAsync());
   builder.AppendLine();
   builder.AppendFormat("Window Bounds w x h: {0} x {1}", Window.Current.Bounds.Width, Window.Current.Bounds.Height);
   builder.AppendLine();
   builder.AppendFormat("Current Orientation: {0}", DisplayProperties.CurrentOrientation);
   builder.AppendLine();
   builder.AppendFormat("Native Orientation: {0}", DisplayProperties.NativeOrientation);
   builder.AppendLine();
   builder.AppendFormat("Logical DPI: {0}", DisplayProperties.LogicalDpi);
   builder.AppendLine();
   builder.AppendFormat("Resolution Scale: {0}", DisplayProperties.ResolutionScale);
   builder.AppendLine();
   builder.AppendFormat("Is Stereo Enabled: {0}", DisplayProperties.StereoEnabled);
   builder.AppendLine();
   builder.AppendFormat("Supports Keyboard: {0}", IsKeyboardPresent());
   builder.AppendLine();
   builder.AppendFormat("Supports Mouse: {0}", IsMousePresent());
   builder.AppendLine();
   builder.AppendFormat("Supports Touch (contacts): {0} ({1})", IsTouchPresent(), new TouchCapabilities().Contacts);
   builder.AppendLine();
   builder.AppendFormat("Is Network Available: {0}", NetworkInterface.GetIsNetworkAvailable());
   builder.AppendLine();
   builder.AppendFormat("Is Internet Connection Available: {0}", NetworkInformation.GetInternetConnectionProfile() != null);
   builder.AppendLine();
   builder.AppendFormat("Network Host Names: ");
   foreach (var hostName in NetworkInformation.GetHostNames())
   {
      builder.AppendFormat("{0} ({1}), ", hostName.DisplayName, hostName.Type);
   }
   builder.AppendLine();
   builder.AppendFormat("Current Memory Usage: {0:f3} MB", GC.GetTotalMemory(false) / 1024f / 1024f);
   builder.AppendLine();
   builder.AppendFormat("App Temp  Folder: {0}", ApplicationData.Current.TemporaryFolder.Path);
   builder.AppendLine();
   builder.AppendFormat("App Local Folder: {0}", ApplicationData.Current.LocalFolder.Path);
   builder.AppendLine();
   builder.AppendFormat("App Roam  Folder: {0}", ApplicationData.Current.RoamingFolder.Path);
   builder.AppendLine();
   builder.AppendLine();

   if (shouldDumpCompleteDeviceInfos)
   {
      var devInfos = await DeviceInformation.FindAllAsync();
      builder.AppendLine();
      builder.AppendLine("Complete Device Infos:");
      foreach (var devInfo in devInfos)
      {
         builder.AppendFormat("Name: {0} Id: {1} - Properties: ", devInfo.Name, devInfo.Id);
         foreach (var pair in devInfo.Properties)
         {
            builder.AppendFormat("{0} = {1}, ", pair.Key, pair.Value);
         }
         builder.AppendLine();
      }
   }

   return builder.ToString();
}

public static bool IsTouchPresent()
{
 return new TouchCapabilities().TouchPresent == 1;
}

public static bool IsMousePresent()
{
 return new MouseCapabilities().MousePresent == 1;
}

public static bool IsKeyboardPresent()
{
 return new KeyboardCapabilities().KeyboardPresent == 1;
}

On my system it provides the following info (anonymized):

***** System Infos *****

DEBUG

Time: Sun, 28 Oct 2012 07:51:11 GMT
App Name: MyApp
App Version: 1.0.0.5
App Publisher: CN=A8E2F0B1-4749-48AE-AE18-C7FFB5B30272
Supported Package Architecture: Neutral
Installed Location: D:\Development\MyApp\bin\Debug\AppX
Store App Id: 00000000-0000-0000-0000-000000000000
Culture: en-US
OS: WINDOWS
System Manufacturer: Dell Inc.
System Product Name: Precision M6600
System Sku: 
System Name: MYLAPTOP
System ID: 18a2e8c6-cc7d-8546-1e9e-56a12ffb32fd
Hardware ID: 03-00-D2-41-03-00-AA-72-08-00-EF-AD-05-AA-B6-4F-06-00-01-01-04-00-31-1B-04-00-0C-48-04-00-D2-55-04-00-D3-66-01-00-0E-D6-02-00-E4-7F-09-00-56-47
User Display Name: Rene Schulte
Window Bounds w x h: 1920 x 1080
Current Orientation: Landscape
Native Orientation: Landscape
Logical DPI: 96
Resolution Scale: Scale100Percent
Is Stereo Enabled: False
Supports Keyboard: True
Supports Mouse: True
Supports Touch (contacts): False (0)
Is Network Available: True
Is Internet Connection Available: True
Network Host Names: mylaptop (DomainName), mylaptop.local (DomainName), 169.254.85.85 (Ipv4), 192.168.2.148 (Ipv4), 
Current Memory Usage: 0.451 MB
App Temp  Folder: C:\Users\Rene\AppData\Local\Packages\MyApp\TempState
App Local Folder: C:\Users\Rene\AppData\Local\Packages\MyApp\LocalState
App Roam  Folder: C:\Users\Rene\AppData\Local\Packages\MyApp\RoamingState

If you know more system information Windows 8 APIs that listing is missing, please add a comment.

Update WriteableBitmapEx for WinRT RTM, WPF, Windows Phone and Silverlight

2012-08-29T11:45:00.000+02:00

The RTM version of Windows 8 is now available and with that WriteableBitmapEx was updated too. WriteableBitmapEx is now available for 4 platforms: WPF, Silverlight, Silverlight for Windows Phone and Windows Store Style WinRT .NET.
You can download the binaries here or via the NuGet package. The packages contain the WriteableBitmapEx binaries. As usual all samples and the source code can be found in the repository.

Since the last WinRT release preview version, a couple of bugs were fixed and a new FromStrean method was added which loads an image stream into a WriteableBitmap. The project was also updated for the Windows 8 RTM version. Please read this blog post for more details about the WinRT version.

Moving on

2012-06-19T17:09:00.001+02:00

Today was my last day at Schleupen AG. I worked there for 5 years as an enterprise developer mainly on back end and algorithmic topics, but also on the front end with .NET, Silverlight and more. We developed some really nice things there and I'm esp. proud of the parallel data processing pipeline which even works with .NET 2.
My co-workers were a great bunch of people, some became my friends and even the godfather of my youngest daughter and I became a godfather too. I thank them all for what I've learned and the great time we had. I wish them nothing less than the best for their future.

I started at Schleupen AG right after my studies and my diploma thesis "Parallelization and Optimization of an OpenGL CAD / CAM Application". I was always a computer graphics aficionado and passionate about physical simulation, image processing, computer vision, real-time computer graphics in general and interactive UIs with a great UX. Just for fun I started to work on some side projects, like this open source OpenGL fluid simulation in 2007, created a few open source libraries and also developed quite popular Windows Phone apps such as Pictures Lab, Funny Faces, Helium Voice, Cloud Recorder and more.

It's an exciting time to be a developer with so much things happening in this industry at the moment. Microsoft's own Surface tablets is just one piece we as software developers can leverage to shape the future. For me it's good fun to use these great new technologies like Windows Phone, Windows 8, WinRT, .NET, DirectX, XBox, Kinect, etc. to create nice experiences and I love to work on this all the time.
A few months ago I got an offer from IdentityMine and if you know their work, you can imagine that it was a no-brainer to accept this nice offer they made me. I'm really looking forward to work with state of the art technologies on exciting projects with experts like Laurent Bugnion and more great people. Another nice aspect of my new gig: I will work from my home office and will be closer to my 3 lovely daughters and wife. I will also be here during the last phase of my wife's pregnancy and can hopefully support her better. And yes, we're expecting our fourth child in September. :-)

Now I will enjoy a couple of days off with my family until I start my new job as Senior Interactive Developer at IdentityMine on July 2nd.

It's Alive! - WriteableBitmapEx 1.0 for WinRT Metro Style, WPF, Windows Phone and Silverlight

2012-06-14T17:13:00.000+02:00

After a few preview versions, I'm happy to announce that the final version of WriteableBitmapEx 1.0 is now available.
A couple of weeks ago we added official WPF support and WinRT Metro Style support. With that WriteableBitmapEx is now available for 4 platforms: WPF, Silverlight, Silverlight for Windows Phone and Metro Style WinRT .NET.
You can download the binaries here or via the NuGet package. The packages contain the WriteableBitmapEx binaries. All samples and the source code can be found in the repository.

Since the last WinRT preview version, a new WinRT sample was added, a couple of bugs were fixed and a new FromStrean method was added which loads an image stream into a WriteableBitmap. The project was also updated for the Windows 8 Release Preview. Please read this blog post for more details (about the WinRT version).

One Bitmap to Rule Them All - WriteableBitmapEx for WinRT Metro Style

2012-05-07T19:51:00.001+02:00

A couple of weeks ago we added official WPF support to WriteableBitmapEx. Today I'm happy to announce that WriteableBitmapEx now also officially supports Windows 8 Metro Stlye WinRT .NET XAML. With that WriteableBitmapEx is now available for 4 platforms: WPF, Silverlight, Silverlight for Windows Phone and Metro Style WinRT .NET.
Although Direct2D is the best solution for fast 2D graphics with Windows 8 Metro Style, I think there are scenarios where the WriteableBitmapEx could be helpful, esp. when using C# with XAML. I also know that some devs were waiting for this to port their Windows Phone apps to Windows 8 Metro Style.

WinRT Differences
Unlike the Silverlight WriteableBitmap, the Metro Style WriteableBitmap doesn't provide the pixel data directly. Its IBuffer PixelBuffer property doesn't have an interface to get the color information. Fortunately there are a few C# extension methods available which provide the pixel data as byte array or stream in the BGRA pixel format. Yes, BGRA and not like all the other platforms supported by WriteableBitmapEx as ARGB. The BGRA format is mainly used by Direct2D, which might be the reason for this hidden, but important difference of the Metro Style WriteableBitmap.
The WriteableBitmapEx algorithms are written for the ARGB pixel format. Fortunately I was able to keep the details away from the library user by leveraging the BitmapContext concept we introduced with the WPF support. This approach makes it possible to share almost the same code for all 4 platforms without being cluttered with #if directives all over place. Actually the most significant WinRT adaptation inside the WriteableBitmapEx methods was done in the FromContent method, which loads an image from the app content and provides it as WriteableBitmap. See this StackOverflow question I answered if you're interested in the details.
Nothing comes for free, but if the BitmapContext is used the right way, the performance hit won't be that much thanks to an internal reference counting WriteableBitmapEx' BitmapContext uses. No worries, you don't have to change all your WriteableBitmapEx calls, just wrap your calls in a simple using(writeableBmp.GetBitmapContext()) and you will only have one buffer conversion instead of one for each draw call.
It's really simple to use:

private void Draw()
{
   // Wrap updates in a GetContext call, to prevent invalidation overhead
   using (writeableBmp.GetBitmapContext())
   {
      writeableBmp.Clear();
      DrawPoints();
      DrawBeziers();
  
   } // Invalidates on exit of using block
}

private void DrawPoints()
{
   foreach (var p in points)
   {
      DrawPoint(p, Colors.Green, PointVisualSizeHalf);
   }
}

private void DrawPoint(ControlPoint p, Color color, int halfSizeOfPoint)
{
   var x1 = p.X - halfSizeOfPoint;
   var y1 = p.Y - halfSizeOfPoint;
   var x2 = p.X + halfSizeOfPoint;
   var y2 = p.Y + halfSizeOfPoint;
   writeableBmp.DrawRectangle(x1, y1, x2, y2, color);
}

private void DrawBeziers()
{
   if (points.Count > 3)
   {
      writeableBmp.DrawBeziers(GetPointArray(), Colors.Yellow);
   }
}

Screenshot WinRT Metro Style sample running in the simulator

All samples were tested with the new version, but due to the refactoring more testing is needed. Please test this version with your projects and report the bugs you encounter. You can download the binaries here. Note that this package only contains the WriteableBitmapEx binaries for Silverlight, Windows Phone, WinRT Metro Style .NET and WPF. All the samples can be found in the source code repository in the branch "WBX_1.0_BitmapContext". If all goes well, this branch will become the trunk and the 1.0 RTM in a few weeks.

WinMD / Windows Runtime Component
There's also a WinMD version available which makes it possible to consume the WriteableBitmapEx library from all the WinRT Metro Style projections, although only C# and C++ XAML make actually sense.
I had to move some parts and leave some functionality like the ForEach out, but it contains 99% of the library's features. Unfortunately the C++ sample I created crashes when the WriteableBitmapExWinMD library is loaded. So for now this WinMD version can be found in a separate branch "WBX_1.0_WinMD" in the source code repository and it won't be part of the trunk and release until it works with the sample. I'm a bit running out of time and don't know where to look for, since it seems all is wired up correctly and compiles fine. If you are a WinMD wizard and have a few minutes, I'd appreciate if you could look into the WinMD version.

Breakdown of Windows Phone App Download Statistics

2012-01-13T20:43:00.003+01:00

Line Graph by Scott Lewis, Noun

In this blog post I want to share the download graph of my quite successful Windows Phone app Pictures Lab and my conclusions. Some of you might have seen a similar graph already on Twitter or on Facebook, but I thought an update and a bit more explanation might probably make sense.

The graph below shows the download statistics of my Pictures Lab app during the last 9 months. The green line represents the daily downloads and the orange are the accumulated downloads. I blurred the numbers at the scales, since I don't release the download number for various reasons. Please note, Pictures Lab offers a trial mode which means users can try the app for free. Therefore the download numbers aren't equal to the number of sold licenses! Although the app has a pretty good conversion rate from trial-to-paid of ~32% worldwide average and in December it even was 97% in Norway! I noticed that all the Scandinavian countries seem to have a great conversion rate in general. Worst seems to be Hong Kong, but that's a story for a different blog post. Just to be clear, Pictures Lab provides me with a solid extra income each month and the trend is positive, but it's not enough to make a living out of that yet.
The absolute download numbers actually don't matter that much for this blog post. The important part of the graph are the text labels I manually added. Those labels mark events I tracked and noted at that time and are usually followed by a download peak.

Pictures Lab download graph from 04-01-2011 to 01-07-2012 (click on the image for the original size).

My conclusions

Steadily releasing updates with new functionality and being featured at the Windows Phone sites has a big impact on the downloads. Keep in mind the major Pictures Lab updates were featured at least at 3-5 of the top Windows Phone sites.
Being featured at the Marketplace also has a significant positive impact on downloads. Of course this depends on how big the local Marketplace is. Unfortunately does the AppHub not provide the information when an app has been featured. There is the Distimo service which tracks that too, but that service needs your live id credentials and I won't give a 3rd party site my live id to store it in their database. Fortunately some of my friends ping me when they see my apps being featured in their local marketplace. I think it's pretty much safe to say that all the graph peaks without a label nearby are related to a Marketplace feature.
The base line increased not only after the 3.0, but also after the peak of the 4.0 update. The 4.0 update brought the multi-language support with 10 languages. The steady download jump after 4.0 is a pretty good indication that localization can help to increase downloads.
The price drop to $0.99 was followed by the highest peak in the graph, but after that it went down pretty quickly. I think the price point is the hardest thing to get right and it largely varies for different kind of apps, so the above Pictures Lab pricing information shouldn't be generalized.
Of course Christmas, the holidays and New Year also resulted in nice peaks. Customers have time to browse the marketplace or just want to fill up their brand new Windows Phones with some apps.

What about the Windows Phone 7.5?
The first Windows Phone 7.5 (previously known as Mango) devices and Nokia hit the market in October and November 2011. This fact also needs to be considered when interpreting the graph. Although from looking at the statistics of my other apps I see that Windows Phone 7.5 didn't have such a huge impact like the updates had. The downloads definitely increased due to the Windows Phone 7.5 launch, but the tracked events align very well with the peaks in the graph and the increase of the base line after.
Let me try to clarify this and my conclusion #3 with the graphs of my successful Helium Voice Free app and my not so successful Benchmark Free app.

Helium Voice Free got a nice update in November, but I didn't inform the Windows Phone sites, so there are no peaks in the graph. You can see that the release of Windows Phone 7.5 slightly increased the downloads and the downward trend turned into an upward trend.

Helium Voice Free graph from 04-01-2011 to 01-07-2012 (click on the image for the original size to see numbers).

In the Benchmark Free download graph there's also this small increase of the average downloads around the Windows Phone 7.5 device launch, but it's not as much as after the updates of Pictures Lab.

Benchmark Free graph from 04-01-2011 to 01-07-2012 (click on the image for the original size).

Considering all that information, I think this means the Windows Phone 7.5 launch of course raised the downloads base line, but the Pictures Lab updates had a bigger impact on the downloads and it's not only the Windows Phone 7.5 launch which raised the average downloads of Pictures Lab.
Keep in mind that such a statistical analysis isn't bullet-proof at all, esp. when there's not all the information available like the date of all marketplace features and more. Please also note not all apps are the same and things that work for Pictures Lab or Helium Voice Free don't necessarily need to be valid for other apps.

Makes sense?
What are your conclusions?

What is your experience?

Thank you!

I would like to thank all the users of the app, the great group of translators who helped me to translate Pictures Lab, the beta testers and not at last all the great Windows Phone sites which keep us informed about all the things happening around Windows Phone and help us developers by informing the world about our apps. Keep it up!

Let me out - Facebook Logout in a Windows Phone App

2012-01-05T21:30:00.000+01:00

A while ago I implemented the Facebook photo endpoint into my Windows Phone Pictures Lab app. The implementation of the login was quite straightforward thanks to OAuth 2.0. Only the logout was way harder than one might expect. This post describes how to logout from Facebook using the Facebook API.

In my Pictures Lab app you can edit photos, make them look awesome and then save or share those with your friends at Twitter or Facebook. The Windows Phone Mango API provides the ShareLinkTask and the ShareStatusTask which can be used by an app to share an URL or text using the social services the user has connected the device to. Unfortunately there's no built-in SharePhotoTask to share a photo using the services the user has already authorized. That's why I had to implement it in a custom way where the user has to authorize again. This blog post by my mate Nick Randolph describes very well how to login to Facebook from a Windows Phone app.

For some situations it might make sense to allow the user to logout from within the app. One might think this can't be hard and in most cases it's pretty easy. Logging out from Twitter is very easy for example. You just have to start the authorization process again. However, logging out from Facebook is way harder since they store a cookie and the WebBrowser control doesn't provide a way to clear the cookies, so just starting the authorization process again doesn't work.
One way to log out from Facebook uses a special Uri that contains a part of the access token which was queried during the app authorization process.

Here's the snippet I use in Pictures Lab to split the access token to get the session key which is then used to build the custom Uri:

public Uri GetLogoutUri(FacebookCredentials credentials)
{
   var sessionkey = ExtractSessionKeyFromAccessToken(credentials.AccessToken);
   var url = String.Format("http://facebook.com/logout.php?app_key={0}&session_key={1}&next={2}", EndpointData.FacebookAppId, sessionkey, EndpointData.FacebookLogoutCallbackUrl);
   return new Uri(url);
}

private static string ExtractSessionKeyFromAccessToken(string accessToken)
{
   if (!String.IsNullOrEmpty(accessToken))
   {
      var parts = accessToken.Split('|');
      if (parts.Length > 2)
      { 
         return parts[1];
      }
   }
   return String.Empty;
}

That logout Uri is then used to navigate the WebBrowser control to it which then correctly triggers the log out process.

Browser.Navigate(FacebookService.GetLogoutUri(EndpointData.Settings.Facebook));

That's it. You just have to know their trick. Hope this helps.

SLARToolkit Samples Updated

2012-01-05T20:25:00.003+01:00

The samples of my open source Windows Phone and Silverlight Augmented Reality Toolkit were updated to the latest version of the WP 7.1 and Silverlight 5 SDKs.
Please note the changed security model in Silverlight 5, which is a big bummer. My Silverlight MVP friend Morten wrote a few true words about it here.
As usual you can find a list of the samples on the project site and also get the code there.

WriteableBitmapEx 1.0 Coming Soon, Test it Now!

2011-12-20T20:37:00.000+01:00

A new version of the WriteableBitmapEx open source library has just been released, but that isn't the last release for this year. No, Andrew Burnett-Thompson and I refactored the library to make it easier portable and we added full WPF support. Andrew did most of the work since he needed the current WriteableBitmapEx library for one of his WPF projects. As a result of the refactoring, WriteableBitmapEx will have maintained support for WPF starting with version 1.0.
I'd have loved to add support for WinRT too, but unfortunately it seems that WinRT only supports streamed reading / writing of the pixel buffer at the moment. I will wait until Microsoft ships the Windows 8 beta early next year and see what they have in there. Many WriteableBitmapEx algorithms need random buffer index access and I don't want to waste my time with massive memory copying now. Who knows what else comes in the beta and it might be better to use a whole different approach for immediate rendering with Windows 8 and WinRT.

All samples were tested with the new version, but due to the massive refactoring more testing is needed. Please test the beta version with your projects and report any bugs you find. You can download the binaries here. Note that this package only contains the WriteableBitmapEx binaries for Silverlight, Windows Phone and WPF. All the samples can be found in the source code repository in the branch "WBX_1.0_BitmapContext". If all goes well this branch will become the trunk in a couple of weeks.

Windows Phone Sites Around the World

2011-12-05T20:52:00.002+01:00

I recently released an update of my Windows Phone app Pictures Lab which brings some nice new features like effect combination, two Christmas and a New Year frame and most importantly 7 new languages. Pictures Lab 4.0 now supports 9 languages: English, German, Japanese, Russian, Dutch, French, Italian, Spanish, Portuguese. This was made possible by a group of awesome native-speaking translators: Alan Mendelevich, Takeshi Miyauchi, David Salazar, Joost van Schaik, Johan Peeters, Paolo Barone, Simone Chiaretta, Pedro Lamas and Quentin Calvez.

I also asked my translators if they know any local Windows Phone news sites / blogs. Below is a list of international and regional Windows Phone news sites we collected. Note, that those are primarily consumer sites read by consumers and not only for developers. This list could be useful if you plan to release a localized version of your app and want to promote it a bit. Most sites have a "Tip Us" or contact form and are mostly happy about new content to write posts about.

Forget this list!
Head over to @ailon's awesome dedicated site: http://windowsphonesites.com

English

German

Russian

Japanese

Dutch

http://wp7.nl

Spanish

Portuguese (Brazilian)

http://windowsphonebrasil.com.br

Italian

French

http://www.monwindowsphone.com

Danish

http://www.windowsphonefan.dk

If you know more sites, please leave a comment and I'll update the list.

Telerik published a nice whitepaper with more resources about app promotion.

WriteableBitmapEx 0.9.8.5 Now Available

2011-10-28T20:36:00.004+02:00

The WriteableBitmapEx open source library has come a long way since I created the CodePlex site in December 2009. A lot of features and the support for new platforms were added in subsequent releases. The package is also available via NuGet since this year.
The new release v0.9.8.5 was just made public. A few new features were added and many small, uncritical issues were fixed (see the list below). The binaries can be downloaded from here or via the NuGet package. Please note that this package only contains the WriteableBitmapEx binaries for Silverlight and Windows Phone. All the samples can be found in the source code repository.

Feature list version 0.9.8.5

Added a Rotate method for arbitrary angles (RotateFree). Provided by montago.
Added Nokola's anti-aliased line drawing implementation.
Updated the Windows Phone project to WP 7.1 Mango.
Added an extension code file for the Windows Phone specific extensions and added SaveToMediaLibrary extensions including support for saveToCameraRoll.
Added an Invert method, which creates an inverted version of the input bitmap. This is useful for WP7 Theme-awareness checks using Application.Current.Resources["PhoneBackgroundBrush"].
Added FromContent method, which provides an easy interface to load a WriteableBitmap from the content of the app.
Added a static overload for the Resize method which takes the pixels array as argument.
Optimized the DrawLine algorithm.
Fixed some issues with DrawRectangle, FillRectangle, DrawEllipse, FillEllipse and DrawPolyline.
Fixed a bug in the bilinear Resize method that appeared when the alpha value is zero.
Fixed other minor issues.

Community Community Community!
Thanks to the community for constantly reporting bugs, suggesting new features and contributing code. That's exactly why open source software is just awesome.

Welcome to Zombieland, the Metro Style Land of WinRT and the Undead

2011-09-17T20:04:00.004+02:00

Silverlight is dead! WPF is dead! .NET is dead! Hey, they didn't talk about SharePoint or SQL Server at the BUILD conference, those must be dead too. Welcome our new Metro-id overlords!
We read such FUD everywhere these days. Actually none of these technologies are dead for the next several years! And you'll probably agree if you know the facts and not the FUD. So here are my thoughts based only on the facts I got from watching BUILD sessions, reading posts that stick to the facts and not listening to people who obviously didn't inform themselves, but nevertheless spread FUD around.

What happened
Microsoft announced the new version of Windows at the BUILD conference and the new Windows 8 runtime called WinRT, which is used to develop Metro style Apps. Metro style Apps are primarily focused on consumers, designed touch first and therefore perfect for modern multi-touch devices like tablets.
The good ol' desktop is still there and the usual Windows applications are now called Desktop Apps.
There's a lot confusion out there at the moment and many think only Metro style Apps are the future. Actually both models can exist side-by-side and I'm sure the mainly used UI will be the classic desktop for the usual business client. Metro is for tablets, maybe later for the phone. Desktop Apps will still work better for the classic business PC scenario in an office.
After trying Windows 8 I don't see myself using Metro style Apps a lot on my PC when I work at the desk. However, I will love to use it when I'm hanging out on my couch with a tablet. The good news is, both models are supported by Windows 8 and therefore can run on one device, including ARM hardware. Awesome!

WinRT

First and foremost, .NET plays a significant role in both development models and is not only used for Desktop Apps. I guess the FUD comes from the architectural picture above, where .NET and Silverlight are only represented by a small box in the lower right corner. Of course Microsoft wants to push the new Metro style Apps, the WinRT model and wants to get web developers into the boat, so they adjusted the graphic and marketing message accordingly.
WinRT is actually a new native COM library, plus some extra infrastructure. Therefore an app developer has not to deal with the COM stuff directly, instead you get a thin / fast projection layer (binding) for each of the supported programming languages. This projection layer is automatically built using WinMD metadata and provides projections for JavaScript, .NET (C# / VB) or C / C++. The UI can be designed with XAML or HTML / CSS in case of JavaScript.
The rendering is completely done using DirectX 11.1 (Direct2D) which results in great performance. In Windows 8 the rendering job is finally fully executed by the right processor. This architecture makes it also possible to implement the complete UI just with DirectX, which will usually be done by games. At the moment it's not possible in the Windows 8 Developer Preview to mix a XAML app with DirectX, which doesn't make much sense since the XAML rendering is done by DirectX. Fortunately all hints that it'll be possible in a later version of Windows 8, maybe in the beta.

The below picture by Doug Seven is a way more accurate picture of the Windows 8 platform architecture.

There are a few good, unbiased posts about WinRT out there, which stick to the facts:
WinRT: An Object Orientated Replacement for Win32
WinRT, the C++ Component Extensions
WinRT demystified
A bad picture is worth a thousand long discussions.
WinRT reference with all namespaces at the unbiased MSDN, where JaveScript is just another language one can use for Metro style Apps.

WinRT and .NET
From what I have seen so far, coding Metro style Apps using C# / .NET seems pretty straightforward if you've done Silverlight, WPF or esp. Windows Phone development before. The BCL used by .NET WinRT is not the full Desktop version of .NET 4.5, it's a reduced set similar to the Silverlight types.
The design of the native WinRT COM library was heavily influenced by .NET. You see it everywhere. Type names, Properties, Events... Even WinMD is the .NET assembly metadata format.
WinRT types map to .NET types and copying of data structures at the boundaries is avoided by the projection layer to get the best performance. Only two types need to be converted using built-in extension methods. The System.IO.Stream can't be mapped to the WinRT stream, so there's an extension method. A Byte[] to WinRT IBuffer conversion is the other extension method.
You can even write your own, custom WinRT components in C# without decorating the classes with ugly COM attributes. Those components are then automatically exposed to the other languages using the generic WinMD metadata. So you can write a component in C++ or in C# and use it from a JavaScript WinRT project. Pretty cool concept if you ask me.
Many WinRT classes look like their origin is .NET / Silverlight and the WinRT WriteableBitmap is also very similar to Silverlight's implementation. So expect a WinRT version of my open source WriteableBitmapEx library in a few weeks when I'm done with the Mango updates for all my Windows Phone apps. After that I will probably port / rewrite some of my WP7 apps for WinRT.

The lifecycle of a WinRT app should be quite familiar for a WP7 developer, since it's similar to the WP 7.5 Mango Fast App Switching. The lifecycle states are: Running > Suspended in RAM > Terminated. Apps get an event for suspending, not for terminate, so the state has to be saved during suspending. Of course an app should only be resurrected from the tombstone state if the app was launched, not when resumed. Apps get 5s for suspending and need to launch within 15s. Unfortunately we don't know the reference machine where those values are measured. Probably the low end configuration.
By the way, the Windows Store will be the better version of the WP7 Marketplace. It will include In-App offers, time limited trials and a very nice dashboard with a lot of analytic capabilities and insights.
In general should a WP7 developer feel most familiar with WinRT. Many concepts made it into WinRT and were improved and actually cleaned up.
My Silverlight MVP friend Morten Nielsen started a blog post series about how to port Silverlight / WPF apps to WinRT.
You should also read this post: WinRT and .NET and watch the ton of great BUILD sessions.

Conclusion

I welcome the new possibilities and great performance we get in Windows 8 with the WinRT. I'm sure Windows 8 will be an awesome tablet OS.
XAML has become a first class citizen and .NET is now installed with Windows 8. Microsoft is also working on a next version of .NET. Version 4.5 of .NET will bring better performance for WPF's ItemsControl for example. Seems not very dead to me. In fact, a Silverlight / WPF / WP7 developer's knowledge of XAML and the Silverlight runtime is more valuable than ever before with WinRT.
Silverlight 5 is still in development and wasn't even released. We don't know if it will be the last version or if Silverlight 6 will see the light of the day. For now I'm quite happy with all the features of Silverlight 5 and we can develop a ton of applications for our customers for the next several years.

One should also not forget how long it will take until Windows 8 is finally out and reached a critical mass. A lot of machines out there still run Windows XP. Windows 7 is way better and I guess the transition to Windows 8 will take even longer. Windows 8 will probably mainly pushed by non-PC multi-touch consumer devices in the near future.

This flowchart by the Telerik guys sums it up pretty nicely.

So get yourself informed and start coding away. Don't give FUD a chance!

Imagine Cup 2011 Worldwide Finals

2011-06-23T22:41:00.004+02:00

The Imagine Cup is one of the world's premier student tech competitions. Students compete in different categories like Software Design, Game Design, Windows Phone 7 and more. The competitions begin each year with local events in over 100 countries. The best of the best will then participate in the worldwide final. The 2011 Imagine Cup Worldwide Finals will be held from 8 - 13 July in New York City, USA.

I'm very honored to take part in the Imagine Cup 2011 Worldwide Finals as a judge in the Windows Phone category. The Windows Phone judges consist of two community members and two Microsoft employees.

People's Choice Award
There's also a People's Choice Award where everyone can vote for their favorite team.

Can you say no if Eva Longoria asks you to vote?

You can find a list of all the Windows Phone finalist teams here. All the teams have built great solutions around the theme for 2011 Imagine a world where technology helps solve the toughest problems.
There's no filter for a category at the People's Choice Award website, that's why I made a list of all the Windows Phone finalists entries:

DREGON by team "Digitron-WP7" from Belgium.
Peekaboo by team "HOMERUN" from Korea.
Hot Potato by team "Zipi Zigi" from Korea.
Recyclocator by team "Code Instincts" from Singapore.
Lifelens by team "The LifeLens Project" from USA.

Please cast your vote.

Cubelicious - Silverlight 5 + Balder + Physics + SLARToolkit Augmented Reality = Triple Win!

2011-06-16T21:45:00.003+02:00

Two months ago I released a new Silverlight 5 sample for my open source Silverlight Augmented Reality Toolkit - SLARToolkit. It uses the new Silverlight 5 hardware accelerated 3D API. You can read more about the new Silverlight 5 XNA 3D API in this blog post.
This post here provides a new demo for SLARToolkit which uses the open source 3D engine Balder by my friend Einar Ingebrigsten. This demo also leverages the open source physics engine JigLibX my buddy Andy Beaulieu ported over to Silverlight. You can try the live sample if you have the Silverlight 5 beta installed or watch a video instead.

I actually spent most of the time on this project a couple of weeks ago. My good Silverlight MVP friend Einar Ingebrigtsen used it in his talk at The Gathering 2011. Now I finally had a bit time to finish the demo, make a video and this blog post.

The SLARToolkit project description from the CodePlex site:

SLARToolkit is a flexible Augmented Reality library for Silverlight with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource or a WriteableBitmap. SLARTookit is based on the established NyARToolkit and ARToolkit.

Live

A webcam and at least the Silverlight 5 beta runtime must be installed to run the sample. It's available here. Alternatively there is also a video of the new sample embedded below.

If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. As an alternative it's also possible to open a marker file on a mobile device and to use the device's screen as marker. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.

Open the sample in a new page.

Simply press the "Start Cam" Button to start the webcam. The properties of the particle system can be changed with Sliders. The "Flip x-axis" Checkbox can be used to flip the video (the webcam output is mirror-reversed by default).

If you click the "Start Cam" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Video

I've recorded a short video of the new sample with Expression Encoder's Screen Capture feature. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is even better. The video is also available at YouTube.

Background music is Lullaby by _ghost

This demo shows how the new Silverlight 5 3D API, the Balder engine and the JigLibX physics library can be used to augment the reality with the help of SLARToolkit.

How it works

This sample uses the webcam video stream which fills a Rectangle shape, the video stream is also constantly captured and fed to the SLARToolkit BitmapMarkerDetector to detect the markers. The detection result contains a transformation matrix for each found marker which is then used to apply a global transformation to the cubes and the plane.
I implemented a particle system with a flexible directed emitter which can be controlled through various properties. The particle system is quite generic and can be used for all kinds of particles (3D objects). The particle collision detection and resolving is based on rigid body physics that was implemented with the help of the JigLibX library my Silverlight MVP buddy Andy Beaulieu ported over to Silverlight.
The rendering and the model loading is done by the 3D engine Balder. It's a fantastic open source engine by Einar Ingebrigtsen. You just need to write a couple lines of XAML and you're good to go. This sample only uses a simple cube model, but Balder has built-in model loaders to load complex 3D models and Einar provides a big sample library. He also brought the engine to a few more platforms like Windows Phone 7, OpenGL and has even a neat software rendering fallback. Read his blog post here.
As part of this sample I needed some vector and quaternion methods which were missing in Balder. I contributed those and the generic particle system to the Balder project. Feel free to use the particle system and the other methods in your Balder projects.

Download it, build your app and augment your reality

The open source SLARToolkit library and all samples are hosted at CodePlex. If you have any comments, questions or suggestions don't hesitate and write a comment, use the Issue Tracker on the CodePlex site or contact me via any other media.
Have fun with the library and please keep me updated if you use it anywhere so I can put a link on the project site.

Nodo vs Mango - Windows Phone ListBox Performance Improvements in Action

2011-05-27T01:05:00.002+02:00

There were quite a few complaints about the Windows Phone's ListBox scrolling performance in the past. The Windows Phone team obviously heard it and heavily worked on the performance and responsiveness of the whole platform and the ListBox in particular. The touch input processing was loaded off from the UI thread to a new separate thread. Additionally the BitmapImage doesn't load the data on the UI thread anymore. I'm sure lots of other tweaks were implemented to increase the performance and the responsiveness of the platform. I think the Windows Phone team did a very good job!

Showdown

The video below shows a side-by-side comparison of a Nodo device with build 7392 and a prototype device with a Mango prebuild. The Nodo device on the left side is a Samsung Omnia 7, the Mango device on the right side is the ASUS prototype.

For comparison I use the official Twitter app build against 7.0 and in the second half of the video I show the effects preview list of my Pictures Lab app. This effects pivot item only uses a ListBox with a DataTemplate that contains an Image control with a fixed size and a TextBox for each item. So there's no background image loading being performed or any other heavy computing, just a static list with a lot of redrawing.

Background music is Taiga by SMILETRON

As you can see, it's quite a huge difference and even this Mango prebuild runs very smooth on this rather old ASUS hardware. There's still some room left for more improvement, like the rasterizer, but imagine the boost on production devices with the final Mango version.

MANGO, MANGO, MANGO!

I don't have a deal with Microsoft or are paid to blog or tweet this. I'm just exited about all the goodies that are coming with Mango. Good times.

Unity 3D and Silverlight in Harmony with SLARToolkit's Augmented Reality

2011-05-25T21:01:00.002+02:00

If you read this blog regularly, you might have noticed a lot of SLARToolkit posts recently. But I'm not the only one using the open source Silverlight Augmented Reality Toolkit. There's great stuff happening out there and I see quite a few projects, especially in the academic field. You can find a full list of projects (I know of) on the SLARToolkit project site at CodePlex.
I love when people use my open source work, adapt or learn from it. This is what open source is all about for me.

One project recently caught my attention in particular. Ioulian Alexeev made a very amazing action game using the Unity 3D game engine and he combined it with SLARToolkit to control the spaceship with a marker. Ioulian studies Multimedia at the Arteveldehogeschool in Belgium and the game "StarFighter" was his bachelors test.

He recorded this nice video.

Note how he compares the performance of the Flash Augmented Reality Tookit and SLARToolkit:

Performance FLARManager: Not good enough for action games
Performance SLARToolkit: Very fast and accurate marker tracking

Both toolkits use the same marker-based AR algorithms from the ARToolkit.

Live
You can play the game here. If you want to try it yourself you need do download the SLAR and / or L marker, print them and point the camera toward these. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. As an alternative it's also possible to open a marker file on a different device and to use the device's screen as marker.
See the SLARToolkit Markers documentation for more details.

Why is a Y in the Windows Phone Mango Camera API

2011-05-25T20:31:00.003+02:00

The release of the new Mango tools brings Windows Phone development on par with Silverlight 4 and will therefore add many great features to the Windows Phone platform. This means it will also contain the Webcam CaptureSource and VideoSink API from Silverlight 4. Additionally it also introduces the new FileSink class which can be used to record the video stream as MP4 to the Isolated Storage. Most important a new PhotoCamera class with a lot of functionality is present in Mango.
This class is used in the latest SLARToolkit Windows Phone sample and in some other new projects I'm working on.

Don't Reinvent the Wheel
The Silverlight 4 webcam API was explained in this detailed blog post almost a year ago. The techniques and concepts I described there can now also be used with Windows Phone Mango.
The updated MSDN documentation has quite a few articles and samples about the new camera API. My MVP buddy Alex Golesh also has a nice write up about the new Camera API.
This blog post tries to fill the gaps and provides some information especially about the PhotoCamera's YCrCb capture methods.

YCbCr vs ARGB
In the well-known RGB color space the red, green and blue information is stored in separate components which also contains the redundant luminance data for each channel. In the YCbCr color space (or YCrCb) the luminance information is stored in the Y component and the chroma (color) information in the Cb component as blue-difference and in Cr component as red-difference. The RGB-YCbCr conversion can be done with simple addition and multiplication operations. The Y component usually ranges from 0 to 1, Cb and Cr from -0.5 to 0.5.

Y = 0.5, Cr [-0.5, 0.5], Cb[-0.5, 0.5]

Humans are more sensitive to luminance information than to chroma, therefore the resolution of the color information can be reduced and only the luminance needs to be stored in full resolution. Many digital camera sensors use the YCrCb color space and make use of this reduced chroma information.

PhotoCamera
The PhotoCamera class has a lot of useful methods to either capture a full resolution image from the camera or to get a smaller (and faster) preview buffer snapshot. The GetPreviewBufferY and GetPreviewBufferYCrCb methods provide the direct data from the camera without a transformation to 32 bit ARGB. Not only is the alpha channel left out in the YCrCb buffer, also the Cr and Cb color components are stored with reduced resolution. This keeps the buffer size smaller and is way faster, but also makes it a bit trickier when the color components (and brightness) need to be extracted from the byte buffer. Fortunately there's the YCbCrPixelLayout property which contains alls the offsets, strides and other needed information.

Conclusion
The GetPreviewBufferYCrCb method is approximately 4 times faster than the GetPreviewBufferArgb32 method and also takes a smaller buffer, therefore the YCrCb methods are the way to go when only the luminance data is needed or the YCbCr color space can be used for the given scenario. For example many computer vision techniques only need the luminance information for processing.
I like that both color spaces are supported by the API. On mobile devices you need all the performance you can get. I actually helped the Windows Phone camera team with quite a bit feedback to decide about this API design. Very smart people by the way.

Augmented Mango - SLARToolkit for Windows Phone

2011-05-24T22:18:00.005+02:00

The beta of the new Windows Phone Developer Tools was just publicly released. The update with the codename "Mango" comes with many new APIs and will finally contain an API for real-time camera access what a lot of developers have been asking for. The new runtime gives us the needed functionality to implement many cool scenarios. One of these scenarios is Augmented Reality, which leads to my open source Silverlight Augmented Reality Toolkit (SLARToolkit).
This post announces the new Windows Phone version of SLARToolkit and also provides a sample. If you're one of those lucky people with a Mango-enabled device you can download the XAP here or just watch a video instead.

The SLARToolkit project description from the CodePlex site:

SLARToolkit is a flexible marker-based Augmented Reality library for Silverlight and Windows Phone with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource, WriteableBitmap or with the Windows Phone's PhotoCamera. SLARTookit is based on the established NyARToolkit and ARToolkit.

Demo

The sample XAP can be deployed to a Mango-enabled device (tested with build 7629). Alternatively there's also a video of the new sample embedded below.

If you want to try it yourself you need do download the SLAR and / or L marker, print them and point the camera toward these. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. As an alternative it's also possible to open a marker file on a different device and to use the device's screen as marker.
See the SLARToolkit Markers documentation for more details.

Video

I've recorded a short video of the new sample with my Samsung Omnia 7. It's a bit blurry, but it demonstrates how well the sample works even on this quite old ASUS prototype, which's camera pipeline seems a bit slow.
The video is also available at YouTube.

Background music is Melo by Mosaik

This demo shows how the new Windows Phone Mango real-time camera API can be used to augment the reality with the help of the SLARToolkit. This can be nice for educational projects and it's actually no problem to add correctly transformed videos or other content to the demo.
The demo demonstrates just some basic UIElements like a TextBox and an Image control. Mango will also enable the combination of Silverlight and XNA, which means that nice 3D AR games can be developed with the help of the SLARToolkit.

How it works

This sample uses the new PhotoCamera and a timer to constantly get a snapshot of the real-time camera stream. This snapshot is then passed to the SLARToolkit algorithms to get the 3D spatial information of the marker. The computed detection results are used to transform the elements perspectively correct.

protected override void OnNavigatedTo(System.Windows.Navigation.NavigationEventArgs e)
{
   base.OnNavigatedTo(e);

   // Initialize the webcam
   photoCamera = new PhotoCamera();
   photoCamera.Initialized += PhotoCameraInitialized;
   isInitialized = false;

   // Fill the Viewport Rectangle with the VideoBrush
   var vidBrush = new VideoBrush();
   vidBrush.SetSource(photoCamera);
   Viewport.Fill = vidBrush;

   // Start timer
   dispatcherTimer = new DispatcherTimer { Interval = TimeSpan.FromMilliseconds(50) };
   dispatcherTimer.Tick += (sender, e1) => Detect();
   dispatcherTimer.Start();
}

The PhotoCamera instance is set up in the OnNavigatedTo event handler of the page and the DispatcherTimer is started. The timer will constantly call the Detect method every 50 milliseconds. Additionally a viewfinder Rectangle is filled with a VideoBrush which in turn has the photoCamera video stream set as source.

void Detect()
{
   if (!isInitialized)
   {
      return;
   }

   // Update buffer size
   var pixelWidth = photoCamera.PreviewBufferResolution.Width;
   var pixelHeight = photoCamera.PreviewBufferResolution.Height;
   if (buffer == null || buffer.Length != pixelWidth * pixelHeight)
   {
      buffer = new byte[pixelWidth * pixelHeight];
   }

   // Grab snapshot
   photoCamera.GetPreviewBufferY(buffer);

   // Detect
   var dr = arDetector.DetectAllMarkers(buffer, pixelWidth, pixelHeight);

   // Calculate the projection matrix
   if (dr.HasResults)
   {
      // Center at origin of the 256x256 controls
      var centerAtOrigin = Matrix3DFactory.CreateTranslation(-128, -128, 0);
            
      // Swap the y-axis and scale down by half
      var scale = Matrix3DFactory.CreateScale(0.5, -0.5, 0.5);

      // Calculate the complete transformation matrix based on the first detection result
      var world = centerAtOrigin * scale * dr[0].Transformation;

      // Viewport transformation
      var viewport = Matrix3DFactory.CreateViewportTransformation(pixelWidth, pixelHeight);

      // Calculate the final transformation matrix by using the camera projection matrix 
      var m = Matrix3DFactory.CreateViewportProjection(world, Matrix3D.Identity, arDetector.Projection, viewport);

      // Apply the final transformation matrix to the controls
      var matrix3DProjection = new Matrix3DProjection { ProjectionMatrix = m };
      Txt.Projection = matrix3DProjection;
      Img.Projection = matrix3DProjection;
   }
}

A snapshot of the current preview buffer is taken in the Detect method using the GetPreviewBufferY method. This method fills up a byte buffer with the luminance data of the current viewfinder frame. This buffer is then passed to the SLARToolkit's MarkerDetector Detect method, which returns the detected marker information. This transformation data is then used to transform the UIElement perspectively correct in 3D.
Read more about the PhotoCamera's YCbCr methods in this blog post.

void PhotoCameraInitialized(object sender, CameraOperationCompletedEventArgs e)
{
   //  Initialize the Detector
   arDetector = new GrayBufferMarkerDetector();

   // Load the marker pattern. It has 16x16 segments and a width of 80 millimeters
   var marker = Marker.LoadFromResource("data/Marker_SLAR_16x16segments_80width.pat", 16, 16, 80);

   // The perspective projection has the near plane at 1 and the far plane at 4000
   arDetector.Initialize(photoCamera.PreviewBufferResolution.Width, photoCamera.PreviewBufferResolution.Height, 1, 4000, marker);

   isInitialized = true;
}

The SLARToolkit's GrayBufferMarkerDetector is created and set up in the PhotoCamera's Initialized event handler. The brand new GrayBufferMarkerDetector uses the byte buffer with luminance data directly without the need of an ARGB 32 bit pixel conversion.

Checkout the source code at CodePlex if you want to see all the details of the sample which were left out for clarity.

Download it, build your app and augment your reality

Have fun with the library and please keep me updated if you use it anywhere so I can put a link on the project site.

WriteableBitmapEx goes NuGet

2011-04-19T00:03:00.002+02:00

The WriteableBitmapEx library is now available as a NuGet package. The package contains the Silverlight and the Windows Phone binaries. The sources and the PDB are also available at SymbolSource.org.
NuGet is a neat package management system for the .Net platform which makes the life of a .Net developer much easier. If you haven't tried NuGet until now, you should definitely give it a try. I'm sure you won't regret it. To install NuGet, open Visual Studio's Tools -> Extension Manager and search the Online Gallery for NuGet. You can then open a project, right click the References and select "Add Library Package Reference".

Silverlight 5 3D + SLARToolkit Augmented Reality = Win!

2011-04-13T21:29:00.000+02:00

It's almost been a year since I wrote a sample for my open source Silverlight Augmented Reality Toolkit - SLARToolkit. The release of the new Silverlight 5 hardware accelerated 3D API was a nice occasion to finally make a new one.
In my last blog post I wrote a summary of all the Silverlight 5 beta features and some notes about the new low-level, XNA 3D API.
This post provides the new demo for SLARToolkit which leverages this fast GPU-based rendering to draw some nice effects with 60 frames per second. You can try the live sample if you have the Silverlight 5 beta installed or watch a video instead.

The SLARToolkit project description from the CodePlex site:

SLARToolkit is a flexible Augmented Reality library for Silverlight with the aim to make real time Augmented Reality applications with Silverlight as easy and fast as possible. It can be used with Silverlight's Webcam API or with any other CaptureSource or a WriteableBitmap. SLARTookit is based on the established NyARToolkit and ARToolkit.

Live

A webcam and at least the Silverlight 5 beta runtime must be installed to run the sample. It's available here. Alternatively there is also a video of the new sample embedded below.

Open the sample in a new page.

Simply press the "Start Fun" Button to start the webcam. The size of the objects can be changed with the "Scale" Sliders. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default). Click the "Glass" Checkbox to apply a glass effect to the sun model.

If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Video

Background music is Neon Aurora by Alpha C

This demo shows how the new Silverlight 5 3D API can be used to augment the reality with the help of SLARToolkit. It also demonstrates how the 3D DrawingSurface can be combined with the webcam video stream and overlaid with ordinary TextBoxes. This can be nice for educational projects and it's actually no problem to add correctly transformed videos or other content to the demo.

How it works

This sample uses the webcam video stream which is used to fill a Rectangle shape, the video stream is also constantly captured and fed to the SLARToolkit BitmapMarkerDetector to detect the markers. The detection result contains a transformation matrix for each found marker which is then used to transform the 3D objects and the TextBoxes.
A couple of vertex and pixel shaders are used to get the desired effects. Techniques like Phong shading, Bump mapping and Refraction mapping (glass) were implemented. The snapshots from the webcam are passed as a texture to the refraction pixel shader to simulate the glass effect.
See the source code if you're interested in the nitty-gritty details. Please note that I wrote an introduction to Pixel Shaders for Coding4Fun a while ago.
The current Silverlight 5 beta doesn't implement RenderTargets, therefore effects like bloom aren't really possible with the limitations of the Shader Model 2. You can add a faded billboard around the sun, but it wouldn't look that nice and when RenderTargets are added we can do way more nice effects in future releases of Silverlight 5.
I also added a new simple anti-jittering functionality to SLARToolkit to prevent the jiggling that mostly occurred due to the varying lightning conditions and noise in the video stream.

Credits
The SolarWind sample by the Silverlight team was used as a base and extended. The sample uses earth textures from the NASA. The pixel shader for the sun uses textures and concepts from an article by Nicolas Menzel. The moon textures are from the Celestia project.

Download it, build your app and augment your reality

XNA for the Web - Silverlight 5

2011-04-13T19:49:00.004+02:00

Many great new things were announced during the second keynote at Microsoft's MIX11 conference. One of it was the Silverlight 5 beta which is available for download here.
The new features are just awesome and many of them were requested by the community. As usual Tim Heuer's great blog post covers all the new goodies in detail and provides videos and source code for most of them. Kudos to the Silverlight team for releasing another great version.
My blog post contains a summary of all the new features the Silverlight 5 beta ships and provides some notes about the new 3D API. The next blog post will bring a new SLARToolkit 3D demo.

Features

XNA 3D Graphics API
Improved Graphics stack – The graphics stack has been re-architected to bring over improvements from WP7, such as Independent Animations.
Hardware video decode for H.264 playback.
Multi-core background JIT support for improved startup performance.
Realtime Sound (low-latency Audio)
Variable Speed Playback (“Trick Play”)
XAML Binding Debugging
Multiple Window Support
Ancestor RelativeSource Binding
Implicit DataTemplates
ClickCount
Binding on Style Setter
Linked Text Containers
Custom Markup Extensions
ComboBox type ahead with text searching.
Full keyboard support in full-screen for trusted in-browser applications, enables richer kiosk and media viewing applications in-browser.
Default filename in SaveFileDialog – Specify a default filename when you launch the SaveFileDialog.
Unrestricted filesystem access – trusted applications can Read write to files in any directory on the filesystem.
More performance optimizations:
XAML Parser performance optimizations.
Network Latency optimizations.
Text layout performance improvements.
Hardware acceleration is enabled in windowless mode with Internet Explorer 9.

Find more details at the Silverlight site and read John Papa's and Pete Brown's blog posts.

XNA for the Web
One of the topmost requested features at the Silverlight Uservoice site was hardware accelerated 3D graphics. The Silverlight team listened closely and added an immediate mode, low-level, XNA 3D API to Silverlight 5. An immediate mode API offers functionality to render graphics directly to the graphics interface without keeping a list of the objects. A retained mode API like WPF 3D on the other hand keeps an internal list of the object graph. An immediate mode API is a great choice since it provides the maximum control over the rendering and mostly results in superior performance compared to retained mode rendering.
The Silverlight 5 3D API doesn't contain all the higher-level classes one might know from the XNA Game Studio. The Content Pipeline, the SpriteBatch class and others were left out. I assume this decision was made to keep the footprint of the Silverlight runtime small. However, the team added all the needed core functionality; features like the SpriteBatch can be build on top of it by the community. My Silverlight MVP friend Bill Reiss is already working on a SpriteBatch version for Silverlight 5.
The new GPU accelerated 3D API gives us Silverlight developers the core XNA functionality with shaders and vertex rendering inside the browser! Shaders aren't easy for most non-graphics programmers, but they give us a huge flexibility and we can build various higher levels on top of the new 3D API.
You can render some pretty impressive things with the new Silverlight 5 3D API and it's possible to draw millions of triangles on the GPU without almost no CPU load.

Here's a sneak peek at the new SLARToolkit demo which also contains a simple real-time glass effect.

Silverlight Integration
The 3D API contains a control called the DrawingSurface, which is basically a rendering surface or a 3D Canvas if you will. The DrawingSurface has the Draw event which is raised for every frame to draw. The event is fired on a render thread which runs in parallel to the UI thread. This is an important concept introduced with Silverlight 5.
Inside a Draw event handler the states and shaders are set and the vertex and index buffers are passed to the GPU for rendering. The vertex and pixel shaders implement the lightning model and define how the vertices are rendered. The current Silverlight 5 3D beta only supports Shader Model 2, but this will probably change in a future release, but even with Shader Model 2 many neat lightning models and effects can be implemented. You will find some nice effects in the new SLARToolkit demo.

Check out the Silverlight 5 3D samples if you want to learn more about the new 3D API. If you're not familiar with 3D graphics concepts or just want to make your life easier, then I strongly recommend to give the 3D engine Balder a try. It's a fantastic open source engine by my good Silverlight MVP friend Einar Ingebrigtsen. You just need to write a couple lines of XAML and you're good to go. Balder has built-in model loaders to load complex 3D models and Einar provides a big sample library. He also brought the engine to a few more platforms like Windows Phone 7, OpenGL and has even a neat software rendering fallback. Read his blog post here.
Another Silverlight MVP friend of mine, Andy Beaulieu made a very cool demo using Balder and a 3D physics engine. He blogged about it and even released the 3D physics engine as open source!

The beta of the 3D API is missing some essential things like RenderTargets at the moment, but it's just a beta and more stuff might be announced at the MIX11 conference. If you would like to know more details about the new Silverlight 5 3D API and the roadmap, then you should definitely check out the MIX11 session Graphics & 3D with Silverlight 5 by the Silverlight program manager Aaron Oneal on Wednesday, April 13 3:30 pm - 4:30 pm PDT. All MIX sessions are recorded and will be available online 24 hours after each session.

Conclusion
No, Silverlight is not dead and it's clear that Microsoft is investing in it. I'm sure the new features and esp. the XNA 3D API will lead to some great applications in the future. The future is bright and we are part of it.