Dotnet by Example

Service announcement - this blog has moved on

noreply@blogger.com (Joost van Schaik) — Wed, 29 Jul 2020 08:36:00 +0000

Intro

In November 2007, I started my journey into blogging. Little did I know I was taking the first step that would ultimately made me a Microsoft MVP, took me several times to the campus in Redmond, and even put me on the road to HoloLens development. Back then I took the easy and cheap route and opted for Blogger, then an independent company. Unfortunately, the company was later acquired by Google and after that the platform did not get much love anymore. The online editor, at the time great, became more cumbersome to use over time. Then came LiveWriter, part of Microsoft Essentials, and suddenly I could blog in a WYSIWIG Word style editor. This made life so much easier.

But Live Essentials was slowly being phased out, and little by little, Live Writer was also sunsetting. But in the end of 2015 the product was given a new leash on life by being rechristened as "Open Live Writer". This was especially important as Google changed the Blogger API regularly. When it became a Store app, life became even easier: it got updated automatically!

That is... for the time enthusiasts were still willing to maintain it. The repo associated with Open Live Writer has been showing little activity the last year. Also the Blogger platform, having received very little love by Google, has lost a lot of usage, to such an extent that repos for blogging tools simply close down issues asking for Blogger support calling it "a bunch of work for little payback". And this comment is from 2017.

Recently I had my laptop reinstalled, and when I wanted to re-configure Open Live Writer again, I was greeted with this:

At this point, I had enough. After almost 13 years jumping through several hoops to keep this puppy alive, it was time to move on.

DotNetByExample - The Next Generation

And here we are. I have copied over all the content to my new home on the web, using GitHub pages, after consulting with my dear friend Matteo Pagani. This is the provisionary look & feel, there is still a lot I don't like, but at least it's accessible and maintainable for me again. I also wrote a little tool that updates all links to previous blogposts inside the imported pages, and even generates a JavaScript snippet that redirects all links to the migrated articles in the new blog.

So after 13 years, it's time to leave this spot that has been my home on the web for so long. I hope you will all join me on my continuing journey in development!

Migrating to MRKT2 - NuGet for Unity crashes when using MRTK2

noreply@blogger.com (Joost van Schaik) — Wed, 22 Jul 2020 13:48:00 +0000

Intro

Recently I was setting up a new HoloLens 2 project. I usually take the following route:

Create a Unity project with the recommended version for the current MRTK (I use 2.4 now, so that is 2018.4.20f1)
Switch build platform to UWP
Import the MRTK2 as source
Add the MRKT2 to the scene
Apply optimization settings
Select the right sound Spatializer (MS HRTF Spatializer)

And then, since recently, I add NuGet for Unity.

Something is rotten in de state of NuGet

Now this comes a nice UnityPackage that you can download from here. Currently it's at 2.0.1. It works fine in plain Unity projects. It works also fine in UWP build. But if you follow the workflow I show above, it breaks. If you open the NuGet package manager, in stead of the UI you expect, you will see only this:

And you will see these errors in the console:

System.TypeInitializationException: The type initializer for 'NugetForUnity.NugetHelper' threw an exception. ---> System.NullReferenceException: Object reference not set to an instance of an object
at NugetForUnity.NugetHelper.LoadNugetConfigFile () [0x0011c] in <b7bde984cef1447da61a3fd28d4789b0>:0
at NugetForUnity.NugetHelper..cctor () [0x001e6] in <b7bde984cef1447da61a3fd28d4789b0>:0
NullReferenceException: Object reference not set to an instance of an object
NugetForUnity.NugetHelper.LoadNugetConfigFile () (at <b7bde984cef1447da61a3fd28d4789b0>:0)

Great. Now what?

The culprit

When you import the the MRTK2, at some point a NuGet.config file is created. This sits in the Assets root and looks like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="MSBuildForUnity" value="https://pkgs.dev.azure.com/UnityDeveloperTools/MSBuildForUnity/_packaging/UnityDeveloperTools/nuget/v3/index.json" />
  </packageSources>
</configuration

For some reason, NuGet for Unity does not get very happy about that. If you quit Unity, delete NuGet.config and the NuGet.config.meta, start again and open the NuGet package manager again, a new NuGet.config is created that looks like this:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <packageSources>
       <add key="NuGet" value="http://www.nuget.org/api/v2/" />
    </packageSources>
    <disabledPackageSources />
    <activePackageSource>
       <add key="All" value="(Aggregate source)" />
    </activePackageSource>
    <config>
       <add key="repositoryPath" value="./Packages" />
       <add key="DefaultPushSource" value="http://www.nuget.org/api/v2/" />
    </config>
</configuration

and if you now click select the NuGet package manger again, it starts up normally.

If tried to add the MSBuildForUnity to the package sources like this, because maybe the MRTK2 needs it for something:

<packageSources>
   <add key="NuGet" value="http://www.nuget.org/api/v2/" />
   <add key="MSBuildForUnity" value="https://pkgs.dev.azure.com/UnityDeveloperTools/MSBuildForUnity/_packaging/UnityDeveloperTools/nuget/v3/index.json" />
</packageSources

but all that does is generate another error:

"Unable to retrieve package list from https://pkgs.dev.azure.com/UnityDeveloperTools/MSBuildForUnity/_packaging/UnityDeveloperTools/nuget/v3/index.jsonSearch()?$filter=IsLatestVersion&$orderby=DownloadCount desc&$skip=0&$top=15&searchTerm=''&targetFramework=''&includePrerelease=false
System.Net.WebException: The remote server returned an error: (404) Not Found."

So I actually have no idea why this NuGet.config is created or how. But it definitely needs to go, apparently.

The short version

Before importing NuGet for Unity, check for an existing NuGet.config and NuGet.config.meta and delete those; your NuGet package manager will then work as planned.

No code today, because all I wrote today here boils down to the one sentence above :)

Migrating to MRTK2: Changing viewpoint rotation angle when pushing the thumbstick

noreply@blogger.com (Joost van Schaik) — Tue, 23 Jun 2020 10:54:00 +0000

Intro

A short one, but one that took me quite some time to find.

If you are using MRTK2 to build a Mixed Reality app to run on Immersive Headsets - or at least an environment where you control the environment using controllers - you can typically 'teleport' forward and backward by pushing the thumbstick forward and backward, and rotate the view by pushing it left and right. Your viewpoint rotates either 90° left or 90° right (and of course you can change the view by rotating your head, but that's not the point). But what if you want to rotate only 45°? Or 22.5°?

I have spent quite some time looking in the MRKT2 config profiles looking for the rotation angle. The reason I could not find it, is because it's not there. At least, not directly. It's in the pointer - the 3D cursor, if you like. Yeah, you read that right.

First, we clone

Starting from a default MRKT2 project, we follow the following steps:

Clone the Toolkit Configuration Profile itself. I used DefaultMixedRealityToolkitConfigurationProfile to start this time.
Under Input, clone the Mixed Reality Input System Profile
Under Pointers, clone the Mixed Reality Input Pointer Profile

Finding the right pointer

If you expand the Pointers section and then the Pointer Options sections, you will see a whole lot of pointers. The second one is the one we are looking for: ParabolicPointer. It's a prefab.

If you click it, it will open the asset folder Assets/MRTK/SDK/Features/UX/Prefabs/Pointers and highlight the prefab we just selected:

Finding the angle

Press CTRL+D to duplicate the pointer, then drag it the Prefabs folder inside your app:

If that folder is not there yet, create it first. Once the prefab is inside the folder, rename it to something recognizable - ParabolicPointer45 for instance. When you are done renaming you have automatically selected the pointer itself. Go over to the inspector, scroll all the way down to the "Parabolic Teleport Pointer" script and there, in the "Teleport Pointer Settings" section, you will finally see where that bloody 90° comes from:

Adapting and applying

So now it's simply a matter of changing the angle to what you want, so for instance 45°. Then you go back to your adapted Mixed Reality Input Pointer Profile and drag your customized point in the right place:

and now your view will only rotate 45° in stead of 90° when you push the thumb stick sideways.

Conclusion

I certainly did not expect this setting being contained in a pointer, as by now I am quite conditioned to look in configuration profiles by now. I was - incorrectly - assuming it would be somewhere in the camera or controller settings. It took me quite some searching and debug breakpoints to find out where things where happening, and I took a few blind alleys before I found the right one. Typically something I blog about so a) I can now easily find back how I did it and b) you now also know where to look for.

Demo project, although it does not do very much, here. When you run it (either from Unity or as an app) in your Immersive Headset, you will see a green 'floor' rotating only 45° when you push the thumbstick stick left or right.

Fixing the "Windows Mixed Reality specific camera code has been moved..." message when upgrading the MRKT2

noreply@blogger.com (Joost van Schaik) — Wed, 03 Jun 2020 09:28:00 +0000

A very short and very simple trick today:

If you - like me - ~~are crazy enough to~~ dutifully update the MRTK2 in your apps whenever a new version arrives, at some point you will notice a kind of puzzling, rather ominous and not very helpful message pop up in your console a couple of times:

"Windows Mixed Reality specific camera code has been moved into Windows Mixed Reality Camera Settings. Please ensure you have this added under your Camera System's Settings Providers, as this deprecated code path may be removed in a future update."

You might have seen this appear earlier, when you upgraded to MRTK 2.3, as you can see in this commit message. Version 2.3 was where it was introduced, in November 2019. The trouble with this message is that the procedure to fix it is not entirely clear - at least, it was not to me. Especially since the first warning originates from the MRTK2 itself (when the MixedRealityCameraSystem is initialized as a service). Also GoogleBinging it yields no result, but I do hope it does so after I post this article. Fortunately it was also originating from my own code, and that was a lot more helpful, as I know what that is about - I wrote it myself, after all.

The culprit was this piece of code:

CoreServices.CameraSystem.IsOpaque

This ends up in Microsoft.MixedReality.Toolkit.CameraSystem.MixedRealityCameraSystem where the IsOpaque property is implemented. That looks for a "IMixedRealityCameraSettingsProvider". That does not help much, but wat does help - in MixedRealityCameraSystem there is a documentation link on top, which at least points you to the right place in the configuration.

So what you need to do is very simple and takes 7 mouse clicks:

Select the MixedRealityToolkit node in your hierarchy
Select the Camera Tab
Expand the Camera Settings Provider

You will now see this:

Then:

Click the "+ Add Camera Settings Provider" button just above "Display Settings"
Expand the "New data provider 0" section"

For "Type", select Microsoft.MixedReality.Toolkit.WindowsMixedReality/WindowsMixedRealityCameraSettings (this is actually two mouse clicks)

The end result should look like this:

.... and you are done. Unless you need to tweak the settings of the default camera settings profile, in which case you have to do the clone-and-adapt dance. But in any case, no more scary messages about deprecated code that might change in some future release. ;)

I omitted the customary demo project this time, as the point of this post was to show you how to get from A to B, not the end result.

Migrating to MRTK2: using a Spatial Mesh inside the Unity Editor

noreply@blogger.com (Joost van Schaik) — Fri, 22 May 2020 04:50:00 +0000

Intro

If you are developing an app using the Mixed Reality Toolkit 2 that requires interaction with a Spatial Mesh, the development process can become cumbersome: add code or assets with Unity and Visual Studio, create IL2CPP solution, wait, compile and deploy, wait, check behavior - rinse and repeat. You quickly learn to do as much as possible inside the Unity editor and/or use Holographic Remoting if you want stay productive and make your deadline. But a Spatial Mesh inside the Unity Editor does not exist.

... or does it? ;)

Begun the Clone Wars have again

You guessed it - before we can see anything at all, a lot of cloning and configuring of profiles needs to be happening first.

Clone the Toolkit Configuration Profile itself. I used DefaultHoloLens2CameraProfile this time.
Turn off the diagnostics (as ever).
Enable Spatial Awareness System
Clone the MixedRealityAwareness profile
Clone the MixedRealityAwarenessMeshObserver profile (the names of these things become more tongue-breaking the deeper you go)
Change the Display option (all the way down) to "Occlusion"

And now the interesting part

On top of the Spatial Awareness System Settings there's this giant button over the whole length of the UI which is labeled "+ Add Spatial Observer".

If you click that one, it will add a "New data provider 1" at the bottom, below the Display settings we just changed the previous step.

Select "SpatialObjectMeshObserver" for type

And if you hit the play button, lo and behold:

Basically you are now already where you want to be, but although the wireframe material works very well inside a HoloLens, it does not work very well in an editor. At least, that is my opinion.

Making the mesh more usable inside the editor

You might have noticed the SpatialObjectMeshObserver comes with a with a profile "DefaultObjectMeshObserverProfile" - I'd almost say of course it does. Anyway, clone that one as well. Then we create a simple material:

Of course using the Mixed Reality Toolkit Standard shader. I only change the color to RGB 115,115,115 which is a kind of battleship grey. You make take any color you fancy, as far as I am concerned. Set that material to the "Visible Material" of the Spatial Mesh Object Observer you just added (not in the material of the "Windows Mixed Reality Spatial Mesh Observer"!)

The result, if you run play mode again, is definitely better IMHO:

Using a mesh of a custom environment

So it's nice to be able to use some sample mesh, but what if you need to the mesh of a real space? No worries, because just like in HoloLens1, the device portal allows you to download a scan of the current (real) space the HoloLens sees:

You can download this space by hitting the save button. This will download a SpatialMapping.obj file. Bring this into your Unity project, then drag it on top of the Spatial Mesh Object observer's "Spatial Mesh Object" property:

And then, when you hit play mode, you will see the study where I have been hiding in during these worrying times. It has been my domain for working for the past 2.5 months for working and blogging, as well as following BUILD and the Mixed Reality Dev Days. If you download the demo project, it will also include a cube that moves forward, to show objects actually bounce off the fake spatial mesh, just like a real spatial mesh.

Note: if you compile and deploy this project to a HoloLens (either 1 or 2) you won't see this 'fake mesh' at all. It only appears in the editor. Which is exactly what we want. It's for development purposes only.

Conclusion

Using this little technique you can develop for interacting with the Spatial Mesh while staying inside the Unity editor. You will need less access to a physical HoloLens 2 device, but more importantly speed up development this way. The demo project is, al always, on GitHub

Migrating to MRTK2: right/left swappable hand menu's for HoloLens 2

noreply@blogger.com (Joost van Schaik) — Sun, 03 May 2020 16:21:00 +0000

Intro

As I announced in this tweet that you might remember from February the HoloLens 2 version of my app Walk the World sports two hand palm menus - a primary menu, often used command menu that is attached to your left hand that you operate with your right hand, and a secondary less-used settings menu that is attached to your right hand - and that you operate with your left. Lorenzo Barbieri of Microsoft Italy, a.k.a. 'Evil Scientist' ;) did the brilliant suggestion I should accommodate left-handed usage as well. And so I did - I added a button to the settings menu that actually swaps the 'handedness' of the menus. This means: if you select 'left handed operation' the main menu is operated by your left hand, and the secondary settings menu by your right.

A little video makes this perhaps more clear:

This blog explains how I made this work. I basically extracted the minimal code from my app and made it into a mini app that doesn't do more than make the menu swappable - both by pressing a toggle button and by a speech command. I will discuss the main points, but not everything in detail - but as always you can download a full sample project and see how it's working in the context of a complete running app.

This sample uses a few classes of my MRTKExtensions library of useful scripts.

Configuring the Toolkit

I won't cover this in much detail, but the following items need to be cloned and partially adapted:

The Toolkit Configuration Profile itself (I usually start with DefaultMixedRealityToolkitConfigurationProfile). Turn off the diagnostics (as ever)
The Input System Profile
The SpeechCommandsProfile
The RegisteredServiceProviderProfile

Regarding the SpeechCommandsProfile: add two speech commands:

Set left hand control
Set right hand control

In the RegisteredServiceProviderProfile, register the Messenger Service that is in MRKTExtension.Messaging. If you have been following this blog, you will be familiar with this beast. I introduced this service as a Singleton behavior back in 2017 and converted it to a service when the MRTK2 arrived.

Menu structure

I already explained how to make a hand menu last November, and in my previous blog post I explained how you should arrange objects that should be laid out in a grid (like buttons). The important things to remember are:

All objects that are part of a hand menu should be in a child object of the main menu object. In the sample project, this child object is called "Visuals" inside each menu.
All objects that should be easily arrangeable in a grid, should be in a separate child object within the UI itself. I always all this child object "UI", and this is where you put a GridObjectCollection behaviour on.

Consistent naming makes a structure all the more recognizable I feel.

Menu configuration

The main palm menu has, of course, a Solver Handler and a Hand Constraint Palm Up behaviour. The tracked hand is set to the left.

The tricky thing is always to remember - the main menu is going to be operated by the dominant hand. For most people the dominant hand is right - so the hand to be tracked for the dominant menu is the left hand, because that leaves the right hand free to actually operate controls on that menu. For left hand control the main menu has to be set to track the right hand. This keeps confusing me every time.

On the palm. It won't surprise you to see the Settings menu look like this:

With the Solver's TrackedHandedness set to Right. But here you also see the star of this little show: the DominantHandController, with the DominantHandController set to off - since I always have the settings menu operated by the non dominant hand, whatever that might be.

DominantHandController

This is actually a very simple script, that responds to messages sent from either a button or from speech commands:

namespace MRTKExtensions.HandControl
{
    [RequireComponent(typeof(SolverHandler))]
    public class DominantHandHelper : MonoBehaviour
    {
        [SerializeField] 
        private bool _dominantHandControlled;
        private IMessengerService _messenger;
        private SolverHandler _solver;

        private void Start()
        {
            _messenger = MixedRealityToolkit.Instance.GetService<IMessengerService>();
            _messenger.AddListener<HandControlMessage>(ProcessHandControlMessage);
            _solver = GetComponent<SolverHandler>();
        }

        private void ProcessHandControlMessage(HandControlMessage msg)
        {
            var isChanged = SetSolverHandedness(msg.IsLeftHanded);
            if (msg.IsFromSpeechCommand && isChanged && _dominantHandControlled)
            {
                _messenger.Broadcast(new ConfirmSoundMessage());
            }
        }

        private bool SetSolverHandedness(bool isLeftHanded)
        {
            var desiredHandedness = isLeftHanded ^_dominantHandControlled ?
                                      Handedness.Left : Handedness.Right;
            var isChanged = desiredHandedness != _solver.TrackedHandness;
            _solver.TrackedHandness = desiredHandedness;
            return isChanged;
        }
    }
}

SetSolverHandedness determines what the handedness of the solver should be set to - depending on whether this menu is set to be controlled by the dominant hand or not, and whether or not left handed control is wanted. That's an XOR yes, you don't see that very often. But write out a truth table for those two parameters and that's where you will end up with. This little bit of code is what actually does the swapping of the menus from right to left and vice versa.

It also returns a value to see if the value has actually changed. This is because if the command is started from a speech command we want, like any good Mixed Reality developer, give some kind of audible cue the command has been understood and processed. After all, we can say a speech command any time we want, and if the user does not have a palm up, he or she won't see hand menu flipping from one hand to the other. So only if the command comes from a speech command, and actual change has occurred, we need to give some kind of audible confirmation. I also added this confirmation only to be given by the dominant hand controller - otherwise we get a double confirmation sound. After all, there are two of these behaviours active - one for each menu.

Supporting act: SettingsMenuController

Of course, something still needs to respond to the Toggle Button being pressed. This is done by the this little behaviour:

    public class SettingsMenuController : MonoBehaviour
    {
        private IMessengerService _messenger;

        [SerializeField]
        private Interactable _leftHandedButton;

        public void Start()
        {
            _messenger = MixedRealityToolkit.Instance.GetService<IMessengerService>();
            gameObject.SetActive(CapabilitiesCheck.IsArticulatedHandSupported);
            _messenger.AddListener<HandControlMessage>(ProcessHandControlMessage);
        }

        private void ProcessHandControlMessage(HandControlMessage msg)
        {
            if (msg.IsFromSpeechCommand)
            {
                _leftHandedButton.IsToggled = msg.IsLeftHanded;
            }
        }

        public void SetMainDominantHandControl()
        {
            SetMainDominantHandDelayed();
        }

        private async Task SetMainDominantHandDelayed()
        {
            await Task.Delay(100);
            _messenger.Broadcast(new HandControlMessage(_leftHandedButton.IsToggled));
        }
    }
}

The SetMainDominantHandControl is called from the OnClick event in the Interactable behaviour on the toggle button:

and then simply fires off the message based upon the toggle status of the button. Note that there's a slight delay, this has two reasons:

Make sure the sound the button plays actually has time to play
Make sure the button's IsToggled is actually set to the right value before we fire off the message.

Yeah, I know, it's dicey but that's how it apparently needs to work. Also note this little script not only fires off HandControlMessage but also listens to it. After all, if someone changes the handedness by speech commands, we want to see the button's toggle status reflect the actual status change.

Some bits and pieces

The final piece of code - that I only mention for the sake of completeness - is SpeechCommandProcessor :

namespace HandmenuHandedness
{
    public class SpeechCommandProcessor : MonoBehaviour
    {
        private IMessengerService _messenger;

        private void Start()
        {
            _messenger = MixedRealityToolkit.Instance.GetService<IMessengerService>();
        }

        public void SetLeftHanded(bool isLeftHanded)
        {
            _messenger.Broadcast(new HandControlMessage(isLeftHanded) 
                                                        { IsFromSpeechCommand = true });
        }
    }
}

It sits together with a SpeechInputHandler in Managers:

Just don't forget to turn off the "Is Focus Required" checkbox as these are global speech commands. Itned to forget this, and that makes for an amusing few minutes of shouting to your HoloLens without it having any effect, before the penny drops.

Conclusion

You might have noticed I don't let the menu's appear on your hand anymore but next to your hand. This comes from the design guidelines on hand menu's on the official MRKT2 documentation, and although I can have a pretty strong opinion about things, I do tend to take some advice occasionally ;) - especially when it's about usability and ergonomics. Which is exactly why I made this left-to-right swappability in the first place. I hope this little blog post will give people some tools to add a little bit to inclusivity for HoloLens 2 applications.

Full project, as mostly always, here on GitHub.

Migrating to MRKT2 - easily spacing out menu buttons using GridObjectCollection

noreply@blogger.com (Joost van Schaik) — Sat, 02 May 2020 15:16:00 +0000

Intro

This is simple, short but I have to blog it because I discovered this, forgot about it, then discovered it again. So if anything this blog is for informing you as well al to make sure I keep remembering this myself.

If you have done any UI design for Mixed Reality or HoloLens, you have been in this situation. The initial customer requirement ask for a simple 4 button menu. So you make a neat menu in a 2x2 grid, and are very satisfied with yourself. The next day suddenly you find out you need two more buttons. So - do you make a 2x3 or a 3x2 menu? You decide on the latter, painstakingly arrange them in a nice grid again.

The day after that, there's 2 more buttons. The day after that, 3 more. And the next day... you discover GridObjectCollection. Or in my case, rediscover it.

Simple automatic spacing

So here is our simple 2x2 menu in the hierarchy. This is a hand menu. It has has a more complex structure than you would imagine, but this is because I am lazy and want an easily adaptable menu that can be organized by GribObjectCollection

The point is, everything that needs to be easily organizable by GridObjectCollection, needs to be a child of the object that has the actual GridObjectCollection behaviour attached. In my case that's the empty gameobject "Controls". Now Suppose I want this menu not to be 2x2 but 1 x 4. I simply need to change "Num Rows" into 1, press the "Update Collection" button and presto:

Of course, you will need to update the background plate and move the header text, but that's a lot less work than changing the layout of these buttons. Another example: change the default setting for "Layout" from "Row Then Column" to "Colum then Row", set "Num Rows" to 1 again (for it will flip to 4 when you change the Layout dropdown) and press "Update Collection" again:

You can also change how the button spacing by changing Cell Height and Cell Width. For instance, if I have a 4x4 grid and a cell distance width of 0.032 they are perfectly aligned together without any space in between (not recommended for real live scenario's where you are supposed to press these buttons - a mistake is easily made this way)

You can also do fun things like having then sorted out by name, child order, and both reversed. Or have them spaced out on not only a flat surface, but on a Cylinder, Sphere or a Radial area.

Note: the UpdateCollection can also be accessed by code, so you can actually use this script runtime as well. I mainly use it for static layouts.

Conclusion

Don't waste time in manual spacing, use this very handy tool in the editor to make a nice an evenly spaced button menu - or for whatever you need to have laid out.

Note:

Make it yourself easy by putting any parts of an UI that should be in a grid in a separate empty game object and put the GridObjectCollection control on that, and place the other parts outside that, so they won't interfere with the layout process.
You can use this behaviour with any type of game object, not only buttons of course
More details about GridObjectCollection can be found on the documentation page on GitHub. This also handles related behaviours like ScatterObjectCollection and TileObjectCollection.

No code so no project, although in the next blog post this technique will be applied 'in real life', so to speak.

Migrating to MRTK2 - configuring, understanding and using Windows Mixed Reality controllers

noreply@blogger.com (Joost van Schaik) — Fri, 24 Apr 2020 11:45:00 +0000

Intro

Although the focus for Mixed Reality Toolkit 2 now understandably is on Microsoft's big Mixed Reality business player - HoloLens 2 - it's still perfectly doable - and viable, IMHO - to develop Windows Mixed Reality apps for WMR immersive headsets. Case in point: most of the downloads I get for my three Mixed Reality apps in the store come from people using immersive headsets - which is actually not that strange as immersive headsets are readily available for individuals whereas HoloLens (either 1 or 2) is not - and they cost 10-15% of an actual HoloLens to boot.

And the fun thing is, if you do this correctly, you can even make apps that run on both - with only minor device specific code. Using MRTK2, though, there are some minor problems to overcome:

The standard MRTK2 configuration allows for only limited use of all the controller's options
There are no samples - or at least none I could find - that easily shows how actually extend the configurations to leverage the controller's full potential
Ditto for samples on how to intercept the events and use those from code.

I intend to fix all of the above in this article. Once and for all ;)

Configuration

If you have worked a bit with the MRTK2 before, you know what's going to follow: cloning profiles, cloning profiles and more cloning profiles. We are going some four levels deep. Don't shoot the messenger ;)

Assuming you start with a blank Unity app with the MRTK2 imported, first step is of course to clone the Default profile - or whatever profile you wish to start with, by clicking Copy & customize.

While you are at it, turn off the diagnostics

Next step is to clone the Input System Profile. You might need to drag the inspector a bit wider or you won't see the Clone button

Step 3 is to clone the Controller Mapping Profile:

Expand the "Controller Definitions" section. If you then select Windows Mixed Reality Left Hand Controller, you will notice a lot of events are filled in for the various controls - but also that a couple are not:

You can select something, but it's not applicable or already assigned to something else. The missing events are:

Touchpad Press
Touchpad Position
Touchpad Touch
Trigger Touch
Thumbstick Press

So we have to add these events. To achieve this, we have to do one final clone: the Default Input Actions Profile.

And then you simply can add the five missing events (or input actions, as they are called in MRKT2 lingo).

Mind to select "Digital" for all new actions except for Touchpad position. Make that a "Dual Axis". That last one will be explained later.

Now you can once again go back to Input/Controller/Input Mappings settings, and, assign the proper (new) events to the controller buttons. Don't forget to do this for both the right and the left hand controller.

And now finally there are events attached to all the buttons to the controllers. Now it's time to show how to trap them.

Understanding and using the events

The important thing to understand is that there are different kind of events, that all need to be trapped in a specific way. When I showed you to add the event types, all but one of them were digital types. Only one was "Dual Axis". There actually are a lot of different types of events:

I am not sure if I got all the details right, but this is what I found out:

a digital event, that's basically a click. You need to have a behaviour that implements IMixedRealityInputHandler to intercept this. Example: a click on the menu button
a single axis event is an event that gives you a single value. The only application for WMR controllers I have found is a way to determine how far the trigger is pushed inwards (on a scale of 0-1). You will need to implement IMixedRealityInputHandler<float>
a dual axis event gives you two values. The only application I found was the touchpad - it gives to the X,Y coordinates where the touchpad was touched. Range for both is -1 to 1. 0,0 is the touchpad's center. You will need to implement IMixedRealityInputHandler<Vector2>
a six dof (degrees of freedom) event will give you a MixedRealityPose. This enables you to determine the current grip and pointer pose of the controller. You will need to implement IMixedRealityInputHandler<MixedRealityPose>

Demo application

I created an application that demonstrates the events you will get and what type they are. If available, it will also display the values associated with the event. It looks like this:

Not very spectacular, I'll admit, but it does the trick. On the top row it displays the type of event intercepted, the bottom two rows show actual events with - in four cases - information associated with the events. When activated: the red circle will turn green.

Observations using the demo app

You will notice you'll get a constant stream of Grip Pose and Pointer Pose events - hence these two events and the MixedRealityPose type events indicator are always green
You will also get a constants stream of "Teleport Direction" events (of type Vector2) from the thumbstick even if you don't touch it. I have no idea why this is so. I had to filter those out, or else the fact Touchpad position is a Vector2 element got hidden in the noise.
Grip press is supposed to be a SingleAxis event, but only fires Digital events
If you touch the touchpad, it actually fires two events simultaneously - the Digital Touchpad Touch and the Vector2 Touchpad position.
Consequently, if you press the touchpad, you get three events - Touchpad touch, Touchpad Position and Touchpad Press.
The trigger button also is an interesting story as that fires three events as well. As soon start to press it ever so slightly, it fires the SingleAxis event "Trigger" that tells you how far the trigger is depressed. But at the lowest scale where "Trigger" registers, it will also fire the Digital "Trigger Touch" event. However, you will usually get a lot more "Trigger" events as it's very hard to keep the trigger perfectly still while it's halfway depressed.
And finally, when you fully press it, the Digital "Select" event will be fired.
Menu and Thumbstick press are simple Digital events as you would expect.

Key things to learn from the demo app

Registering global handlers

On top you will see the the ControllerInputHandler implementing being derived from BaseInputHandler and the four interfaces mentioned.

public class ControllerInputHandler : BaseInputHandler, 
    IMixedRealityInputHandler, IMixedRealityInputHandler<Vector2>, 
    IMixedRealityInputHandler<float>,
    IMixedRealityInputHandler<MixedRealityPose>

The important thing to realize is that this behaviour needs to handle global events. This implicates two things, first, you will have to register global handlers

protected override void RegisterHandlers()
{
    CoreServices.InputSystem?.RegisterHandler<IMixedRealityInputHandler>(this);
    CoreServices.InputSystem?.RegisterHandler<IMixedRealityInputHandler<Vector2>>(this);
    CoreServices.InputSystem?.RegisterHandler<IMixedRealityInputHandler<float>>(this);
    CoreServices.InputSystem?.RegisterHandler<IMixedRealityInputHandler<MixedRealityPose>>(this);
};

(and unregister them of course in UnregisterHandlers)

but second, if you use this in Unity, uncheck the "Is Focus Required" checkbox

This will ensure the global handlers being registered properly and being intercepted by this controller.

Discriminating between events of same type

It might not be immediately clear, but the only way I have been able to determine what exact event I get is to check it's MixedRealityInputAction.Description property. In the code you will seen things like

var eventName = eventData.MixedRealityInputAction.Description.ToLower();
if (eventName == "touchpad position")

In fact, you will see that the names of the event displayers in Scene hierachy are bascially the names of the events without spaces. I simply find them by name

After I simply load them in a dictionary in Start by looking for children in the "Events" object

foreach (var controller in _eventDisplayParent.GetComponentsInChildren<SingleShotController>())
{
    _eventDisplayers.Add(controller.gameObject.name.ToLower(), controller);
}

I simply find the back by looking in that dictionary and activating the SingleShot Controller. This class is part of a prefab that I used and explained in an earlier post.

private void ShowEvent(string eventName)
{
    var controller = GetControllerForEvent(eventName);
    if (controller != null)
    {
        controller.ShowActivated();
    }
}
private SingleShotController GetControllerForEvent(string controllerEvent)
{
    return _eventDisplayers[controllerEvent.ToLower().Replace(" ", "")];
}

I must say I feel a bit awkward about having to use strings to determine events by name. I guess it's inevitable if you want to be able to support multiple platforms and be able to add and modify events without actually having to change code and introduce types. This flexibility is what the MRTK2 intends to support, but it still feels weird.

Combining events

In the Immersive headset version of Walk the World you can zoom in or out by pressing on top of at the bottom at of the touch pad. But as we have seem, it's not even possible to detect where the user has pressed, only that he has pressed. But we can detect where he last touched, which most likely is at or very near where he last touched. How you can combine these the touch and press events to deserve and effect like I just described, is showed in the relevant pieces of the demo project code that copied below:

Vector2 _lastpressPosition;

public void OnInputChanged(InputEventData<Vector2> eventData)
{
    var eventName = eventData.MixedRealityInputAction.Description.ToLower();
    if (eventName == "touchpad position")
    {
        _lastpressPosition = eventData.InputData;
    }
}

public void OnInputDown(InputEventData eventData)
{
    var eventName = eventData.MixedRealityInputAction.Description.ToLower();
    if (eventName == "touchpad press")
    {
        // Limit event capture to only when more or less the top or bottom 
        // of the touch pad is pressed
        if (_lastpressPosition.y < -0.7 || _lastpressPosition.y > 0.7)
        {
            ShowEvent(eventName);
        }
    }
}

First, the touchpad position event keeps the last position into a member variable, then when the touchpad is pressed, we check where it last was touched. The event is only fired when the front 30% or back 30% was last touched before it was pressed. If you press the sides (or actually, touch the side before you press) nothing happens.

Conclusion

Interacting with the controller has changed quite a bit since ye olde days of the HoloToolkit, but it still is pretty much doable and usable if you follow the rules and patterns above. I still find it odd I have to determine what event is fired by checking it's description, but I may just be missing something. However, this method works for me, at least in my application.

Also, I am a bit puzzled by the almost-the-same-but-not-quite-so events around trigger and touchpad. No doubt some serious considerations have been made implementing it like this, but not having been around while that happens, it leaves confused about the why.

Finally, bear in mind you usually don't have to trap Select manually, and neither is the Thumbstick ('Teleport Direction') usually very interesting as those events are handled by the environment by default - the only reason I showed they them here was to demonstrate you could actually intercept them.

Demo project, as always, here on GitHub.

Migrating to MRKT2 - multi-device behaviour switching and scaling

noreply@blogger.com (Joost van Schaik) — Mon, 13 Apr 2020 10:31:00 +0000

Intro

The MRTK2 allows for development for both HoloLens 1, HoloLens 2 and Windows Mixed Reality immersive headsets with nearly identical code. And a growing number of other platforms, but the focus is now understandably on HoloLens 2. Yet, if you want to make apps with a broad range, you might as well use the capabilities the toolkit offers to run one app on all platforms.

Rule-of-thumb device observations

On HoloLens 2, you typically want interactive stuff to be close by and relatively small, so you can leverage the touch functionality
On HoloLens 1 interactive stuff needs to be further away since the only control option you basically have is the air tap. But because it's further away, it needs to be bigger
On Windows Mixed Reality immersive headsets you want it also further away but even bigger still, as I have observed things seems to appear smaller on an immersive headset compared to HoloLens, and lower resolution headsets makes things harder to see things like small print compared to HoloLenses.

Basically this boils down to scaling and distance. Scaling usually is pretty simple to fix, but distance behavior is a bit more difficult, especially since the MRTK2 contains so much awesome behaviours for keeping for instance a menu in view, but it does not support different behavior for different devices.

I have come up with a rather unusual solution for this, and it works pretty well.

Meet the twins

I made two behaviours that work in tandem. The first one is pretty simple enough and is called EnvironmentScaler.

This simply scales the current game object to the value entered for the specific device type. Notice also there is a drop down that will enable you to select view how platform specific sizes will appear inside the Unity Editor.

The second one is a bit more odd. You see, for determining the right distance, I would like to use the standard Solver and RadialView combo. Of course I could have written a behaviour that changes the RadialView values based upon the detected platform. But then it would only have worked for RadialView. So I took a more radical and generic approach

As you can see, there is one Solver but no less that three RadialViews on the menu. They all have slightly different values for things like distance and Max view degrees. An if you start Play mode:

It simply destroys and removes the behaviors for the other platforms. Crude, but very effective. And no coding required. The only thing is - there is no way to distinguish those three RadialViews, so it's best to add them to your game object in the same order as they are listed in the EnvironmentSwitcher: for HoloLens 1, HoloLens 2 and WMR headsets.

The nuts and bolts

Both the switcher and the scaler have the same generic base class:

public abstract class EnvironmentHelperBase<T> : MonoBehaviour
{
    [SerializeField]
    private EditorEnvironmentType _editorEnvironmentType = EditorEnvironmentType.Hololens2;

    protected T GetPlatformValue(T hl1Value, T hl2Value, T wmrHeadsetValue)
    {
#if !UNITY_EDITOR
        if (CoreServices.CameraSystem.IsOpaque)
        {
            return wmrHeadsetValue;
        }

        var capabilityChecker = CoreServices.InputSystem as IMixedRealityCapabilityCheck;

        return capabilityChecker.CheckCapability(MixedRealityCapability.ArticulatedHand) ?
                hl2Value : hl1Value;
#else
        return GetTestPlatformValue(hl1Value, hl2Value, wmrHeadsetValue);
#endif
    }

    private T GetTestPlatformValue(T hl1Value, T hl2Value, T wmrHeadsetValue)
    {
        switch (_editorEnvironmentType)
        {
            case EditorEnvironmentType.Hololens2: return hl2Value;
            case EditorEnvironmentType.Hololens1: return hl1Value;
            default: return wmrHeadsetValue;
        }
    }
}

The GetPlatformValue method accepts three values - one for every platform supported - and returns the right one for the current platform based upon this simple rules:

If the headset is opaque, it's a WMR headset
If it's not opaque and it supports articulated hands, it's a HoloLens 2
Otherwise it's a HoloLens 1

And there's also the GetTestPlatformValue that returns a platform based upon what's selected in the _editorEnvironmentType field, that can be used for testing in the editor. I have noticed that the editor returns false for opaque and true for the articulated hand support, so by default the code acts like it's in running in a HoloLens 2. Hence my 'manual switch' in editorEnvironmentType so you can see what happens for the various devices inside your editor. For runtime code, whatever you selected in editorEnvironmentType in either behaviour is of no consequence.

EnvironmentScaler implementation

This is the very simple, as all the heavy lifting has already been done in the base class:

public class EnvironmentScaler : EnvironmentHelperBase<float>
{
    [SerializeField]
    private float _hl1Scale = 1.0f;

    [SerializeField]
    private float _hl2Scale = 0.7f;

    [SerializeField]
    private float _immersiveWmrScale = 1.8f;

    void Start()
    {
        gameObject.transform.localScale *= GetPlatformValue(_hl1Scale, _hl2Scale,
                                                            _immersiveWmrScale);
    }
}

Simply scale the object to the value selected by GetPlatformValue. Easy as cake.

EnvironmentSwitcher implementation

public class EnvironmentSwitcher : EnvironmentHelperBase
{
    [SerializeField]
    private MonoBehaviour _hl1Behaviour;

    [SerializeField]
    private MonoBehaviour _hl2Behaviour;

    [SerializeField]
    private MonoBehaviour _immersiveWmrBehaviour;

    void Start()
    {
        var selectedBehaviour = GetPlatformValue(_hl1Behaviour, _hl2Behaviour, 
                                                        _immersiveWmrBehaviour);
        foreach (var behaviour in new[] {_hl1Behaviour, _hl2Behaviour,
                                         _immersiveWmrBehaviour})
        {
            if (behaviour != selectedBehaviour)
            {
                Destroy(behaviour);
            }
        }
    }
}

Very much like the previous one, but now the values are not floats (for scale) but actual behaviors. It find the behaviour for the current device, the destroy all others.

The fun thing is - in this I used this specifically for three identical behaviours (that is, they are all RadialView behaviours) - one for every device. But it's just as easily possible to use three completely different behaviours, one for each device, and have the 'wrong' ones be rendered inoperative by this behaviour as well. This makes this approach very generically applicable.

Conclusion

Multi device strategy does not have to be complex. With these two behaviours you can make your app appear more or less the same on different devices, and still adhere to the device's unique capabilities.

Complete project, as always, here. Enjoy

Migrating to MRTK2 - using the non-native keyboard in touch scenarios

noreply@blogger.com (Joost van Schaik) — Wed, 01 Apr 2020 06:18:00 +0000

Prelude

With apologies for the uncharacteristic hiatus in my blog - last month I had a HoloLens 2 available for development and test purposes, and utilizing that opportunity to the max had a bit more priority than actually blogging about it. Then the world got hit head-on by the Corona madness and I had other things on my mind. Now, in self-isolation, hoping to avoid the virus (as no doubt most of you are right now), I have finally started to crank out the backed up blog posts I had chalked up 'for later' while I was converting apps for HoloLens 2.

Intro

In the Mixed Reality Toolkit 2 you can use the beautiful system keyboard for text input and that works amazingly well - in HoloLens 2. Since MRKT2 development prioritizes HL2 development, and for good reason, this is not surprising. But in Immersive Headsets it works not so very well, and in HoloLens 1 it has the same problem.
Now since MRTK2.3 there's a new keyboard available - although actually it's an old keyboard - the Keyboard prefab, that used to reside in HoloToolkit\UX\Prefabs, has been renamed to the NonNativeKeyboard prefab and now sits in MixedRealityToolkit.SDK\Experimental. It has a few advantages over the native keyboard:

It is a Unity object, not a native object, so you can control size, rotation and position just like any other Unity object
It has basically the same API and usage as the old keyboard, which makes it attractive to use in existing applications.
It has a built-in button for speech recognition
It gives a consistent look & feel for your apps.

It also has a few quite distinct disadvantages:

It does not support touch events for HoloLens 2
It does not take into account the differences in apparent size in WMR headsets and HoloLenses
It should act different in various environments, (like being close when in HL2, and further away and bigger in other cases) which it does not

Now this, my friends, can be mitigated with a pretty simple add-on behaviour that I created. It takes care of positioning, platform dependent scaling and distance - but above all, it adds touch to the nonnative keyboard in a very simple way.

How it works

The start is simple enough: just some settings for each platform:

public class KeyboardAdapter : MonoBehaviour
{
    [SerializeField]
    private float Hl1Distance = 1.0f;
    [SerializeField]
    private float Hl1Scale = 1.0f;
    
    [SerializeField]
    private float Hl2Distance = 0.3f;
    [SerializeField]
    private float Hl2Scale = 0.3f;

    [SerializeField]
    private float WmrHeadSetDistance = 0.6f;

    [SerializeField]
    private float WmrHeadSetScale = 0.6f;

    [SerializeField] 
    private AudioClip _clickSound;

    private AudioSource _clickSoundPlayer;
}

Basically a couple of settings. For every platform support (HoloLens 1, HoloLens 2 and Mixed Reality headsets there is a distance from the user it will appear, and an apparent scale it will have. Also, you can assign a click sound for when a key is hit.
Almost all the heavy lifting happens in Start:

private void Start()
{
    _clickSoundPlayer = gameObject.AddComponent<AudioSource>();
    _clickSoundPlayer.playOnAwake = false;
    _clickSoundPlayer.spatialize = true;
    _clickSoundPlayer.clip = _clickSound;
    var buttons = GetComponentsInChildren<Button>();
    foreach (var button in buttons)
    {
        var ni = button.gameObject.AddComponent<NearInteractionTouchableUnityUI>();
        ni.EventsToReceive = TouchableEventType.Pointer;
        button.onClick.AddListener(PlayClick);
    }
}

The first four lines simply add and initialize the sound that is played when you tap a button. The next ones do the real work: they find every Button object in the keyboard, add a NearInteractionTouchableUnityUI to it, set the events to receive to "pointer" and add an event listener to the Button - that basically only serves to play the sound
The keyboard is built of Unity UI components, and adding a NearInteractionTouchableUnityUI and setting the EventToReceive to Pointer is all the is necessary to make the button 'think' it's clicked when it's actually touched. And if you have set the ClickSound to an audio file in the editor, it plays a sound now when you tap our touch it, too.
Then we have these two properties who return the right value depending on the platform the app is running on:

private float Scale => GetPlatformValue(Hl1Scale, Hl2Scale, WmrHeadSetScale);
private float Distance => GetPlatformValue(Hl1Distance, Hl2Distance, WmrHeadSetDistance);

Which is done by this little method:

private float GetPlatformValue(float hl1Value, float hl2Value, float wmrHeadsetValue)
{
    if (CoreServices.CameraSystem.IsOpaque)
    {
        return wmrHeadsetValue;
    }

    var capabilityChecker = CoreServices.InputSystem as IMixedRealityCapabilityCheck;

    return capabilityChecker.CheckCapability(MixedRealityCapability.ArticulatedHand) ? 
            hl2Value : hl1Value;
}

If the headset is opaque, then it's a Windows Mixed Reality headset (or at least not a HoloLens), and otherwise we determine based upon the capability of tracking hands whether it's a HoloLens 1 or 2.
This is used to determine to show the keyboard on the desired place and at the desired scale.

public void ShowKeyboard()
{
    NonNativeKeyboard.Instance.PresentKeyboard();
    NonNativeKeyboard.Instance.RepositionKeyboard(CameraCache.Main.transform.position + 
                                                  CameraCache.Main.transform.forward * 
                                                  Distance, 0f);
    NonNativeKeyboard.Instance.gameObject.transform.localScale *= Scale;
}

And that is really all. Some creative use of components already in the MRTK2.

Usage

Just drop a NonNativeKeyboard prefab from the MRTK2 in your scene, and drop this behavior on it. You then only have to take care of two things. First, set Min and Max scale both to 1, otherwise this will interfere with the way the keyboard is scaled by this behavior:

And of course, you have to set some parameters for the behavior itself. I have chosen what I like to think are reasonable settings for every platform:

And of course the sound that appears when you tap or touch the keyboard

How it looks

On a HoloLens 2( in my app Walk the World for HoloLens 2) it looks like this:

It is pretty close, as you are supposed to be able to touch it
On HoloLens 1 it looks like this:

Pretty much the same - bigger, but further away and thus easier to control with the air tap. Since this is a 2D picture, the differences between HoloLens 1 and 2 are actually hard to spot. And on Mixed Reality it looks like this - it look smaller, but that's because in a headset everything looks smaller. It's apparent size is the same as in HoloLens 1.

And finally, and action movie of touch enabled non native keyboard on a HoloLens 2. You can actually touch the buttons and they respond with a button press sound, as you see

Why no solvers?

You might have noticed I did not use any MRTK2 solvers to keep the keyboard floating in view when you when you move your head, our keep a dynamical distance. I did initially, but I found out that a keyboard that actually moves is very annoying when you are trying to type, especially when using touch type. Then the keyboard is close, and a move is easily triggered when you want to do things like first type the A (left), then the P (totally right). So I decided just to let it appear right in front of the user, and keep it there. If you don't use it, it automatically disappears after a configurable timeout. This is built into the keyboard, that is not my doing.

Conclusion

It's quite remarkable how you can make 'old' components interplay with the new HoloLens 2 interaction possibilities using the new components and a bit of imagination. Also, the MRTK2 made making apps that run on both HoloLens 1, HoloLens 2 and Windows Mixed Reality headsets a lot easier. Although it's clear MRTK2 prioritizes HoloLens 2 above all - and that's logical, because there's where the action and the business is. I fear the venerable trailblazing HoloLens 1, will quickly lose mindshare and disappear from the radar when HoloLens 2 becomes more widely available. I am not quite sure what to do with mine, but time will tell.
As usual, you can find the code of the little project here. If you want to run the app and see the keyboard, just say "Open Keyboard".

Post scriptum

I put the part that makes the NonNativeKeyboard touchable in a pull request to the MRTK2 that was merged at April 14, and will be part of the MRTK 2.4.0 release.

HoloLens 2 - it's the interaction, stupid!

noreply@blogger.com (Joost van Schaik) — Wed, 12 Feb 2020 15:50:00 +0000

Intro

Tuesday, February 7, 2020 marked an important occasion to me. A HoloLens 2 arrived at my door. For the first time since March 2019, where I got to try a HoloLens 2 during the MVP Summit for a few minutes, I actually had my hands on a device. And what's more - I got a month to play, test and develop with it, courtesy of fellow MVP and Regional Director Philipp Bauknecht of MediaLesson GbmH, a real community hero, who has graciously provided me with this learning opportunity. I hope I will be someday be able to repay him this enormous favor.

Just having this device around and being able to test and develop with it, quite changed my views of it, what's actually important - and what makes it a game changer.

Display

First of all, I am going to bring your hopes down a little. To an extent, HoloLens 2 suffers a bit from what I would like to coin as "the Apollo 12 effect". The whole world followed Neil Armstrong, Buzz Aldrin and Michael Collins to the Moon and where glued to very bad black & white screen while the first two men took their steps to the Moon. But a lot less people watched Apollo 12. Successive flights got even less attention (bar Apollo 13, but that was not because they landed, but almost died). People had literally seen this before and were - I kid you not - complaining about footage of men on the Moon eating up precious TV time from the football games. Subsequent flights after 17 were cancelled. People are extremely well equipped at accepting 'magic' and then getting bored with it.

As far as display goes, HoloLens 2 shows you virtual objects in 3D space that can interact with reality. This, my friends, is exactly what HoloLens 1 did.

It does this a lot faster, the view is brighter, the holograms are a lot more stable, and the thing almost everyone harped on - the field of view - has been considerably increased. I can almost imagine Alex Kipman shouting "we gave you bloody magic and all you kept telling me was the view was not big enough - are you happy now???"

There are other things: the device is a lot more ergonomic, it feels lighter but actually isn't very much - it's just better balanced. Donning it as easy as cake, taking it off as well, and charging via USB-C is a godsend - no more fiddling with MicroUSB on a wobbly end. I have seen more than one HoloLens 1 with a damaged charging port.

That's all very fine and welcome. But that's not what I mean by game changing. If we stay in space terminology - HoloLens 1 was like we suddenly had a fusion rocket. It is awesome, but parts of it were messy.

HoloLens 2 has a warp drive.

Interaction, interaction, interaction

Everyone who has ever used HoloLens 1 - or better still, tried instructing a newbie user to use one - knows the challenge. You can select something by pointing your head to a Hologram, then perform an air tap. Just tap your finger and thumb together. Easy as cake. And yet I have witnessed people who for some reason could not perform this simple task successfully. Either they pointed the cursor not correctly, or they made gestures that were almost but not quite an air tap, made it to slow or too fast, contorted their hands in a way that apparently confused the device, or started to make up gestures - that of course did not work at all. Whatever. Most people got it, but between 10-20% just never could get it to work reliably, if at all. The HoloLens 1 came with a little clicker for those people, apparently a last-minute addition - that almost no-one ever used. It either lost its charge at an inconvenient time, or (in most cases) got completely lost at all, it being a small device that easily was forgotten or dropped somewhere.

HoloLens 2 does not come with a clicker, and that's for a reason. If you make a gesture that even remotely resembles an air tap, it registers it as such - with such ferocity and accuracy that if you have a large contact surface you might even get some inadvertent air taps in (I will have to look into that for my app Walk the World, for instance).

In addition, what everyone saw being demoed first by the amazing Julia Schwarz - the ability just to touch, grab and move Holograms works amazingly well. To such an extent that you can push buttons like they are real, grab, move and rotate things like they are real... everything with amazing accuracy. You can even have your hand visualized and then it looks like you have some computer-generated glove on your hands - it follows every little movement. The resulting interaction model is very natural. So natural that you actually at first expect haptic feedback when you push a 'button'. Maybe something for HoloLens 3 ;).

There are a few things you might want to explain when you instruct someone to use the device for the first time to speed things up - like that if you want to use the start button, that's on your wrist. No more bloom gestures – the Italians will appreciate that ;). You might want to explain how an air tap works, but it's likely people find that out by themselves as it is so easy now. Also, the device goes out of it's way to explain itself on first startup.The fact that it can not only track your hand but also recognizes all kinds of hand postures and gestures allows for much more detailed control , and my personal favorite is having menus popup when you hold one hand in a certain position. These hand palm menus can be very easily made, using no code at all, just using stuff that's included in the MRKT2 out of the box.

But wait, there's more

Voice commands, remember that? The thing that everyone used like crazy and then quickly came back from, as it did not always work in noisy environments, especially with a lot of talk around. And making an odd gesture in empty space and looking weird is one thing, but shouting repeatedly at a device makes you feel very awkward indeed. Whatever they did to it, it's now way more accurate and confident at recognizing speech. Even in a very loud room with people talking. Speech control is everywhere in HoloLens 2, and very easy to use reliably.

And then there's eye tracking. Remember you had to move your whole head to point the gaze cursor? It now tracks your eyes. I knows what your are looking at. I use this in AMS HoloATC to make an image of the actual airplane pop up when you look at the model. There are four (or five, depending on what you include) events that you can easily track. I also learned that on a real life device I make that happen way too fast and too nervously. Having a real device, I will be able to fix this in the near future.

Eye tracking also has some extra benefits - first of all, it allows the device to use Windows Hello login using iris recognition. Second, calibrating is a lot easier and faster. No longer do you have first close your one eye, then very very precisely move your finger in the right slot for a couple of times, and repeat that for the other eye - you now simply have to track a few holograms with your eyes, as they move though your view. And you really should do that - Microsoft pushed the envelope a lot further when it comes to display technology, so if you don't calibrate properly, there's a lot more chance of having a fuzzy view. Fortunately the device has a setting that automatically starts the calibration routine when it detects the user has changed (which it presumably does using the iris scan).

In conclusion

HoloLens 2 is an amazing device, with an amazing display technology - but it's the interaction model that makes it really special. This is what takes in over the top, makes it natural, simple to learn and easy to use. The hand/eye tracking removes the barrier of artificial gestures and make wandering around and interacting with Holograms at lot easier. This will make use of the device in business settings - especially industrial and manufacturing environments - a lot easier.

I love living in the future:)

Mixed Reality apps failing the WACK - revisited

noreply@blogger.com (Joost van Schaik) — Wed, 22 Jan 2020 06:57:00 +0000

Intro

Last September I wrote about various hoops you had to jump through to get a Mixed Reality app to appear in the store and not have it to fail the Windows Application Certification Toolkit (WACK). It's time for a little update, since part of what I wrote is no longer necessary, and some things have changed subtly.

The good news

I wrote how you manually had to mess with Package.AppManifest.xml, particularly how you had to remove the DirectX 10 dependency to make it (still) downloadable for HoloLens 1 devices. Provided you have been diligently installing the service updates that are issued for this device, by now your HoloLens 1 should have been updated to no longer reject apps that have this requirement. So no need to take this step anymore.

The bad news

The trick I wrote how to make sure some unsupported DLLs were not included does not work anymore. We have to resort to more drastic measures.

The problem

To recap: if you generate a C++ solution from the recommended Unity version (2018.4.x, LTS branch) you get the following errors in the WACK

HolographicAppRemoting.dll has failed the AppContainerCheck check.
PerceptionDevice.dll has failed the AppContainerCheck check.
UnityRemotingWMR.dll has failed the AppContainerCheck check.
The Supported API test will list 10 errors concerning UnityRemotingWMR calling unsupported APIs
The Debug configuration test will tell youUnity RemotingWMR is only built in debug mode
And if you try to build for x86 or ARM, the Package sanity test will tell you HolographicAppRemoting.dll, PerceptionDevice.dll and UnityRemotingWMR.dll are only available for x64.

I suggested editing the “Unity Data.vcxitems” file that is inside your store projects and change the contents of the DeploymentContent-tags for these files to false. That worked swell. Until, somewhere between Unity and Visual Studio versions upgrade, some brilliant person thought it a good idea to recreate that file whenever you change the architecture. So when you change the architecture from "x86" to "ARM" - something that happens automatically while creating multi-platform packages - you are greeted with this:

M previous solution does not work anymore, as the three offending DLLs are automatically added back to the file, and thus the WACK fails again. Some more brute force is being required.

The solution

If you expand the Il2CppOutputProject, you can simply see the offending DLL's. The solution is as simple as selecting and deleting them.

And then you can generate your packages, run the WACK again... and may get another failure:

This depends on the version of the WACK you have. As as you can read here, the Store has been updated to ignore this error. In other words, if you are down to this error, you are good to go for submission and (successful) certification in the Windows Store.

Mind you - as soon as you regenerate the solution from Unity, the project file may or may not be overwritten and you will need to repeat this. So - always always run the WACK before you submit - but then again, that's what you should do anyway.

Conclusion

The ever changing constellations of Unity and Visual Studio keep us throwing curveballs, but fortunately there's always been a way around. So far. Happy submitting!

Mixed Reality apps failing MS Store validation on "10.5.1 Personal information" - and how to fix it

noreply@blogger.com (Joost van Schaik) — Wed, 15 Jan 2020 13:33:00 +0000

Intro

After fellow MVP (and RD, although not fellow-) Philipp Bauknecht discovered a bug in my AMS HoloATC app while testing it on his HoloLens 2 (what lunatic runs HoloLenses with DE-DE settings anyway :P) I created a fix for that and submitted the fixed app to the Microsoft Store on Saturday January 12th. I assumed it would pass right though. It did not. Monday morning I got a failure message:

That was not what I hoped, and certainly not what I expected

Parsing the message

I will admit to being rather confused, because I have a privacy policy link. In the Store. But apparently something has changed. If you want all the details, you can find them in the new App Developer Agreement, but the text in the failure basically says it all:

"For in-product include the privacy policy URL under the settings section"

So I need to add a privacy policy in the app. Okay. But they are expecting it under the settings section. This is clearly written with a desktop app in mind. But I don't have a main screen, let alone I can put a hamburger menu somewhere and put a "Settings" entry in this.

Solution part 1

So I could clearly not obey the letter of the law, but I can try to act in spirit. So I ventured out to see if I could get a privacy policy in the app in a way that would be acceptable to the Store testers. All my apps have like a "help" function, that you can activate by saying "help" or "show help". I actually advertise that on start-up with a floating text that is shortly visible after starting up the app.

This help screen looked like this:

And now looks like this

I actually added a text and a quad behind it, just ahead of the quad that is the main backdrop.

And on that quad there a very simple and crude behavior

using Microsoft.MixedReality.Toolkit.Input;
using UnityEngine;
using UnityEngine.WSA;

public class BrowserActivator : MonoBehaviour, IMixedRealityPointerHandler
{
    [SerializeField]
    private string _urlToOpen;

    public void OnPointerDown(MixedRealityPointerEventData eventData)
    {
        Launcher.LaunchUri(_urlToOpen,false);
    }

    public void OnPointerDragged(MixedRealityPointerEventData eventData)
    {
    }

    public void OnPointerUp(MixedRealityPointerEventData eventData)
    {
    }

    public void OnPointerClicked(MixedRealityPointerEventData eventData)
    {
    }
}

that launches a browser window when you tap on the text. In the app it looks like this:

And if you tap the indicated text, the app will move to the background and you will see this:

Or whatever URL you put into _urlToOpen in the editor. I took the exact same URL as in the Store. I am pretty sure this won't win me any design awards - but that's not the goal here. Note: these images where created with a HoloLens 1, as I am still lacking a HoloLens 2 (if anyone from Microsoft reads this - yes, that's a hint ;) ).

Solution part 2

Since I have no settings section, I thought it prudent to tell the testers where they could find the link to the privacy policy. So under "Submission options" I explained the why, where and how like this:

"IMPORTANT NOTE: this app failed App Policies: 10.5.1 Personal Information. Specifically, it missed an in app privacy policy. I was suggested to add it under section "settings" but there is no sections "settings", in fact, there are not sections at all, since this is not running in a window.
I have added a privacy policy tappable text in the help screen. You can access that by saying "help" or "show help", after you have initially placed the airport. This opens a browser and shows the privacy policy."

Proof of the pudding

As you can see, following this procedure took me through Store validation successfully. In fact, it took me through it twice - as it's Atlanta twin app ATL HoloATC got certified within hours after submitting with this addition.

Conclusion

Resuming:

Have some way in your app to popup the privacy policy web page.
Explain in Submission options to the testers where they can find it.

We can discuss at length how useful things like this are, if and how this could be communicated better to developers, if there needs to be guidance first before this is implement for the nice market of Mixed Reality apps - that don't have a window nor a settings section - whatever, this works, and while there is no official guidance on how to deal with privacy policies in apps like this, this article show you a way forward. Presumably, for the testers it's also just simply a matter of being mandated to tick off a box required by the legal department. And this is apparently an acceptable way of getting the box ticked. Job done.

No code sample this time - I assume everyone making MR apps will be able to re-create the two lines of actual code in the behaviour. And possible come up with a more elegant solution. But that was not the point of this - the point was getting in the store without a legal blocker.

Migrating to MRKT2 - Interaction with irregular or complex objects

noreply@blogger.com (Joost van Schaik) — Sun, 24 Nov 2019 15:17:00 +0000

Intro

Interaction with objects that are not simple shapes like cubes, spheres, capsules etc. poses some challenges. The Mixed Reality Toolkit 2 offers some great components, but they all require a top level collider. Now consider this helicopter:

It consists of a lot of small objects. Default it does not even have a collider. You cannot add a Near Interaction Touchable on top of the object because it simply cannot find a collider. Now you can generate those on import, but that makes the object kind of heavy with regards to required processing power, and hooking all those colliders up up to their own Interaction Touchable is a lot of work.

There is a simpler way of doing that, fortunately. Actually, there are two variants of this, but I am going to show the one I think works the best (and is the most beautiful).

Adding a colliding 'catcher' to rule them all

First of all, we are going to add a surrounding object inside the model itself. I took a capsule, as this gives IMHO the most beautiful result

The result now is the helicopter is now almost completely covered in what looks like a giant suppository, which is definitely not what you want.

So by fiddling around I created this material. Note: it actual color is fully transparent black - basically 0,0,0,0. so that we can see the helicopter again.

But more importantly, it has a hover light override color of green with a light intensity of 0.4.

And now, if a cursor strikes the object, you get this what I think is rather pretty ghostly glow indicating this is your focused object.

Making it interactable

Making it actually interact with events is now pretty simple. Assuming your actual 'controller' behaviour needs to control the whole game object, it needs to sit on top - so it can do more things than just interaction (like moving the helicopter, for instance - which it does not do now). It looks like this:

using System;
using TMPro;
using UnityEngine;

public class InteractionResponder : MonoBehaviour
{
    [SerializeField]
    private TextMeshPro _text;

    private int _timesClicked;

    private int _timesTouched;

    private int _timesFocus;

    public void Click()
    {
        _timesClicked++;
        UpdateText();
    }
    
    public void TouchStart()
    {
        _timesTouched++;
        UpdateText();
    }

    public void OnFocus()
    {
        _timesFocus++;
        UpdateText();
    }

    private void UpdateText()
    {
        _text.text = string.Format("Clicked: {0}{1}Touched: {2}{1}Focused: {3}", 
            _timesClicked, Environment.NewLine, _timesTouched, _timesFocus);
    }
}

Super simple, basically only three event response methods that can be called - and it tries to display the text in a TextMeshPro object, that also sits in the HologramCollection, just like the Helicopter it self. As I said, the InteractionResponder behaviour will be sitting on the helicopter object itself:

Now we need go back to the SurroundingCapsule and add an Interactable and a Near Interaction Touchable Volume script to that. The latter is apparently new or something I missed: where an ordinary Near Interaction Touchable only takes a rectangular collider, the Volume script also takes a capsule:

Then, you will need to select "Select" for Input Actions, then drag the helicopter in the little box under OnClick like you usually do in this kind of event hookup, and select ÏnteractionResponder.Click.

Then you click "Add Event", select "InteractableOnTouchReceiver" and hook the On Touch event up to the InteractionResponder.TouchStart method (we will ignore the On Touch End event in this sample).

In a similar fashion, you will add another event, select "InteractableOnFocusReceiver" and hook that to the InteractionResponder.OnFocus event.

And if you have done everything right, and hit the play button in the editor, you will see this happening when you click, touch or focus:

Conclusion

With very little code and one extra component you can make an irregular and complex object like a helicopter have an easy to touch/focus/click target that also gives some subtle but very clear visual feedback about what is happening. And this is only a start - we might as well have it respond to touch by showing some direct feedback, or show that it knows it's focused, even without a cursor hitting it (eye gaze!). Life is going to get interesting and much more immersive once HoloLens 2 comes around!

Demo project can be found here.

Migrating to MRKT2 - using extension services for dependency injection

noreply@blogger.com (Joost van Schaik) — Mon, 18 Nov 2019 15:53:00 +0000

Intro

Coming from business development, you might get a little shock coming into Unity - traditionally, game developers are much more focused on making the outside pretty than the inside. Things like dependency injection are kind of unheard of or considered 'too heavy' for game development. But if you are still in the process of development, actually being able to access a consistent (mock) data service in stead of the real live data service might be a big advantage, especially when that data service is rate limited or expensive.

The Mixed Reality Toolkit 2 offers a great feature for that: extension services. And it's actually pretty easy to use, and I am going to show a simple sample. I have written about this in very early alpha stage almost a year ago, but it's now to a point that it's actually usable.

Setting the stage

Using Unity 2018.4.6f1, I created a simple project MKRT2DepInject using the 3D template, imported the MRKT2 and TextMeshPro. For the latter I usually take essential resources only.Then I and added the MRKT2 to the SampleScene in the project. For the default profile, I usually take the DefaultHololens2Profile. Also, don't forget to set the platform to UWP (File/Build settings)

Also - and this is important - import JSON.net from the Unity store.

Extension services

A service requires an interface, an implementing class, optionally an inspector, a profile, and a default profile asset. Now the latter three may sounds maybe a bit abstract but it actually boils down to this:

An inspector is something that can be used to show the runtime status of a service in the editor. It's basically a debugging tool. It's entirely optional and in most cases it's not necessary.
A profile is a class holding configuration info for a class. If you have been using the MRKT2 for a while, you have been using them all along - cloning profiles and changing settings.
a default service profile asset is basically a serialized version of a profile class.

This may seem like a lot of work, but there's actually a nice tool for generating the boiler plate for all that - although had to get in a few pull request myself to getting it to work as I assume was intended ;)

Creating an extension service

Select Mixed Reality Toolkit/Utilities/Create Extension Service. This will bring up this UI:

Name the service "DataService". You will notice the "Service" suffix is mandatory. Choose "Services" for namespace. Then click the "Next" button. This will show you the next stage.

Now I like to organize my stuff a little, so I tend to put things in folders. The scripts go in a scripts/services folder, the profile in profile. You can set this by dragging the folder from the assets. Notice also I have disabled the inspector:

Hit next, and on the next screen click not now because otherwise you will be editing the default profiles - effectively, you are modifying the default settings of the MRKT2. You can do this only after you have cloned the proper profiles.

You will also notice that although you specified the default asset should have been created in the Profiles folder, it is in fact created in the Services folder. Look I am going to need to make another pull request. Anyway. I moved the DefaultDataServiceProfile to profiles, and let it sit there

Registering the service

First, we clone the top profile.

Then we disable the profiler, because that's annoyingly in the way when you want to demo something

Then we select the Extensions tab, and clone the "DefaultMixedRealityRegisteredServiceProvidersProfile" (the creators of the MRTK2 seem to have taken a liking to rather verbose names, as you might have noticed) to MyMixedRealityRegisteredServiceProvidersProfile

Now you can actually click the "+ Register a new Service Provider" button and register the service

Then you have to click the Configuration Profile drop down, which unfortunately shows you all possible profiles, and you have to pick the one you need, which is DefaultDataServiceProfile, which is fortunately at the top of the list

The end result should look like this:

Now the configuration stuff is finally done, and we are going to add some code.

The data and the data set

My simple sample is going to read a json file from the web and show the contents in the text. Therefore we need a data file, and a class to deserialize it in.

The data file sits here, and the class in which in can be deserialized looks like this

using Newtonsoft.Json;

namespace Json
{
    public class DemoData
    {
        [JsonProperty("firstName")]
        public string FirstName { get; set; }

        [JsonProperty("lastName")]
        public string LastName { get; set; }
    }
}

Configuration profile

So to make the configuration profile actually configurable, the DataServiceProfile class needs to be changed. We actually need to make a property to store an URL in. So, we add a serializable field and a read only property. Like this:

using System;
using UnityEngine;
using Microsoft.MixedReality.Toolkit;

namespace Services
{
    [MixedRealityServiceProfile(typeof(IDataService))]
    [CreateAssetMenu(fileName = "DataServiceProfile", 
        menuName = "MixedRealityToolkit/DataService Configuration Profile")]
    public class DataServiceProfile : BaseMixedRealityProfile
    {
        [SerializeField]
        private string _dataUrl;

        public string DataUrl => _dataUrl;
    }
}

Added code in red/bold. If you go back to the inspector, you will see there is a Data Url field now added to the DataService profile.

So let's clone that default profile to SchaikwebProfile:

And enter for Data Url: https://www.schaikweb.net/demo/DemoData.json. Result:

You can now already see how you can quickly change from one configuration profile to another. You could actually clone the schaikwebprofile to another profile with different settings. Now it has only one property, but it can have a lot - and you can change from one setting to another just by selecting a new profile.

Implementing the actual service

The generated code for the service - a bit abbreviated - looks like this:

namespace Services
{
    [MixedRealityExtensionService(....
    public class DataService : BaseExtensionService, IDataService, 
      IMixedRealityExtensionService
    {
        private DataServiceProfile dataServiceProfile;

        public DataService(IMixedRealityServiceRegistrar registrar, ....) 
        {
            dataServiceProfile = (DataServiceProfile)profile;
        }

        public override void Initialize()
        {
            // Do service initialization here.
        }

        public override void Update()
        {
            // Do service updates here.
        }
    }
}

You can see the profile - the class holding the settings - is being fed into the constructor. Now we don't need Initialize and Update in this simple service, so we delete that and add this:

public async Task<IList<DemoData>> GetNames()
{
    using (var request = new HttpRequestMessage(HttpMethod.Post, 
                                                dataServiceProfile.DataUrl))
    {
        using (var client = new HttpClient())
        {
            var response = await client.SendAsync(request);
            response.EnsureSuccessStatusCode();
            var result = await response.Content.ReadAsStringAsync();
            return JsonConvert.DeserializeObject<IList<DemoData>>(result);
        }
    }
}

Notice feeding in the URL from the dataserviceProfile!

Of course, we need to add this method to the IDataService interface as well:

public interface IDataService : IMixedRealityExtensionService
{
    Task<IList<DemoData>> GetNames();
}

And now some action...

So I created this little MonoBehaviour that actually accesses and uses the service.

public class NamesReader : MonoBehaviour
{
    [SerializeField]
    private TextMeshPro _text;

    private IDataService _dataService;
    void Start()
    {
        _dataService = MixedRealityToolkit.Instance.GetService<IDataService>();
    }

    void Update()
    {
        if (Input.GetKeyDown(KeyCode.Alpha3))
        {
            LoadNames();

        }
        if (Input.GetKeyDown(KeyCode.Alpha4))
        {
            _text.text = "";
        }
    }

    private async Task LoadNames()
    {
        var names = await _dataService.GetNames();
        _text.text = string.Join(Environment.NewLine,
            names.Select(p => $"{p.FirstName} {p.LastName}"));
    }
}

You can see how it simply gets a reference to the service in the start method. If you run this in the editor and you press "3" it will try to load the values from the service, and show them in as TextMeshPro _text (pressing "4" clears it again). The extremely spectacular result looks like this:

Basically a direct dump from the data file on my website:

[
    {
        "firstName": "Scott",
        "lastName": "Guthrie"
    },
    {
        "firstName": "Alex",
        "lastName": "Kipman"
    },
    {
        "firstName": "Scott",
        "lastName": "Hanselman"
    }
]

Mocking service access

Now let's assume, for the moment, this data service is extremely expensive, slow or otherwise limited in access. Or you need to test certain edge cases but the data service does not always give them when you need them. In other words, you want to make a fake service - a mock service. This, now, is very simple.

So let's build a mocking service:

[MixedRealityExtensionService(....
public class MockDataService : BaseExtensionService, 
                               IDataService
{
    public MockDataService(IMixedRealityServiceRegistrar registrar, ....
    {
    }

    public async Task<IList<DemoData>> GetNames()
    {
        var data = new List<DemoData>
        {
            new DemoData {FirstName = "Joost", LastName = "van Schaik"},
            new DemoData {FirstName = "John", LastName = "Doe"},
            new DemoData {FirstName = "Kermit", LastName = "the Frog"},
        };
        await Task.Yield();
        return data;
    }
}

So we implement the same interface, but it does not take a DataServiceProfile configuration (although it perfectly could if I implemented the constructor). And now a second implementation version of the service appears in the drop down:

Sow you can quickly now change a single service from a production implementation to test implementation. The mock service will show this:

But what is even more cool is when you make a 'mock profile' from the RegisteredServiceProfile profile. For if you have like 20 services (and believe me, the number of services goes up pretty quickly) you can change from test to production by simply switching the profile. So I cloned the MyMixedRealityRegisteredServiceProvidersProfile itself to MockMixedRealityRegisteredServiceProvidersProfile and now, by simply switching profiles - you can change the whole extension service definition with one simple dropdown.

Conclusion

Extension services are a really powerful feature of the MRTK2, that can be used for central access of data services - typically stuff you would use Singletons for in ye olde HoloToolkit. But using service profiles also offers a quick and easy way to switch between real and mock implementations, brings an important part of enterprise level development into the traditional - ahem - more chaotic Unity development environment.

Demo project can be found here.

Migrating to MRKT2 - making a 'hand palm' menu

noreply@blogger.com (Joost van Schaik) — Sun, 17 Nov 2019 17:04:00 +0000

Intro

In all modesty I think I pretty much nailed making simple 'user interfaces' for HoloLens 1 - simple dialogs and stuff. Clients told me 'even a moron can understand how to operate this' so I think I did allright. But stuff floating in thin air wasn't always ideal. And now with HoloLens 2, we can actually let stick stuff to hands, in stead of just the gaze. So I decide to check if I could make a menu that stuck in the palm of your hand. Turned out I could.

And it's ridiculously easy to boot.

Creating a hand palm menu

This was actually just built from the UX components in the MRKT2.

These are literally only two "PressableButtonHoloLens2" prefabs floating about 2cm before a quad that has this material:

Then I added a Solver Handler to the HandMenu

I had to fiddle a bit with the settings, especially the additional offset - to determine where exactly the menu is going to appear in relation to the palm. How these parameters exactly work is a bit unclear with me, so I used the scientific method: I started to change the numbers till things happened the way I liked ;)

And finally I added a Hand Constraint Palm Up with these settings

And this is basically all you have to do to have the menu appear.

Some configuration settings

Although this works with the 'normal' simulated hand setting as well, it's best viewed with the 'flat' hand palm. To get this, starting with

The Mixed Reality Toolkit Profile (Cloned from DefaultHoloLens2ConfigurationProfile)
The Input System Profile
The Input Simulation Profile

Then you have to expand the "Input Simulation Service" section, scroll all the way down to "Hand Gesture Settings" and change the settings there as follows:

These are my default settings - but for this demo actually only the top setting is important (Default Hand Gesture to "Flat")

Testing the menu

I think it's safe to say that most of the people reading this - including me - don't have a HoloLens 2. Fortunately you can test this quite easily in the editor. To get to the point shown in the first picture in this post, simply do the following:

Start play mode
Move the mouse cursor inside the game window
Press the Space bar and keep it down- the right hand should now appear
While keeping the space bar pressed down, also press the left control key on your keyboard and keep it down
With your other hand, slowly move the mouse to the left. The hand should now start to rotate in stead of move
When you have rotated the hand just past the 90° angle, the menu should appear.

Making it actually do something

You might have noticed the buttons don't do anything at this stage. So I created a little helper behaviour that shows "Yes" or "No" depending on which button you press. It's not very sophisticated:

It simply follows the actual hand menu around
It has a public method "ShowYes" and "ShowNo" that can be hooked up to the buttons to show something when they are pressed.

It sits in a "ResponseHelper" game object and the hookup to the button is therefore simple, as the YesButton shows:

The actual working is displayed in this little video:

The trick to make the right hand stay while you operate the left hand is:

Press Y once the right hand is in the desired position and rotation
Release space bar and control key (right hand stays where it is, you are just longer controlling it)
Press the left shift key - the left hand should now appear
While keeping the left shift key pressed, you can move the left hand with the mouse. To move it forward and backward like in the movie, rotate the mouse wheel. This way you can actually make the index finger touch the buttons

Conclusion

Building a hand palm menu is really easy and basically requires no code. How this will work on a real HoloLens 2 - only time will tell. In the mean time you can find the demo project here.

Migrating to MRTK2 - setting up and understanding Eye Tracking

noreply@blogger.com (Joost van Schaik) — Thu, 26 Sep 2019 18:33:00 +0000

Intro

One of the exiting features HoloLens 2 brings us, is Eye Tracking. On HoloLens 1, you had to move your whole head to move the gaze cursor. and while that works well enough for a lot of applications and it seems most people pretty quickly got used to it, mother Nature has equipped us with roving eyes. HoloLens 2, when calibrated for Eye Tracking, can actually track what your are looking at, not merely where your head is pointed at.

Although there is a nice demo in the Mixed Reality Toolkit 2, it took me a while to find out how all the events actually work and need to be hooked up to get it to work consistently. So I made a little demo that works like this:

Events and tracking them

The little blue globe is the target, is equipped with and EyeTrackingTarget script from the MRTK2 that supports five events, which you can see going off as the red spheres turns green

LS: On Look At Start
WL: While Looking At Target
LA: On Look Away
DW: On Dwell
S: On Selected

The EyeTrackingTarget is configured as follows:

In the scene, the whole thing showing the images (the five little red-turning-green globes with labels) is one prefab containing 5 little spheres with a label above it - each a prefab on its own. Every sphere has a "Single Shot Controller" script that turns it's sphere green for 0.5 seconds when an event is called.

It's a super simple script, the interesting part is even shorter:

public void ShowActivated()
{
    _timeActivated = Time.time;
}

void Update()
{
    var desiredColor = Time.time - _timeActivated > _resetTime ? 
        _originalColor : _activatedColor;
    if (_material.color != desiredColor)
    {
        _material.color = desiredColor;
    }
}

When ShowActivated is called, the _timeActivated field is set to now. The Update loop then checks every 60th of a second whether it should set the color to red or green, depending on the fact if the latest call to ShowActivated is already half a second ago.

What happens when

The event names are pretty straightforward, and things happen more or less than you expect, although

What is actually happening:

When the user first looks at the eye tracked object, "On Look Start" is fired once
While the user keeps looking, "While Looking At Target" keeps being called. Thus, the green sphere stays green. The calling seems to at the same instant - or nearly the same instant - as the previous event
As soon as the user stops looking at the sphere, "On Look Away" is called and "While Looking At Target" is stopped being called
"On Dwell" is being called after the time defined in the "Dwell Time in Sec" slider has passed has the user is still looking at the object. I took the ridiculously user-unfriendly time of three seconds to make sure this event was easily distinguishable from the other events. Here's the thing though - it's being called once. That kind of confused me.
"On Selected" is being called when then object being looked at and you say "Select". This is one of the predefined commands in the default speech commands profile (DefaultMixedRealitySpeechCommandsProfile)

Setting up and configuring eye tracking in profiles

Coming from the default profile, you will need to configure at least profiles, and better still three.

First, you will need to clone the Default Toolkit profile itself. First thing I do, while still in the early phases, is disabling the diagnostics system as I don't want that profiler in my face the whole time:

Next, you will have to clone the Input System Profile and add a Windows Mixed Reality Eye Gaze Provider"

As you can see, this sits in namespace "Microsoft.MixedReality.Toolkit.WindowsMixedReality.Input"

And then finally, and optionally, if you want this to work in the editor too, you will have to configure the Input Simulation Service. You do that by cloning the default input system profile and check the "Simulate Eye Position" checkbox

One more thing: setting capabilities

You will notice now the Gaze cursor turning up in the editor so you might thing you are done. Well, almost. There's the small matter of capabilities. C++ or not, the result is still a UWP app, and Gaze Input is a capability that you need to ask consent for. This, unfortunately, is not yet implemented in Unity. So after you generated the C++ app, you will need to open it in Visual Studio, select the Package.appmanifest file, and select there the Gaze Input capability.

If you deploy the resulting solution to an emulator (or, if you are one of the lucky ones out there, an actual HoloLens 2) and it asks for your consent, you did it right.

Conclusion and final words

Setting up Eye Tracking is not that hard, but it takes a few steps. Mind you the MRTK2 comes with a few profiles that make settings things up easier - I just wrote down the steps from scratch. The demo project shows this in all it's glory ;) and allows you to play with it yourself without having to set it up. Notice there's hardly any code outside of the MRKT2 itself involved - there's only one custom script (my SingleShotController) and that's very simple.

By the way - in my own (so far single) HoloLens 2 app I only use the On While Looking event. This seems to be the most trustworthy. I previous iterations of the MRTK2 and/or the HoloLens emulator the other events did not go off reliably enough (IMHO) to use them for real. Of course, this may all be different now, and most likely is completely different (better) on a real device. We are still waiting for that.

On a final note – leaving the eye cursor visible can be confusing and/or annoying, or so I have been told. So under normal circumstances, it should be turned off – the object being looked should give some indication it is notices being looked at. I have found a way to do this myself, but that’s pretty complex, and just as I was about to blog about that, Julia Schwarz herself added a (better) sample to turn off pointers by code to the main MRTK2 repo.

Migrating to MRTK2–submitting a HoloLens 2 app to the Microsoft Store

noreply@blogger.com (Joost van Schaik) — Thu, 05 Sep 2019 18:58:00 +0000

Intro

On Monday September 2 version 4.0.19 of my first HoloLens Store app, AMS HoloATC, became available in the Microsoft Store. This version has been completely rebuilt using the Mixed Reality Toolkit 2, and includes some HoloLens 2–only functionality: you can actually touch the airplanes now, and using gaze tracking it will show you a picture of the actual aircraft, if available.

Hoops to jump

As you might recall from earlier posts, things have changed quite a bit when it comes to actually deploying apps on HoloLens. Now, we will need compiling and submitting a Unity-generated C++ solution to the store. Although the process looks very much like we used to do for Unity apps running on the .NET backend, there are three things you might run into:

Your WACK test will most likely fail
If you have submitted your app as a bundle before, make sure you submit it as a bundle again. A Unity generated C++ solution does not have this as a default setting
If you create (like me) an app that is supposed to run on Desktop (in immersive head sets), HoloLens 1 and HoloLens 2 you may find out your app cannot be downloaded by a HoloLens 1 anymore – or the HoloLens 2 emulator, for what matters

Fixing the WACK fail

Before you actually submit an app to the Store, you do the Windows Application Certification test first, to prevent embarrassing easy-to-prevent fails, right. (right?). And if you do so, you will see it fail. It will spout quite some errors at you.

The Windows security features test will complain about:

HolographicAppRemoting.dll has failed the AppContainerCheck check.
PerceptionDevice.dll has failed the AppContainerCheck check.
UnityRemotingWMR.dll has failed the AppContainerCheck check.

The Supported API test will list 10 errors concerning UnityRemotingWMR calling unsupported APIs
The Debug configuration test will tell youUnity RemotingWMR is only built in debug mode
And if you try to build for x86 or ARM, the Package sanity test will tell you HolographicAppRemoting.dll, PerceptionDevice.dll and UnityRemotingWMR.dll are only available for x64.

The solution is bit weird, but can be found in this Unity forum post, and involves manually hacking the “Unity Data.vcxitems” file that is inside your store projects. Open in it in a text editor, and search for “HolographicAppRemoting”. This will show this piece of XML:

<None Include="$(MSBuildThisFileDirectory)HolographicAppRemoting.dll">
  <DeploymentContent>true</DeploymentContent>
  <ExcludeFromResourceIndex>true</ExcludeFromResourceIndex>
</None>

Now simply change the value “true” inside the DeploymentContent to false:

<None Include="$(MSBuildThisFileDirectory)HolographicAppRemoting.dll">
  <DeploymentContent>false</DeploymentContent>
  <ExcludeFromResourceIndex>true</ExcludeFromResourceIndex>
</None>

Repeat this for PerceptionDevice and UnityRemotingWMR. Rebuild your app, generated packages again and presto, your WACK test will pass. That is literally all that’s needed to get rid of this multitude of errors.

Bundling your app

I don’t know exactly what changed, but all my HoloLens apps that I created with previous Unity versions were uploaded as bundles. To be honest, I never did pay much attention to it. But the default setting of the generated C++ setting is this:

which generates an appx per platform (in my case three: one for x64, one for x86 and one for ARM for, in the same order, WMR immersive headsets, HoloLens 1 and HoloLens 2). If you try to upload those files as updates to an app that was previously submitted as a bundle you will be greeted with:

And this can simply be fixed by changing the setting “Generate app bundle” from “If needed” to “Always”

Make your app (still) downloadable for HoloLens 1

To be honest, 4.0.19 was not the first HoloLens 2 enabled version I submitted. That was 4.0.17. It got certified – as one of the first if not the very first indie HoloLens 2 app. I was very happy about this – for about 25 seconds. And then I got a very unpleasant surprise: I could not download it anymore on a HoloLens 1. Sure enough, you could find it in the store, but the “Install” button was greyed out (well, light blue in stead of dark blue, but in any case unoperational). Curiously enough, a HoloLens that had it already installed did get the updated version though.

The reason for this behavior can be found down in this post in the Unity forums. Basically, Unity dropped support for anything lower than DirectX 10 and this is listed now in the app’s store manifest. Unfortunately, when the Store on the HoloLens 1 (and the HoloLens 2 emulator, incidentally) checks for DirectX 10, the device apparently reports “don’t have that” and the Store consequently blocks download.

Now I think this will be fixed shortly, but in the mean time here’s a workaround for if you need do to a submission right now:

First, open the Package.AppManifest.xml file in a text editor. Find these lines:

<TargetDeviceFamily Name="Windows.Desktop" MinVersion="10.0.16299.0" MaxVersionTested="10.0.18362.0" />
<TargetDeviceFamily Name="Windows.Holographic" MinVersion="10.0.16299.0" MaxVersionTested="10.0.18362.0" /

Comment out the second line. Then proceed to build a bundle, but for x64 only.

Go back to Package.AppManifest.xml, uncomment the first line, and re-activate the second. Now find the StoreManifest.xml file – open it in a text editor. It should look like this:

<?xml version="1.0" encoding="utf-8"?>
<StoreManifest xmlns="http://schemas.microsoft.com/appx/2015/StoreManifest">
    <Dependencies>
        <DirectXDependency Name="D3D11_HWFL_10_0" />
    </Dependencies>
</StoreManifest>

Simply remove the line <DirectXDependency Name="D3D11_HWFL_10_0" />

Now build a package for x86 and ARM. I am not sure if this is essential, but I made sure the x86/ARM bundle was had a one release number one point higher than the x64.

Now proceed to upload both bundles into a submission and set check boxes as needed. In my store submissions it looks like this:

Now as you can see version 4.0.17 still contains all platforms, but that is not necessary. But because the 4.0.19 has a higher version number, it will be offered first to HoloLens 1 and 2.

Anyway, now your app, once certified, should be downloadable for all devices. On x64 the DirectX 10 check will be still in place, for other devices it’s disabled.

Conclusion

It’s early days for HoloLens 2 (I built my app without having direct access to it) but I think it’s pretty cool to have an app armed and ready for it. It takes some fiddling around with xml files to get it right, but I am sure things will be better soon and these work arounds won’t be necessary anymore.

Enjoy building the next generation Mixed Reality apps!

Migrating to MRTK2–interacting with the Spatial Map

noreply@blogger.com (Joost van Schaik) — Tue, 13 Aug 2019 09:45:00 +0000

Intro

One of the HoloLens’ great features is the ability to interact with real physical objects. This allows apps to place holograms on or adjacent to real objects, enables occlusion (the ability to let holograms appear to be hidden because they disappear behind physical objects), etc. This is all done using the Spatial Map, a graphical representation of whatever the HoloLens has observed to be present in the physical reality. Interacting with the Spatial map used to be easy – and it actually still isn't that hard, it’s just that - as with most of the things in the MRTK2 - quite some cheese has been moved

This blog post handles a common and a not so common scenario for interacting with the Spatial Map:

Placing objects on the Spatial Map
Programmatically enabling and disabling/clearing the Spatial Map

I have included a demo project that allows you to place cylinders on the Spatial Map by air tapping - and you can turn the Spatial Map on and off using a floating button.

Placing objects on the Spatial Map, MRKT2 style

I wrote about this already in November 2017 in my article about finding the floor using a HoloLens. In MRTK2, that process is a bit much different. Create a raycast from the Camera along the camera viewing angle and try to hit the Spatial Map. For this, you need the Spatial Map Layer mask. In the HoloToolkit you could simply access.

SpatialMappingManager.Instance.LayerMask

to get to that layer mask. Finding that now is a wee bit more complicated. You see, first, you need to extract the configuration from the Spatial Awareness System service like this:

var spatialMappingConfig =
      CoreServices.SpatialAwarenessSystem.ConfigurationProfile as
     MixedRealitySpatialAwarenessMeshObserverProfile;

The spatial mapping config contains a property called ObserverConfigurations containing a list of of configurations (apparently taking provisions there might actually be more than one configuration). For each configuration you can take the profile from it's ObserverProfile property - that you have to cast to MixedRealitySpatialAwarenessMeshObserverProfile. Then you find the layer used by this config in it's MeshPhysicsLayer property.

I repeat - you can find the layer.

That is not the layer mask. It took me quite some time debugging to find out what was going on here - because if you feed that layer number into the raycast, it won't 'see' the Spatial Map. I have no idea why this was changed. Anyway, to get the layer mask, as required by raycast methods, you have to bit shift the actual layer number, like this

1 << observerProfile.MeshPhysicsLayer

So what used to be a single property, now requires this method:

private static int GetSpatialMeshMask()
{
    if (_meshPhysicsLayer == 0)
    {
        var spatialMappingConfig = 
          CoreServices.SpatialAwarenessSystem.ConfigurationProfile as
            MixedRealitySpatialAwarenessSystemProfile;
        if (spatialMappingConfig != null)
        {
            foreach (var config in spatialMappingConfig.ObserverConfigurations)
            {
                var observerProfile = config.ObserverProfile
                    as MixedRealitySpatialAwarenessMeshObserverProfile;
                if (observerProfile != null)
                {
                    _meshPhysicsLayer |= (1 << observerProfile.MeshPhysicsLayer);
                }
            }
        }
    }

    return _meshPhysicsLayer;
}

private static int _meshPhysicsLayer = 0;

And I added a static backing variable to speed up this process, otherwise this whole loop will be run 60 times a second in my TapToPlaceController, as well as every time you air tap to place a cylinder.

The method to find a point on the Spatial Map simply is then simply this:

public static Vector3? GetPositionOnSpatialMap(float maxDistance = 2)
{
    RaycastHit hitInfo;
    var transform = CameraCache.Main.transform;
    var headRay = new Ray(transform.position, transform.forward);
    if (Physics.Raycast(headRay, out hitInfo, maxDistance, GetSpatialMeshMask()))
    {
        return hitInfo.point;
    }
    return null;
}

This sits in the updated LookingDirectionHelpers class. In the demo project you can see how it is actually used.

In the TapToPlaceController, the Update method will flip the text from “Please look at the spatial map max 2m ahead of you" to "Tap to select a location" when the gaze strikes the Spatial Map (and the Spatial Map ONLY, not another hologram).

protected override void Update()
{
    _instructionTextMesh.text = 
         LookingDirectionHelpers.GetPositionOnSpatialMap(_maxDistance) != null ? 
         "Tap to select a location" : _lookAtSurfaceText;

}

If you then air tap, it will place a squatted cylinder on the spatial map at the place you are looking to. This is done in the OnPointerDown method - using the same call to LookingDirectionHelpers.GetPositionOnSpatialMap to get a point to place the cylinder.

You will notice a floating cube as well. You can't place a cylinder on the cube - it only finds the Spatial Map. Demonstrating that you can't place a cylinder on it, is the cube's sole purpose ;). What might happen is that you place a cylinder behind the cube on the Spatial Map, if your opposite wall is closer than 2 meters. It requires additional logic to handle that situation, but that is beyond the scope of this blog post.

Starting, stopping and clearing the Spatial map

For some apps, most notably my AMS HoloATC app, the Spatial Map is used to help getting an initial place to put an object but then it needs to go away, as to not get the view blocked by occlusion. Making the Spatial Map transparent sometimes helps, but then still the walls get in the way of selecting objects as they block the gaze and other cursors. Long story short – it is sometimes desirable to be able to turn the Spatial map on and off. And this is actually pretty simple:

public void ToggleSpatialMap()
{    
     if( CoreServices.SpatialAwarenessSystem != null)
     {
         if( IsObserverRunning )
         {
             CoreServices.SpatialAwarenessSystem.SuspendObservers();
             CoreServices.SpatialAwarenessSystem.ClearObservations();
         }
         else
         {
             CoreServices.SpatialAwarenessSystem.ResumeObservers();
         }
     }
}

Note that “ClearObservations” is necessary, as merely calling Suspend only stops the updating of the Spatial Map – the graphic representation still stays active. This was actually added after feedback from yours truly ;)

As to checking whether or not the observer is / observers are actually running I have devised this little trick

private bool IsObserverRunning
{
     get
     {
         var providers = 
           ((IMixedRealityDataProviderAccess)CoreServices.SpatialAwarenessSystem)
             .GetDataProviders<IMixedRealitySpatialAwarenessObserver>();
         return providers.FirstOrDefault()?.IsRunning == true;
     }
}

I check if there’s an observer and assume that if the first one is running, so is probably the rest. Although in practice, on a HoloLens, there will be only one observer running anyway.

You can activate and de-activate the Spatial Map by pressing the floating button, where the SpatialMapToggler behaviour is attached to.

Conclusion

If you run and deploy the demo project you will find a button floating before you (in the direction that you looked when the app started) that you can use to toggle the Spatial Map, and to the right a little cube. In addition, a text floating in your vision instructs you either to look at the spatial map or air tap when you actually do – and then a cylinder will appear. Like this in this little video:

Fixing error Failed to locate “CL.exe” or MSB8020 when deploying IL2CPP solution

noreply@blogger.com (Joost van Schaik) — Tue, 30 Jul 2019 13:55:00 +0000

Symptom

You have created a Unity project to create an app using MRTK2, and you want to use the new IL2CPP backend. You open the solution in Visual Studio 2019, you try to deploy it by using Build/Deploy and all the way at the end the compiler complains about “CL.exe” missing.

Alternatively, you might get the slightly more verbose error:

error MSB8020: The build tools for Visual Studio 2017 (Platform Toolset = 'v141') cannot be found. To build using the v141 build tools, please install Visual Studio 2017 build tools. Alternatively, you may upgrade to the current Visual Studio tools by selecting the Project menu or right-click the solution, and then selecting "Retarget solution".

Cause

You have most likely used the the recommended Unity version (2018.4.2f1) to create the project. This version – the name gives it away – was released before Visual Studio 2019, and therefore assumes the presence of Visual Studio 2017 and it’s accompanying C++ tools set, ‘V141’. So Unity generated a C++ solution referencing that tool set.

But now it’s 2019, you have kissed Visual Studio 2017 goodbye, installed Visual Studio 2019. And that comes with tool set V142.

Solution

Either you install V141 using the Visual Studio Installer, or you tell the generated solution to use V142. I personally prefer the last one, because newer is always better right ;)

Simply right-click the project in the solution that has “(Universal Windows)” behind it’s name, select properties, tab general and then the problem is already pretty evident:

Simply select Visual Studio 2019 (142) for Project Toolset and you are good to go. This setting will stay as long as you don’t delete the generated project – Unity will simply change what needs to be changed, and leave as much as it can (to speed up the generation process).

Conclusion

Simple fix, but can be hard to find. Hence a simple blog about it

Minimal required software for MRTK2 development for HoloLens 2 and Immersive headsets

noreply@blogger.com (Joost van Schaik) — Mon, 29 Jul 2019 17:20:00 +0000

Intro

A short one this time – and codeless to. You see, next Saturday I will be giving an workshop for MixUG Netherlands about development with the Mixed Reality Toolkit 2 for Immersive headsets, together with my colleague, partner in crime and fellow MVP Alexander Meijers. One of the things that came up preparing for this workshop was what you would actually need to develop with the Mixed Reality Toolkit 2. Since ye olden days of the HoloToolkit, quite a few things have changed – Unity, the minimal OS version, and there’s even a new version of Visual Studio. So I set out to complete a minimal shopping list with a few optional items. Fortunately, our friends over at Microsoft Azure make it quite simple to spin up a totally pristine machine so you don’t run into the typical developer machine issues – multiple versions of Visual Studio with different workloads and a myriad of Unity versions – which makes it hard to tell sometimes what is required for what app.

OS version

Easy one. Windows 10, 1809 or (recommended) 1903. Everything I tested, I tested on Windows 10 Pro

Visual Studio

You will need Visual Studio 2019 community edition. 2017 will work too, but is much slower. Download Visual Studio 2019 community from this link and choose the following work loads:

UWP development with optional components USB connectivity and C++ (V142) UWP tools checked
Game development with Unity with the optional component 2018.3 64-bit editor unchecked

In images:

Make sure you install Visual Studio before Unity.

Offline installer

A fun trick – if you want to make an offline installer for the community edition for these particular workloads, open a command prompt after downloading the installer, and type (on one line):

vs_community.exe --layout c:\vsinstaller
--add Microsoft.VisualStudio.Workload.ManagedGame
--add Microsoft.VisualStudio.Workload.Universal
--add Microsoft.VisualStudio.Component.Windows10SDK.IpOverUsb
--add Microsoft.VisualStudio.ComponentGroup.UWP.VC --lang en-US

In c:\vsinstaller you will then find a complete install ‘layout’ for all the necessary components. Might be useful if you want to prepare multiple computers.

Unity

2018.4.2f1, taken from ProjectSettings/ProjectVersion.txt in the mrtk_development branch. This particular version can be downloaded directly from this link.

Choose as minimal components

Unity 2018.4.2f1
UWP Build Support

Mind you – this sets you op for HoloLens 2 and Windows Mixed Reality Immersive headsets only.

Optional – HoloLens 2 emulator

I have already written extensively about it. You can get it here. Be aware that it requires Hyper-V being installed. If you have installed Windows 10 1903, it will run right away. On 1809 you will need some trickery.

Conclusion

It’s not that hard to get up and running for MRTK2 development for HoloLens 2 and Windows Mixed Reality Immersive headsets. And now you have a nice complete ‘shopping list’ for when you want to prepare your PC.

Migrating to MRTK2–manipulating holograms by grabbing

noreply@blogger.com (Joost van Schaik) — Sun, 14 Jul 2019 18:39:00 +0000

Intro

To be honest, the title of this blog post is a bit weird, because in Mixed Reality Toolkit 1 the concept of grabbing was unknown, as HoloLens 1 does not support this kind of gestures. But nevertheless, as I am on this quest of documenting all the gems I discover while migrating an existing app to Mixed Reality Toolkit 2, this is one of the things I came across so I am shoehorning it in this blog post series – the 8th installment of it already. And the fun thing about this one if that although there is a demo project available, I am going to write no code at all. The whole concept of manipulation by grabbing can be done by simply dragging MRTK2 components on top of a game object.

'Far manipulation'

This is really extremely simple. If I want to make a cube draggable in the 'classic' sense - that is, point a cursor to it, pinch and move my hand, and then the cube follows, all you have to do is add a ManipulationHandler to the cube, with default settings:

And then you simply point the 'hand ray' to it, pinch and move:

But as you could see, I can only drag it. I can't move it anymore - or rotate - as my hand comes closer, like at the end of the movie. I fact, I can't do anything anymore.

Allow grabbing and moving

For that, we will need to add another script: Near Interaction Grabbable.

And now, if the hand comes close to the cube, you can do all kinds of crazy stuff with it

Some settings to consider

If you don't want to allow 'far manipulation' (the first type) but only want to allow the manipulation by grapping, you can uncheck the "Allow Far Manipulation" on the ManipulationHandler.
If you want to see where the actual grab connection point is, check the "Show Tether When Manipulating" checkbox on Near Interaction Grabbable. This will look like this:

I bet there are more settings to consider, but I haven't tried those yet (or felt the need to do so).

Conclusion

The code of this completely code-less sample can be found here. I can't wait to add code like this to real-world HoloLens 2 projects. But alas, we still need to wait for the device :)

Migrating to MRTK2– handling tap, touch and focus ‘manually’ (in code)

noreply@blogger.com (Joost van Schaik) — Tue, 09 Jul 2019 07:30:00 +0000

Wait a minute – you did handle tap before, right?

Indeed, dear reader, I did. But I also had signed up for a MixUG session on Wednesday July 3. And while making demos for that I learned some other ways to handle interaction. Once again it shows that the best way to learn things is to try to teach them – because the need to explain things induces the need to actually obtain a deeper knowledge.

Ye olde way

In the MRTK 1, it was thus:

Handle tap – implement IInputClickHandler
Handle drag – implement IManipulationHandler
Handle focus – implement
Handle touch – forget it. ;)

The new way

In the MRTK 2 it is now

Handle tap – implement IMixedRealityPointerHandler
Handle drag – see above
Handle focus – implement IMixedRealityFocusHandler
Handle touch – IMixedRealityTouchHandler

Now I am going to ignore drag for this tutorial, and concentrate me on tap, focus and touch

IMixedRealityPointerHandler

This requires you to implement four methods:

OnPointerDown
OnPointerDragged
OnPointerUp
OnPointerClicked

OnPointerClicked basically intercept a tap or an air tap, and will work as such as you deploy the demo project to a HoloLens 1. After being bitten by people tapping just a tiny bit to slow and therefore not getting response, I tend to implement OnPointerDown rather than OnPointerClicked to capture a 'tap' event, but that's a matter of preference.

IMixedRealityFocusHandler

You will need to implement:

OnFocusEnter
OnFocusExit

The method names are the same as in MRKT1, only the signatures not - you now get a parameter of type FocusEventData which does give you some more information - by what the object was focused (we have multiple ways of doing that now!), what the previous focused object was, and what the new focused object is.

IMixedRealityTouchHandler

This requires you to implement

OnTouchStarted
OnTouchCompleted
OnTouchUpdated

But there is a twist to that. As we will soon see.

Show-off

To show off how it all works, I have created a little demo project. You can run it either in the emulator or the Unity editor (or a HoloLens 2, if you are in the HoloLens team and part some very few selected parties - I am unfortunately neither).

I have created a little script CodedInteractionResponder that shows off how this works. This script implements all the three interfaces I just wrote about. If you open the demo project in Unity show itself like this. All three cubes have the script attached to them.

The text above the cubes will show how much times a cube has been either focused, touched or clicked. I you press play and then the space bar, the right hand will appear (or use ctrl for the left hand). Moving the hand can be done by using the mouse - if you move the hand ray over the cubes it will trigger a focus event, if you tap the left mouse button you will trigger a tap, and if you move the hand towards the cube (using the WASD keys) it will trigger a touch event.

That is to say - you would expect that. But that is not always the case

What happens is this:

You can click or focus the green cube, but you cannot touch it. Nothing happens if you try.
You can click, focus or touch the red cube, but if you touch it, the number of times it's clicked is increasing. Not the number of touches.
Only the blue cube works as expected.

Yet they all have the CodedInteractionResponder. How does this compute?

NearInteractionTouchable

The best way to explain this, is an image showing the bottom half of all the three cubes

The green cube misses the NearInteractionTouchable. This script is necessary to have touch events being fired at all. So unlike IMixedRealityPointerHandler and IMixedRealityFocusHandler, where a mere implementation of the interface will trigger an event, a touch event - that is, methods in IMixedRealityTouchHandler being called - requires the addition of a NearInteractionTouchable script.

And NearInteractionTouchable has another trick up it's sleeve. Suppose you have a button - whether it's (air) tapped or actually touched/pressed, you want to activate the same code. If you change "Events to Receive" from it's default "Touch" to "Pointer" (as I did with the red cube) touching the cube will actually trigger a pointer event. This saves you a few lines of code. So basically NearInteractionTouchable can act as a kind of event router. And this is why the red cube never shows a touch event - but a click event instead.

Be aware NearInteractionTouchable needs a collider to work on. This collider needs to be on the same object the script is on. So if you make an empty game object as a hat stand for a a bunch of smaller game objects, make sure to manually add a collider that envelops all game objects, otherwise the 'touch' won't seem to work.

What, no code?

Yes, there is code, but it's pretty straightforward and if you want to have a look at CodeInteractionResponder, have a look in GitHub. It's actually so simple I felt it a little bit overdone to verbatim repeat parts in this blog post itself.

Migrating to MRTK2 - missing Singleton and 3DTextPrefab

noreply@blogger.com (Joost van Schaik) — Wed, 19 Jun 2019 15:03:00 +0000

Intro

If you are migrating from the HoloToolkit to Mixed Reality Toolkit 2 'cold turkey', as I am doing for my AMS HoloATC app, a lot of things break, as I already said in the first post of this series. For things that you can tap, you can simply change the implementing interface from IInputClickHandler or IManipulationHandler to a couple of other interface and change the signature a bit - that's not complex, only tedious, depending on how much you have used it.

What I found really hard was the removal of the Singleton class and the 3DTextFab. I used both quite extensively. The first one I needed for like data access classes as the concept of services that was introduced in the Mixed Reality Toolkit 2 was not yet available, and the other... well basically all my texts were 3DTextPrefabs so any kind of user feedback in text format was gone. Because so much breaks at the same time, it's very hard to step by step rebuilding your app to a working condition. Basically you have to change everything before something starts to work again. Since I was still learning by doing, there was no way to test if I was doing things more or less right. I got stuck, and took a radical approach.

Introducing HoloToolkitCompatiblityPack

I have created a little Unity Package that contains the things that made it hard for me to get a step-by-step migration to the MRTK2 and christened it the HoloToolkitCompatiblityPack. It contains minimal amount of scripts and meta files to have Singleton and 3DTextFab working inside an MRTK2 built app. As I will be migrating more apps, I will probably update the package with other classes that I need. You can find the package file here and the project here. If you take your existing HoloToolkit based app, yank out the HoloToolkit, replace it by the MRTK2, then import the HoloToolkitCompatiblityPack package, you at least have to fix a few less things to at least get your app to a minimal state of function again.

Caveat emptor

Yes, of course you can use the HoloToolkitCompatiblityPack in your production app, and ship a kind of Frankenbuild using both MRTK2 and this. Do let yourself be tempted to do that. See this package as a kind of scaffolding, or a temporary beam to hold up the roof while you are replacing a bearing wall. For 3DTextFab I tend to turn a blind eye, but please don't use Singleton again. Convert those classes into services one by one. Then remove the Singleton from the HoloToolkitCompatiblityPack to make sure everything works without. This is for migration purposes only.

Take the high road, not the low technical debt road.

Conclusion

Making this package helped me forward with the migration quite a lot. I hope it helps other too. I'd love to hear some feedback on this.

Migrating to MRTK2 - looking a bit closer at tapping, and trapping 'duplicate' events

noreply@blogger.com (Joost van Schaik) — Wed, 29 May 2019 11:30:00 +0000

Intro

In my previous post I wrote about how game objects can be made clickable (or 'tappable') using the Mixed Reality Toolkit 2, and how things changed from MRTK1. And in fact, when you deploy the app to a HoloLens 1, my demo actually works as intended. But then I noticed something odd in the editor, and made a variant of the app that went with the previous blog post to see how things work- or might work- in HoloLens 2.

Debugging ClickyThingy ye olde way

Like I wrote before, it's possible to debug the C# code of a running IL2CPP C++ app running on a HoloLens. To debug using breakpoints is a bit tricky when you are dealing with rapidly firing event - stopping through the debugger might actually have some influence on the order events play out. So I resorted to the good old "Console.WriteLine-style" of debugging, and added a floating text in the app that shows what's going on.

The ClickableThingy behaviour I made in the previous post then looks like this:

using Microsoft.MixedReality.Toolkit.Input;
using System;
using TMPro;
using UnityEngine;

public class ClickableThingyGlobal : BaseInputHandler, IMixedRealityInputHandler
{
    [SerializeField]
    private TextMeshPro _debugText;

    public void OnInputUp(InputEventData eventData)
    {
        GetComponent<MeshRenderer>().material.color = Color.white;
        AddDebugText("up", eventData);
    }

    public void OnInputDown(InputEventData eventData)
    {
        GetComponent<MeshRenderer>().material.color = Color.red;
        AddDebugText("down", eventData);
    }

    private void AddDebugText( string eventPrefix, InputEventData eventData)
    {
        if( _debugText == null)
        {
            return;
        }
        var description = eventData.MixedRealityInputAction.Description;
        _debugText.text += 
            $"{eventPrefix} {gameObject.name} : {description}{Environment.NewLine}";
    }
}

Now in the HoloLens 1, things are exactly like you expect. Air tapping the sphere activates Up and Down events exactly once for every tap (because the Cube gets every tap, even when you don't gaze at it - see my previous post for an explanation)

When you run the same code in the editor, though, you get a different result:

Tap versus Grip - and CustomPropertyDrawers

The interesting thing is, when you 'air tap' in the editor (using the space bar and the left mouse button), thumb and index finger of the simulated hand come together. This, now, is recognized as a tap followed by a grip, apparently.

So we need to filter the events coming in through OnInputUp and OnInputDown to respond to the actual events we want. This is where things get a little bit unusual - there is no enumeration of sorts that you can use to compare you actual event against. The available events are all in the configuration, so they are dynamically created.

The way to do some actual filtering is to add a property of type MixedRealityInputAction to your behaviour (I used _desiredInputAction). Then the MRTK2 automatically creates a drop down with possible to events to select from:

How does this magic work? Well, the MRTK2 contains a CustomPropertyDrawer called InputActionPropertyDrawer that automatically creates this drop down whenever you add a property of type MixedRealityInputAction to your behaviour. The values in this list are pulled from the configuration. This fits with the idea of the MRTK2 that everything must be configurable ad infinitum. Which is cool but sometimes it makes things confusing.

Anyway, you select the event you want to test for in the UI, in this case "Select":

And then you have to check if the event methods if the event matches your desired event.

if (eventData.MixedRealityInputAction != _desiredInputAction)
{
    return;
}

And then everything works as you expect: only the select event results in an action by the app.

How about HoloLens 2?

I could only test this in the emulator. The odd things is, even without the check on the input action, only the select action was fired, even when I pinched the hand using the control pane:

So I have no idea if this is actually necessary on a real live HoloLens 2, but my friends and fellow MVPs Stephen Hodgson and Simon 'Darkside' Jackson have both mentioned this kind of event type check as being necessary in a few on line conversations (although I then did not understand why). So I suppose it is :)

Conclusion

Common wisdom has it that the best thing about teaching is that you learn a lot yourself. This post is excellent proof of that wisdom. If you think this here old MVP is the end-all and know-all of this kind of stuff, think again. I knew of customer editors, but I literally just learned the concept of CustomPropertyDrawer while I was writing this post. I had no idea it existed, but I found it because I wanted to know how the heck the editor got all the possible MixedRealityInputAction from the configuration and show that in such a neat list. Took me quite some searching, actually - which is logical, if you don't know what exactly you are looking for ;).

I hope this benefits you as well. Demo project here (branch TapCloseLook).