Tero Teelahti

Using Protocol buffers v3 with ASP.NET core

2016-03-11T00:00:00+02:00

By default ASP.NET core API methods operate on JSON: they deserialize JSON from request/response body to model type and back. JSON is everywhere and works well… unless you have very high throughput requirements. There are many alternative formats, but Google’s serialization format Protocol Buffers is one of the most used. It has overgone some changes recently: the old proto2 syntax is replaced with proto3. The latter even has an official C# support.

The old proto2 used to have unofficial C# ports, and many ASP.NET MVC samples on the internet are based on those. I couldn’t find a working proto3 version, so I created my own.

To create custom input and output types for ASP.NET two interfaces need to be fullfilled: IInputFormatter and IOutputFormatter. The easiest way to do this is to inherit from InputFormatter and OutputFormatter base classes. Basically ASP.NET MVC tells the content type, content and desired target type, and then custom formatter needs to act on those.

Naturally this all needs to work for all possible types, otherwise the formatters would not be reusable. Proto3 has some strangeness in its APIs, like some of the useful constructors being internal. Luckily with some source code reading one can find the method that does the real work when the actual type is not known on compile time: IMessage.MergeFrom(). Working Input and output formatters are below:

// The input formatter reading request body and mapping it to given data object.
public class ProtobufInputFormatter : InputFormatter
{
    static MediaTypeHeaderValue protoMediaType = MediaTypeHeaderValue.Parse("application/x-protobuf");

    public override bool CanRead(InputFormatterContext context)
    {
        var request = context.HttpContext.Request;
        MediaTypeHeaderValue requestContentType = null;
        MediaTypeHeaderValue.TryParse(request.ContentType, out requestContentType);

        if (requestContentType == null)
        {
            return false;
        }

        return requestContentType.IsSubsetOf(protoMediaType);
    }

    public override Task<InputFormatterResult> ReadRequestBodyAsync(InputFormatterContext context)
    {
        try
        {
            var request = context.HttpContext.Request;
            var obj = (IMessage)Activator.CreateInstance(context.ModelType);
            obj.MergeFrom(request.Body);

            return InputFormatterResult.SuccessAsync(obj);
        }
        catch (Exception ex)
        {
            Console.WriteLine("Exception: " + ex);
            return InputFormatterResult.FailureAsync();
        }
    }
}

// The output object mapping returned object to Protobuf-serialized response body.
public class ProtobufOutputFormatter : OutputFormatter
{
    static MediaTypeHeaderValue protoMediaType = MediaTypeHeaderValue.Parse("application/x-protobuf");

    public override bool CanWriteResult(OutputFormatterCanWriteContext context)
    {
        if (context.Object == null || !context.ContentType.IsSubsetOf(protoMediaType))
        {
            return false;
        }

        // Check whether the given object is a proto-generated object
        return context.ObjectType.GetTypeInfo()
            .ImplementedInterfaces
            .Where(i => i.GetTypeInfo().IsGenericType)
            .Any(i => i.GetGenericTypeDefinition() == typeof(IMessage<>));
    }

    public override Task WriteResponseBodyAsync(OutputFormatterWriteContext context)
    {
        var response = context.HttpContext.Response;

        // Proto-encode
        var protoObj = context.Object as IMessage;
        var serialized = protoObj.ToByteArray();

        return response.Body.WriteAsync(serialized, 0, serialized.Length);
    }
}

Formatters need to be registered for ASP.NET MVC to use them. This can be done in the ConfigureServices method:

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();
    services.Configure<MvcOptions>(options => {
        options.InputFormatters.Add(new ProtobufInputFormatter());
        options.OutputFormatters.Add(new ProtobufOutputFormatter());
    });
}

And that’s it, now you can control the desired format with requests by using either application/json or application/x-protobuf as content and accept types. You can even mix and match: send in JSON but request protobuf back.

Don’t let techies design game user experience

2016-02-02T00:00:00+02:00

Today I had some 20 minutes spare time and I wanted to try games my co-workers had recommended for me.

Heartstone

First off was Heartstone - Heroes of warcraft. I had opened the game day before, and was in the middle of the tutorial story. Opening the game again got me slowly to the same tutorial spot… except when I clicked “Play” I got this:

Smells like an american engineer.

Three hour maintenance break is looong, but I can live with that; there might be a valid reason (like earthquake or tsunami or blown up data center). Instead, what irritated me, was that

the game did not tell me from the get-go that service is closed. Made me wait extra 30 s.
times are not in local format; it should not come as a surprise to anyone that AM and PM are not universal notations.
I’m required to know both current time, and my time difference to PST time.

The last one is the worst: it puts extra cognitive load on the player. Personally I never figured out when the maintenance break ends - I would have needed to open a timezone application or google timezones to figure that out.

Instead of the above message I would like to see maintenance break expressed as duration, e.g.

We are back online in 1 hour.

Vain glory

As I never got to play Heartstone, I took the next game in queue: Vain Glory. This game I had installed, but never opened. The first user experience I got when opening was this:

Some game jargon for you.

First of all: why does the load take 15 minutes when I’m on a 100 Mb network? Very few people are willing to wait this long for a game session. Normally 500 MB of content downloads fast, but this game is different.

Secondly, what are the random numbers on the lower right of the modal window: no units, no formatting, just random. There already is a very clear percentage and the progress bar, why add extra clutter?

And last: does “Asset File” mean something for normal people? In my opinion that is pure game development jargon and should not be shown to players. Most people don’t even know that there are real files loaded in the background.

After all the waiting - just when I thought I would get to play - the game disappointed me with second obligatory wait! And of course with another completely meaningless message: “Installing Data”:

Installing what?

I have the most powerful Android device ever made - the Nexus 6P - and this phase still took 10 minutes. And I have no idea what the game did for all that time; maybe it calculated prime numbers or did some crowdsourced computing for public good? If there is this heavy computing involved, why isn’t it pre-computed on the game backend?

As a player I would like to see a maximum of one wait with clear progress bar. It should take max minute or two on a fast network, and have an uplifting message made for humans. I’m not an english native, but something along the lines of

We are loading some high quality content for you and it will take a while, but it will be awesome!

Alternatively no waiting at all for first time experience, and downloads could continue in the background or when player advances in the game.

Result?

In the end my 20 minute of time was spent and I did not get to play for a second. Game studios did not get a dime from me, and my time was wasted. Loose-loose.

Using VS Team Services with go get

2016-01-19T00:00:00+02:00

I use VS Team Services for some of my repositories. I use the Git repository type, and most of the time everything works fine. Until today I reorganized some of my Go code to be more idiomatic, meaning that I leverage the native Go package system as it is designed: the only way to reference packages is by their full repository URL:

package main

using (
    "myorg.visualstudio.com/DefaultCollection/_git/MyRepo"
)

and you can also load packages with Go get command, like:

go get myorg.visualstudio.com/DefaultCollection/_git/MyRepo

I can (barely) live with the fact that VS Team Services adds the unnecessarily ugly “DefaultCollection” and “_git” in to the URL. But the real problem is that the above doesn’t work with go cmd, you just get very misguiding message:

package myorg.visualstudio.com/DefaultCollection/_git/MyRepo: unrecognized import path “myorg.visualstudio.com/DefaultCollection/_git/MyRepo”

Adding the verbose flag (-v) to the command gives one extra tidbit of information:

Parsing meta tags from https://myorg.visualstudio.com/DefaultCollection/_git/MyRepo?go-get=1 (status code 203) import “myorg.visualstudio.com/DefaultCollection/_git/MyRepo”: parsing myorg.visualstudio.com/DefaultCollection/_git/MyRepo: http: read on closed response body

My first guess was an authentication issue, and making a curl request for the address supported my guess as it was an authentication redirect. But Go uses Git internally, and git clone worked. I couldn’t find this issue anywhere related to VS Team Services (maybe gophers don’t use it?), but I found a same issue for Gitlab. It turned out, that neither Gitlab or VS Team Services adds automatically the .git ending to the URL, and therefore go get command never reaches the actual repository. This is fixed in Gitlab since, but issue remains in VS Team Services. The fix is to make the URL even uglier:

package main

using (
    "myorg.visualstudio.com/DefaultCollection/_git/MyRepo.git"
)

and the corresponding go get command:

go get myorg.visualstudio.com/DefaultCollection/_git/MyRepo.git

I hope this helps someone else to fix the problem quicker.

Azure Blob Storage CORS headers and ScriptCS

2015-05-29T00:00:00+03:00

One of Basware’s products I work for uses CDN to deliver content for end users. CDN provider is Edgecast, and primary & secondary origins are Azure Blob storage accounts. So far we have not needed any cross domain access to the CDN, but now a new feature required Javascript requests from our application domain to the CDN domain… and browsers naturally block this nowadays.

I knew right away that I need to set the Cross Origin Resource Sharing (CORS) headers to our origin servers, but setting this up was harder than it is supposed to be: Azure’s Powershell SDK does not have support to alter this value, and there is no UI to set it in the management portal. There is of course the management REST API you can use to do anything, but calling it with curl is hard due to the authentication scheme. Setting the DefaultServiceVersion property proved to be as complex before, so I knew what to expect.

I checked Github and there were a couple of projects that matched my problem. Still I found none of them immediately useful; this kind of tool that you use only once should have a very low barrier of entry: git clone and run. So I decided to try to create one myself. With some help from blog posts like Bill Wilders post on the subject I was able to create a working version in an hour. My tech stack for this is ScriptCS, as it supports Nuget references out of the box. I referenced the WindowsAzure.Storage package that had the methods I needed.

The end result is a tool that (given you have ScriptCS installed) you can just clone and run - ScriptCS takes care of the package restoration automatically. Tool supports dumping current values to console, adding CORS rules, and clearing rules. And the syntax is easy enough for anyone to use:

scriptcs addCors.csx -- [storageAccount] [storageAccountKey] [origins]

ScriptCS runs also on Mono, so you could even say this is cross platform. Not as good as Node or Go based solution would have been, but still good enough.

Naming is the hardest part… this tool turned out to be just “AzureCorsSetter”.

ASP.NET vNext presentation at Microsoft devdays 2014

2014-12-02T00:00:00+02:00

At the beginning of October Microsoft Finland held the yearly developer conference, this time with name Devdays. This year’s conference felt slightly smaller than previously.

As there is is lots of churn around the ASP.NET right now and I have a history with that framework, I proposed a presentation about ASP.NET vNext. Gladly it got accepted, and I had to dig deeper into what’s coming from the ASP.NET team. I played with the framework, watched every video and and read every blogpost about it from Scott Hanselman, David Fowler and others. I also prepared some demos, even a Linux demo which I had to scrap on last minute because I had only 45 minutes time to present. I tried to give the audience some guidelines how they can prepare to what’s coming, in order for the upgrade from current ASP.NET to be as easy as possible. It was nice to prepare and present, I hope it helped someone.

P.s. I waited for Microsoft to release the video recordings before posting this, but still after two months there is only a couple of videos available, and they are on strangely named Youtube channel, different from previous years. I do not know what happened as I have not seen any communication from MS about the recordings, and I have yet received no answer to my question. So I have to say this aspect of the conference was poorly executed this year. Also, there was no common feedback collection, which means that presenters did not get any proper feedback. For me it is important to see the video and get some feedback to be able to do better next year.

How-to deploy to Azure website deployment slot from TFS

2014-10-24T00:00:00+03:00

I changed some of my websites deployment to use different deployment slots on a single Azure web site instead of having different web sites for different staging areas. I deploy all my staging areas automatically from TFS (using the GitContinuousDeploymentTemplate.12.xaml process), each area from different Git branch. Works for my setup.

What did not work was deploying to other slots than the main slot. On Azure portal different slots have name and address scheme like mywebsite-slotname. I tried to use this name as deployment target:

Failing configuration.

…and got failed build with error like:

An attempted http request against URI https://management.core.windows.net/long-guid-string-here-53e1f2/services/webspaces/WestEuropewebspace/sites/mywebsite-slotname/publishxml returned an error: (404) Not Found.

So clearly mywebsite-slotname is not the correct scheme. And there is no documentation available, thus this blog post.

I went on and downloaded publishing profile for the site slot. It had double underscore naming mywebsite__slotname, but that did not work either. Nor did single underscore. What finally worked, was the name the old Azure portal used: mywebsite(slotname). This is how my build process deployment target looks now, and deployment to the slot works.

Working configuration.

I hope this gets better documented. Luckily one can create pull request for Azure documentation nowadays; I might document this myself.

How to change your Outlook default calendar

2014-08-06T00:00:00+03:00

Since Office 2013 was launched, I’ve had some problems with Outlook and account settings: I can set up all the accounts, but Outlook refuses to display my default Exchange calendar on the todo-pane, and instead shows an empty calendar from one of the other accounts that I use for email only. I like the todo-pane as its easy to glance whats coming with it. I’ve searched for resolution a couple of times, and most Microsoft community answers suggest to change the default data file on via Outlook > Account Settings:

Unfortunately changing that setting back and forth does not affect what’s shown on the todo-pane; I already had my Exchange account as default. Next suggestion I found was to change the same setting via Windows Control Panel’s mail section:

…but that was exactly the same setup than the previous one done via Outlook, nothing changed. Out of curiosity I decided to check what’s under the mail profiles section.

And from there I hit properties, and under properties you can find yet another data file selection dialog. Changing this data file under the profile finally changed my Outlook’s default calendar. Shouldn’t be this hard.

Yet another place to set the default Outlook data file.

What has pharmacy to do with programming?

2014-06-03T00:00:00+03:00

I visited my local pharmacy last Friday to get some prescription drug. I sat in front of the pharmacist, who gave me very thorough guidance about the usage. At the time of payment - before she handed me the boxes - she suddenly said:

Excuse me, but I need to get a signature from someone else, as I am a trainee.

Immediately another pharmacist came by, put her smart card into the computer, checked the boxes against the electronic prescription, and then signed the delivery. The whole process took maybe 20 seconds, and I’m certain trainee felt safer as her work is checked against mistakes. And myself - as a client - I felt like they care.

With the risk of sounding like Uncle Bob: this episode reminded me about how one should act if you really care about quality. This model of signing other people’s work is built into some software development processes, like the Github flow. If you are not using pull requests, you can still simulate this kind of apprenticeship model with strict use of code reviews before merging into your trunk. Remember, that this must not be for all your codebase; you can be very strict on important, core modules, and let others evolve freely.

The promise of OWIN starts to materialize

2014-05-09T00:00:00+03:00

OWIN stands for the “Open web interface for .NET”. Basically it is a reasonably simple specification that defines how data goes through the request pipeline, and how to attach to that pipeline. It is a specification for both the server and the application (middleware on OWIN’s terms) part.

When I first saw the project I was not that convinced, but since then lots of applications that rely on OWIN and not the old System.Web stack has emerged, and also there are some hosting components that implement the spec. SignalR is a good example of middleware, and Katana a host. For me the new project Helios is also very interesting and I hope the project succeeds, as that would make hosting ASP.NET WebApi very lightweight. And light is never bad.

So the ecosystem has matured, then what? What really made me to support OWIN is what it did to my codebase. In one of my pet projects I use claims based authentication and authorization with Windows Azure Access Control Services; That is a great project (although being replaced with Azure AD), but on the .NET MVC application side it has been a pain to integrate. The amount of web.config carbage it needs is huge, and I have broken it multiple times. Luckily Microsoft released some OWIN-based implementation of the server side components, and promised drastically simplified configuration model. You just register th middleware to OWIN, specify where to find the metadata, and give the application identifier:

public void ConfigureAuth(IAppBuilder app)
{
    app.UseCookieAuthentication(
        new CookieAuthenticationOptions
        {
            AuthenticationType =
               WsFederationAuthenticationDefaults.AuthenticationType
        });

    app.UseWsFederationAuthentication(new WsFederationAuthenticationOptions
        {
            MetadataAddress = "https://login.windows.net/some-azure-ad.onmicrosoft.com/federationmetadata/2007-06/federationmetadata.xml",
            Wtrealm = "http://myapps/somerealm",
        });
}

For me simplified configuration was not the only benefit: OWIN registration also gave me an option to register everything authentication related at the same place, which makes the code very readable. Before OWIN I had :

Various XML configurations for WS Federation registration
Custom ClaimsAuthenticationManager to do in-the-app claims transformation (look for database for some extra information and include that in claims)
Account controller to handle sign in and sign out actions
Handler to add user’s roles to all outgoing request for better usability (hide client side elements on single page application based on user’s role)

Now I have instead something along these lines:

public void ConfigureAuth(IAppBuilder app)
{
	app.SetDefaultSignInAsAuthenticationType(WsFederationAuthenticationDefaults.AuthenticationType);
	app.UseCookieAuthentication(
		 new CookieAuthenticationOptions
		 {
			 AuthenticationType = WsFederationAuthenticationDefaults.AuthenticationType,

			 // Make claims transformation to avoid using an external
			 // STS to map certain users to certain role claims
			 Provider = new CookieAuthenticationProvider
			 {
				 OnResponseSignIn = ctx =>
				 {
					 ctx.Identity = TransformClaims(ctx.Identity);
				 }
			 }
		 });

	app.UseWsFederationAuthentication(new WsFederationAuthenticationOptions
	{
		MetadataAddress = ConfigurationManager.AppSettings["medatata"];,
		Wtrealm = ConfigurationManager.AppSettings["realm"];
	});

	// Map sign in action
	app.Map("/signin", map =>
	{
		map.Run(async ctx =>
		{
			if (ctx.Authentication.User == null ||
				!ctx.Authentication.User.Identity.IsAuthenticated)
			{
				ctx.Response.StatusCode = 401;
			}
			else
			{
				ctx.Response.Redirect("/");
			}
		});
	});

	// Map signout action
	app.Map("/signout", map =>
	{
		map.Run(async ctx =>
		{
			ctx.Authentication.SignOut();
			ctx.Response.Redirect("/");
		});
	});
}

private static ClaimsIdentity TransformClaims(ClaimsIdentity identity)
{
	// ... add what ever claims needed based on your own data source
	return identity;
}

Kudos for Dominick Baier for his clear post on this subject that helped me forward with sign in and out actions.

WebApi + OWIN

At the same time I also moved Web API to OWIN based hosting, even though I actually run on IIS. Reasoning was the same than with claims auth: I find the configuration model better.

If you’re an ASP.NET developer, I suggest you start experimenting with the OWIN pipeline. It will pay out.

Yet another take on Internet Explorer compatibility mode

2014-02-04T00:00:00+02:00

I have touched this subject already twice: first I I blogged about forcing site rendering to be done with Internet Explorer’s latest engine. Then I faced a situation where separate intranet zone (bad idea, Microsoft!) fallbacks to compatibility mode and does not respect the IE=edge meta tag as internet zone web sites do.

Well… the saga isn’t over, as I faced this situation at work today. Again. I was going to put the IE=11 meta tag in place to force normal mode, but then I started to doubt how older IE’s (9, 10) would interpret the “11” tag. Short answer is: they don’t. Luckily you can specify many different modes, and the browser will pick the first one it supports. To apply this use either a meta tag in your page:

<meta http-equiv="X-UA-Compatible" content="IE=11; IE=10; IE=9; IE=8; IE=7; IE=edge" />

Or apply this IIS configuration to add the correct headers:

<system.webserver>
  <httpProtocol>
    <customHeaders>
      <!-- No need to expose the platform -->
      <remove 
        name="X-Powered-By" />
      <!-- Do not show IE compatibility view -->
      <remove 
        name="X-UA-Compatible"/>
      <add 
        name="X-UA-Compatible" 
        value="IE=11; IE=10; IE=9; IE=8; IE=7; IE=edge"/>
    </customHeaders>
  </httpProtocol>
</system.webserver>

Not nice, but works.