CodeBetter.Com

Slice of Life: Converting my Personal Site

Dru Sellers — Sun, 13 May 2012 16:32:36 +0000

Over the last few months I have been having a lot of fun converting my personal site (http://drusellers.com) from a pure static html site, to one that uses a back end server side language.

When I first started the main purpose was to learn a little more about python. So I started off using Flask, a micro framework for python. Flask is a lot like Sinatra for ruby and the comparison was a lot of fun because my good friend Ryan Rauh (RyRy) was writing a site in Sinatra at the time so we were able to compare and contrast the two approaches quite easily.

As I learned more about flask and all that it could do, I was impressed by how much I could do with so little. It served as a solid reminder that we should strive for simpler solutions.

In addition to learning python I also took the time to dive into LESS. Organizing the CSS content of applications has always been difficult for me. Finally, with LESS I had variables and the ability to nest selectors. Basically, all of the things that I have wanted CSS to do for a very long time. I started out using the runtime compilation modes of LESS, but as I learned more I eventually switched to running the watch mode on the compiler and then stripping that functionality out of the app. Right now, I really like this approach. It feels more like PROD vs DEV, but I still have the ability to iterate quickly. The real choice to use LESS, was that the popular CSS framework from Twitter called Bootstrap had just launched. By studying their approach and trying to deconstruct things I have learned a lot about CSS and its application to better CSS approaches.

But as with all things in the technical world, a change was on the horizon. That change was SASS/SCSS. RyRy read a post somewhere comparing SCSS and LESS and he came away with the impression that SCSS RULED, and LESS DROOLED. This was one of those moments, where I gave RyRy a stern look and then sat down to start converting my website from LESS to SCSS, because when the guy you ask all of your questions too, changes directions sometimes its just easier to go with the flow. Thankfully it wasn’t that hard, allowed me to revisit my CSS and improve things further.

Around the same time, I discovered semantic.gs, a SCSS friendly grid system, finally I can get rid of the BS classes like span-4 and what not that have been plaguing me since I first found blueprint.css and 960grid. Whoot! I have been able to further clean up the website’s CSS approach.

Well, of course the project can come to a close quite that simply. After improving my CSS and getting an understanding of python, I of course, had to completely rebuild it something new and shiny. Damn, my attention span.

So, yeah. Node.js the new shiny language/framework. My JavaScript skills have always suffered, so I rationalized changing my backend code to JavaScript as a means to improve my JS skillset. So, I converted my site again, learned a lot about JavaScript in the process, melted my brain trying to get things to work in this under powered language (under powered because I am used to my language providing much more to me).

So now my site is running Node.js – its a whole lot cleaner and easier to work on. I love how simple the underlying code is, and I have really learned a lot about some of the new languages and frameworks out there.

Azure SDK for node 0.5.4 is out! More secure and now with less angle brackets

Glenn Block — Sat, 12 May 2012 18:33:15 +0000

As Yavor said, Azure SDK for node 0.5.4 is out with a bunch of goodies!

Closing a security hole

Recently a vulnerability was detected in node.exe that could theoretically allow an attacker to perform a header-spoofing attack. Version 0.6.17 contains a fix for this attack. We take security very seriously, so we’re releasing this update which includes node 0.6.17 to remove the vulnerability.

Please go download the latest bits to remove this vulnerability!

Less angle brackets, more YAML

iisnode offers some really nice hosting capabilities like spinning up and managing multiple node procs, allowing access to logs over HTTP, providing good debugger errors in the browser for diagnostics and supporting node-inspector for debugging.

To access any of these benefits however you have to travel the sea of angle brackets known as web.config. For .NET / Windows developers, this is the norm. However, we heard a lot of feedback from folks in the node-a-verse, in particular coming from on a Mac / *nix that this feels very strange that they have use web.config in order to config node-specific things in Windows Azure, especially in light of the other offerings out there. Looking around we saw that a common pattern was to use a simple key-value format for specifying similar settings with YAML being a very popular format.

iisnode.yml

And so our team racked our brains a bit, got a bunch of feedback and iisnode.yml was born and implemented by Tomek! iisnode.yml is an optional file that sits along side web.config. It allows you to set all of our iisnode settings without having to ever touch web.config. Below is a really simple example.

# This is a really simple iisnode.yml file

node_env: development

devErrorsEnabled: true

logggingEnabled: true

The settings set the node_env environment variable to development, enables logging all node.exe output and enables developer errors.

For example, the code below has an error in that it requires a module that does not exist, also it uses spaces in the module name.

var http = require('http');

var notPresent = require('some awesome module');

http.createServer(function (req, res) {

  throw "error";

  res.writeHead(200, {'Content-Type': 'text/plain'});

  notPresent.doSomethingAwesome();

  res.end('Hello World Again\n');

}).listen(process.env.PORT);

If I don’t enable devErrors this is what I get when I do a request.

However, look at what I get when I enable devErrors with logging.

Above you an see that an error occurred because it could not find my wacky module.

Developer errors is a pretty cool feature that allows iis to output in the response any errors that occurred right in the browser. Combined with logging, it’s really useful for debugging on a remote/staging server.

Of course you don’t want anyone seeing that in your live production site, so you should probably shut that off.

myfile.yml

By convention we look for iisnode.yml. If you are not happy with that name however, you can set your own name in the iisnode element of web.config by using the configOverrides property.

<iisnode configOverrides="myfile.yml"/>

Note: configOverrides also allows you to do environment variable expansion. Thus instead of having a static file name you can have a name that includes an environment variable value. More on that in the future.

But wait, don’t I still need a web.config when I publish to Windows Azure?

Great question! Today you still need a web.config though as Tomek said that can be boilerplate and you don’t have to look at it. Tomorrow however….

Go get the latest SDK here.

Honored to have participated in the ACT Fly-In

johnvpetersen — Fri, 11 May 2012 12:03:47 +0000

You can read about the event here.

Every so often, you get the opportunity to participate in a process that affords you the opportunity to talk to those who make policy that affects our personal and business lives. During the Fly-In, a group of us engaged with members of Congress on topics that ranged from privacy, bandwidth and entrepreneurship.

My two big issues: privacy and the available bandwidth spectrum.

Privacy:

What people need is the ability to provide informed consent. It’s very difficult, if not impossible, for government to formulate regulations that covers every conceivable behavior. All these regulations serve to do is stiffle innovation and harm those that play by the rules. What we do know when we see it is bad conduct. The FTC has broad enforcement and police power and they need to use the power Congress gave them. Punish the bad actors and make an example out of them to provide the necessary incentives to handle online privacy the right way.

Bandwidth Spectrum:

Recently, Congress mandated FCC Spectrum Auctions. Are you tired of dead spots and sluggish performance over 3G/4G networks? Did you know the carriers stand at the ready to fund the billions of $’s necessary to put up and light up more towers? To make that work, the amount of available bandwidth has to be increased. Congress has taken an important first step – but more needs to be done. Increasingly, we rely on more than just plain text. Mobile Apps are providing (at least are trying to provide) rich content. What’s holding things back? Lack of available bandwidth. The bandwidth is out there, we just need to make it available and use it.

Node.js and Azure on .NET Rocks!

Glenn Block — Wed, 09 May 2012 08:26:28 +0000

Recently I had the privilege to talk to my good friends Richard and Carl about the work we’re doing enabling Node.js development with Windows Azure along with our friends in the node community. It was also a great opportunity for me to practice my new subdued Glenn persona You have to listen to believe it!

http://dotnetrocks.com/default.aspx?showNum=763

Carl and Richard talk to Glenn Block about node.js on Azure. Glenn digs into how node.js has grown in popularity, and points to the Cloud9 IDE as a development environment in the Cloud as an example of a node.js application – which happens to be able to build node.js applications. And Cloud 9 can deploy node.js to Azure. The conversation digs into the Azure stack and the diversity of technologies (including node.js) that run great in the cloud.

I’ll be at GOTO CPH and QCON NYC spreading the node.js love

Glenn Block — Fri, 04 May 2012 08:06:56 +0000

Conference season is coming again. I am excited this year that I’ll be finally speaking at goto AND that I’ll be speaking at the first QCon NYC. The topic, well you guessed it, node.js and Windows Azure.

May 21-23

First stop will be Copenhagen where I will be speaking at goto in the “Microsoft, beyond the echo chamber track” I am really looking forward to this event. I’ve watched it for years before when it was JAOO but never got the chance to attend. I am happy to say this year that will be changing. I hope to see you there!

Presentation: Breaking the Barrier with Node.js on Windows and Azure

June 18-22

I’ll be speaking at QCon NYC in the “Battle of the Clouds track”. If you are planning to attend, you can save $100 by using this promotion code when you register: BLOC100

Presentation: “Unlock your Inner Node.js in the Cloud with Windows Azure”. I’m planning to show some really cool stuff that we have coming down the pike soon at this event, so be there!

After the event, I’ll also be delivering two half day tutorials on node.js.

“node.js, jumping in with both feet”. This tutorial will be a basic primer around node.js and take you from 0 to 60 with node.js development. It will cover the basics, common practices as well as popular third party modules.
“node.js in Windows Azure”. In this tutorial we will dive into deploying node.js applications to Windows Azure as well as using Azure storage services and ServiceBus from within your node.js apps. We’ll also cover development tools like Cloud9 IDE that you can use for deploying node apps to Azure.

If you are looking to get up to speed with node or to start deploying node apps to Azure come to these tutorials.

While I am in town for both events I’ll be doing a bunch of user group meetups and such. I’ll cover that in my next post

Cron and AppEngine

Kyle Baley — Sat, 28 Apr 2012 15:52:14 +0000

Quick PSA on using cron jobs with Google App Engine because it almost wreaked havoc for us.

App Engine has a lovely feature of having different versions of your app. You can upload a new version but not make it the default until you’re good and ready. We do this all the time for deployment. Deploy to a new version and try it out, then make it the default when we’re ready to unleash it. Often, we deploy to the new version a day or so in advance.

Cron jobs, it seems, are handled outside this versioning mechanism. If you upload a new cron.xml file, it’s active. Right now. Doesn’t matter if the version it was deployed in is the default or not. As soon as it’s uploaded, it’s the new cron-ness.

Where this almost bit us is that we added a new cron job in our most recent release (deployed yesterday but not active) to use a dynamic backend. As soon as the cron job got uploaded, it started running. I didn’t notice until this morning when our backend usage reflected the new cron job. Some quick research and here we are.

What this means long term is that cron.xml is no longer going to be deployed as part of our application anymore. It now becomes an entirely separate process. I’m a little annoyed that we have to wait until we pull the trigger on the new version before we can upload the new cron.xml but it’s a quick step.

Kyle the Mis-scheduled

State Machines and Business Users

gregyoung — Tue, 24 Apr 2012 09:21:13 +0000

I was at a workshop over the weekend talking about the use of Finite State Machines as coordination mechanisms for systems. These things might be called sagas, activities, workflows, process managers (my preferred term lately from PEI), orchestrations, choreographers, and I am sure I am leaving out a few.

One of the guys in the talk (quite knowledgable) brought up that they used to use petri nets for this as well as pure finite state machines and there had been a push to more procedural mechanisms as business users don’t think in terms of finite state machines. I noted that while we have been moving to more procedural methods, as most of these procedural methods are in their hearts asynchronous our procedural techniques tend to be domain specific languages that internally generate state machines anyways, often based in the join calculus (think futures or the async keyword in C# when joining).

However later in the day I had a realization connecting two things. We like to describe business situations in gherkin style (not strict) specifications.

Given: Some initial starting point
When: Something happens
Then: Something occurred

We as an industry have been very successful getting specifications written in this style. We regularly do analysis and write acceptance criteria using this style. Business users seem to have been fairly capable of understanding it.

This style is basically describing a state machine.

Given: some initial state
When: a stimulus
Then: Something occurred (message/transition depending on style of testing)

How can we be so successful with this style of specification and in the next breath say that business users can’t understand state machines? The trick is they don’t understand the actual machine but they understand (and can specify very well) the exterior behaviours of such a machine.

Try getting a business user to create for you the state machine to control a simple DVD player for you. Likely they will scratch their heads and politely become very busy so they won’t be able to attend the next meeting. Try getting them to specify

Given a video is playing
When fast forward is pressed
Then the play goes at double speed

Maybe the problem here that caused our move towards procedural style controllers was the goal all the middleware seems to push of having business users writing the activities as opposed to reading about/specifying/understanding them. I have yet to see in all the organizations I have been to a single one where the dream of having business users creating these types of long running procedures actually panned out. to be fair: I have seen it work but the business user had formerly been a developer and BA

The Economics of Ergonomics

Kyle Baley — Sun, 22 Apr 2012 22:04:39 +0000

Let it not be said there are no downsides to living in the Bahamas (though if you’ll permit a little boasting, a shortage of fantastic venues if you’re lucky enough to be in a band is not one of them).

The desk where I work is too high, plain and simple. So much so that I’ve recently abandoned my Kinesis keyboard because it is not what you might consider “low form factor”. I’ve started feeling some twinges in my lower forearm that my unscientific diagnosis is attributing to the height of my hands while I type. Dumping the Kinesis has helped but it has also led to the return of stress in other areas of my hands. And no amount of raccoon skinning seems to alleviate the pain.

Getting a lower desk is easy enough but I’d actually like to do a little experimenting with two alternatives. Alas, neither are easily done in the Bahamas. The underlying problem is availability. The desks/equipment I want to test are not available here so I would have to order them in. Which means both shipping charges and import duties, the latter of which is a major source of income for the Bahamian government to offset the fact that there is no income tax. So returning said equipment is just not practical if it doesn’t work out. Nor is there much of a reseller market.

So I’m hoping I can get some comments from people who have done something similar.

Adjustable height desk

These are, of courses, desks where you can adjust the height easily. I like the idea of these for two reasons:

They can be set low
They can be set high

I’ve never tried a desk that you stand at but I’ve always wanted to. Working on my own, I tend to get up and wander a lot while I’m thinking. I also pace when I’m on the phone with someone for any length of time so it would be more convenient to walk up to the computer during the conversation should the need arise. (“You want to know the right pattern of plaid for a first date with your second cousin? Let me look that up.”)

I went desk-shopping over the weekend and the closest thing I saw was in an office supply store. And it wasn’t on the showroom floor. Off in the corner of the store were the employee desks. They were all essentially plywood based, laminated desktops all mounted in warehouse style shelving frames. They sat on brackets in the frame which means you could set the height to whatever you want. It wasn’t something you could easily adjust on the fly and my wife wasn’t too thrilled at the industrial look so it was a fleeting idea at best.

Command centre

This is an idea I’ve had ruminating in my head for a while now. I would get rid of the desk altogether in favour of a comfortable command-centre style or gaming chair. In front of it it, I’d mount my monitors on a couple of flexible arms somehow, possibly on the armrests or on stands on either side of the chair. The important thing is that I can slide the monitors out of my way when I want to get out of the chair, and slide them back in when I sit down.

The keyboard would rest either on my lap or on some flat surface on my lap. Or maybe go with a split keyboard (though one without a wire between the two) and have one piece mounted on each armrest. Haven’t quite worked out how the mouse would fit in though. A trackball on some little platform on the side makes sense but I’ve got one now and it doesn’t feel as productive as just a regular mouse.

I feel like this would be more comfortable and would reduce much of the muscle stress that seems to have become more prominent since hitting 40 earlier this year. All of this kind of makes sense in my head but the logistics of getting the stuff here is such that I don’t want to make the investment unless I’ve had a chance to try it out at least for a few days. There’s a chance my tendency to get up and wander might make this impractical. Or maybe cord management would be an ongoing problem.

The device shown at right, which I discovered while researching this article, is essentially what I’ve described. It’s some US$2750. Duty would add about 50% and shipping would likely bring the total price above five large. There’s another potential hurdle in that it may not be available anymore given the company’s domain seems to point to a parking spot. But even building my custom version will cost enough in non-refundable cash dollars for me not to head over eBay.

Instead, I hunt for a standard desk about four to six inches lower than the one I’ve got. Not as exciting, possibly not as ergonomic, but easier to replace.

So my question to you, my honorary hillbillies, for anecdotal evidence. Have you tried either of these devices? What’s good and bad? Good return for the money or does it sit in the garage next to the Bowflex you bought in a fit of New Year’s anxiety?

Kyle the Unreturnable

Sunday Thought: Patient / Fast

Dru Sellers — Sun, 22 Apr 2012 14:38:55 +0000

Some times I am such a dolt.

I open a new OSS project and my first thoughts are usually “This library is ^&*$% it doesn’t do anything right.”

“RAGE! (╯°□°）╯︵ ┻━┻” don’t they know that, MY WAY is better!

Its sad, but that is exactly what I am thinking. I am on such an immediate gratification high, thanks to my 8 core, 8GB RAM, SSD, super high speed internet capable machine that I forget that not everything can be measured in sub-second latency. Things like instant on-demand movies, and Amazon 2 day shipping don’t help with my attitude either. When it comes to learning, this demand for instant gratification has, many times, gotten in the way of what it is I really want to accomplish.

So, I ran into this situation again this week, and after IM’ing the maintainer – complaining about how this and that worked (ok, lets be honest – I was whining like a 4 year old) – I kinda just snapped out of it. I recalled a quote from the book The Toyota Way, where they stated that the #1 problem with new engineers is that they can’t slow down. They just want to charge into things head first and solve all of the problems. Hmm, sounds a lot like me. Except that I have 10 years underneath my belt now. So, why am I still acting like a new developer? My biggest guess, is that I haven’t really honed my process. I need more people to catch me acting this way and to call me on it. But that can be hard to find in our community, because we all tend to act this way.

We, developers, are always on the look out for the next shiny library or framework that we can “learn” and by learn I mean implement. We like libraries so simple that a new programmer could understand them. We don’t want a learning curve at all. I think that this is one of the reasons things like express.js, flask, and sinatra play so well in our community. You can see, immediately, how it works. They have almost no abstraction over the underlying concepts. Let’s not forget, that its the abstractions that bring the real power, and that not every abstraction is a poor or leaky one.

Its Sunday, which is an introspective day for me, and I want to try and improve myself. I think that if I could work on this aspect of my skills and attitude, I would become a better developer. Being a better developer is something that I want to be. Ok, so how am I going to do this. After much thought, I was reminded of a similar experience that worked well for me outside of the software development world.

For that last three months I have been training in Olympic Lifting. Now for those of you that have never performed olympic lifts, you need to know that while strength is a big part of it, skill also plays a huge part. You can’t just hoist 150 pounds over your head with out a level of skill. The snatch is an explosive lift, that requires an insane amount of patience and balance. My coach and teammates have a phrase that I have come to love. ”Patient / Fast”, which means to me that you have to move fast, but at the same time be patient in the pull. It took me a long time to really grok that concept. It feels like a damn Zen koan at times, but I keep repeating the phrase and each day it reveals itself to me a bit more.

So there I was, at the keyboard, about to burst a vein, when I woke myself up from my emotions, calmed down – and asked myself what would “Patient / Fast” look like here? I decided that I would patiently look through the code, asks questions to myself via comments and then go answer them, and start to understand what the code was doing, on the flip side I would be very fast to start writing, running, and debugging unit tests to get ahold of the code base. Sure enough, the code started to reveal itself to me, I could see the author’s intentions much better now. I started to build up a good amount of understanding, and was able to effectively solve my problem but was also even able to make a nice pull request back to the library that I think cleaned up a very small part of the code base.

In the end, I spent more time than the old me would have liked understanding the library. But the reality of it is, this small investment in understanding a key part of my infrastructure is going to pay dividends in the long run. And less than a day understanding how a critical part of my system works, is indeed a small investment. So, from here on out I am going to remember. Patient / Fast

Ruby and RSpec: Powerful Languages Allow Simpler Frameworks

James Kovacs — Fri, 20 Apr 2012 04:32:51 +0000

Recently I was doing a simple kata – the Roman Numeral kata – to practice my Ruby and RSpec skills. (The Roman Numeral kata is to build an algorithm test-first that converts a number into its Roman numeral equivalent. For example, 1 to i, 7 to vii, 10 to x, etc. The problem is intended to be simple so that you can focus on the process.) I had the following RSpec code:

require 'roman_numeral'

describe RomanNumeral do
  # specify is just an alias for it. specify reads better in this case
  specify '1 should be i' do
    numeral = RomanNumeral.new(1)
    numeral.to_s.should == 'i'
  end
end

I go through the Red/Green/Refactor cycle and get this first spec working. (I won’t show the production code as it’s not that interesting.) I move onto the second spec.

require 'roman_numeral'

describe RomanNumeral do
  specify '1 should be i' do
    numeral = RomanNumeral.new(1)
    numeral.to_s.should == 'i'
  end

  specify '2 should be ii' do
    numeral = RomanNumeral.new(2)
    numeral.to_s.should == 'ii'
  end
end

Red/Green/Refactor and all is good. But I’m seeing some repetition in my specs and it’s only going to get worse. The specs are almost identical except for the data. So I start hunting through the RSpec docs for something akin to NUnit’s [TestCase] or xUnit’s [Theory]. I find nothing. Maybe, like MSpec, RSpec wasn’t designed to do this sort of data-driven testing.

Undaunted I turn to Google and stumble upon this StackOverflow question asking exactly the same question that I had. Matt Di Pasquale, the OP (original poster), found his own answer and that answer was fascinating! No, RSpec doesn’t have a syntax for test cases because it doesn’t need one. Use the Ruby, Luke!

require 'roman_numeral'

describe RomanNumeral do
  cases = {
    1 => 'i',
    2 => 'ii'
  }

  cases.each do |k, v|
    specify "#{k} should print as #{v}" do
      numeral = RomanNumeral.new(k)
      numeral.to_s.should == v
    end
  end
end

For those not as well-versed in Ruby, “cases” is simply a variable that contains a hash. I then iterate over the key/value pairs and define my specifications. It’s that simple. I didn’t need any special attributes or framework support from RSpec. I didn’t need to learn the inner workings of RSpec to write my own custom extension or attributes. I simply wrote some Ruby code. Let that sink in for a second. I simply wrote some Ruby code.

Now think about that a bit more. I’m writing regular Ruby code to implement the notion of test cases. Rather than defining test cases, I could have used very similar code to define a benchmark that verifies an algorithm scales linearly with number of input elements. Or I could have fuzz tested an external API for my application. The possibilities are only limited by my imagination and don’t require me to gain an ever deeper understanding of my testing/specing framework to implement.

For someone who cut his teeth on C, C++, and C# (with some JavaScript thrown in for good measure) and is used to learning (or building) frameworks to solve problems, the notion that you can just write plain old code to solve these types of meta problems is eye-opening. It is one of those ah-ha moments that matures you as a developer. Not every problem needs a framework to solve it. Sometimes it just requires a little code.

Using oData with the NuGet API

johnvpetersen — Sun, 15 Apr 2012 16:49:49 +0000

Update…

As a commenter astutely pointed out, we can see the generated URL in LinqPad..

Still good to know and understand how to use Fiddler – especially if you want to flip between json and xml formats!!

————————–

Quick tip here on using oData with NuGet.

If you have visited the NuGet Gallery, you may have wondered were the RSS feed is. Try as you might, you won’t find it. There’s good reason for that given the fact that NuGet is backed by a data feed – a data feed with many data fields that you can query to properly scope your request. NuGet has a rich API -which you will see in a moment. Usually, when we work with NuGet, we do so via the NuGet Package Manager Dialog and/or the Package Manager Console. There may be times when you wish the raw feed. To do that, you need the API url. By default, NuGet gives you one package source, the official public NuGet Feed – which can be found at https://nuget.org/api/v2. The following image illustrates how the official feed is wired into your NuGet installation. You can get to this dialog by navigating to:

Tools\Library Package Manager\Package Manger Settings

One of the most useful features in .NET is Linq and the best tool available to learn Linq is LinqPad. In the following image, I set a connection to https://nuget.org/api/v2. As you can see, there are a number of fields in the package feed. With LinqPad, it’s easy to enter a Linq Query (whether it is the long form or as a lambda expression).

Packages.OrderByDescending (x => x.LastUpdated)
   .Select (
	  x =>
		 new
		 {
			Id = x.Id,
			Tags = x.Tags,
			Title = x.Title,
			Description = x.Description,
			ProjectUrl = x.ProjectUrl,
			GalleryDetailsUrl = x.GalleryDetailsUrl,
			Version = x.Version,
			LastUpdated = x.LastUpdated
		 }
   )

In this query, we are taking certain data fields and are ordering by the LastUpdated DateTime field descending.

Another useful tool is fiddler. The combination of LinqPad and Fiddler makes it easy to see how Linq queries get translated to oData URL's:

Here's how the oData URL appears as rendered in a browser:

Here is the URL:

http://nuget.org/api/v2/Packages()?$orderby=LastUpdated%20desc&$select=Id,Tags,Title,Description,ProjectUrl,GalleryDetailsUrl,Version,LastUpdated

Going back to Fiddler, if you want JSON returned instead of XML, just tweak the Accept Header to:

Accept: application/atom+json,application/json

That's all there is to accessing NuGet data via the NuGet API with oData.

Why I’m moving away from Apple, and specifically, the iPhone and going to the store

johnvpetersen — Sat, 14 Apr 2012 20:52:51 +0000

And it has nothing to do with technology for the most part. On balance, Apple makes some great products. But there are two things that Apple sucks at:

- Customer Service (especially at an Apple Store – more on this later)
- Battery life

First off, I’d like to thank Luke C., one of the managers at the King of Prussia Apple Store for wasting 90 min of my life today. The same goes for the other manager of whom I didn’t get his name.

I have said this every time I’ve gone into an Apple Store lately….”Never again.” They are crowded and unless you are there to buy something, the Apple retail crew is not really interested in helping you. The tough thing with batteries is that you have to go through some difficulty to get issues addressed. That often means going to the store – and make an appointment at the “Genius Bar.” Don’t know about you, but when I pay premium prices for a product, I expect superior customer service. And more often than not, I usually get it. Tumi is a great example of a company that stands behind their products.

Anyway, for some time now, I’ve had issues with battery life wtih my iPhone. I had the phone replace in December – and for the most part – it’s been OK. Today, Luke C. stated that they had their process. They had to make sure it wasn’t a software related problem. I was agreeable. Fast forward 50 minutes and while they told me everything was OK – turns out that it wasn’t. The battery drains at a rate of about 1% every 2-3 min. The funny thing is, when confronted wtih an outcome he didn’t expect, Luke and his buddy stated – “I could buy a new battery for $89.” I then became disagreeable. What is it with these folks that work at the Apple Stores? Is it a requirement to be a pendantic d-bag? They clearly have lying down to a fine art. Whatever – I have to consider the source. These guys work in retail and are a reminder why I prefer the online experience when it comes to making purchases.

Often, when considering things like phones, tablets, etc – we evaluate based on technology. We really should be looking at customer service as well. And in the case of Apple, that charges a premium price for their products – they have a greater responsibility to provide superior customer service. The irony as to the store is that for all of the “experience” the Apple folks worry about, an Apple store today, has got to be the worse customer experience out there today. Others may disagree – but that is my opinion.

I’m just one guy and Apple won’t care as it’s market cap continues to creep toward a trillion dollars. Apple makes a profit of over $350 per iPhone and it is hypothesized that Apple generates $30 of profit on an iPhone for every $1 of labor. And the $89 battery? What’s that cost? $5…maybe.

I’m all for companies making as much as they can. I’m a capitalist. But…I’m also a “fairist” as well. In other words, I believe that companies should earn the premium they charge. Standing behind their products, when they are defective, is such a case when they need to earn that premium. Well…I have my remedy as well.

Audit Fields in Google AppEngine

Kyle Baley — Sat, 07 Apr 2012 23:52:41 +0000

Executive summary: Here’s how we’re implementing audit fields in AppEngine. IT’S BETTER THAN THE WAY YOU’RE DOING IT!

I considered saying “I hope there’s a better way of doing it” but I believe I’ll get more responses if I frame it in the form of a challenge.

For all entities in our datastore, we want to store:

dateCreated
dateModified
dateDeleted
createdByUser
modifiedByUser
deletedByUser

Here are the options we’ve considered

Datastore callbacks/Lifecycle callbacks

AppEngine supports datastore callbacks natively. If you use Objectify, they have lifecycle callbacks for @PrePersist and @PostLoad. The former works fantastic for dateCreated, dateModified, and dateDeleted. Objectify can handle all three easily as well provided you use soft deletes, which we do. (And they aren’t as bad as people would have you believe, especially in AppEngine. You’d be surprised how many user experience problems you discover strolling through deleted data.)

Both of these led to problems for us when we tried to use them for the createdByUser et al methods. We store the current user in the session and access it through a UserRetrievalService (which, at its core, just retrieves the current HttpSession via a Guice provider).

If we want to use this with the Objectify lifecycle callbacks, we would need to inject either our UserRetrievalService or a Provider into our domain entities. This isn’t something I’m keen on doing so we didn’t pursue this too rigorously.

The datastore callbacks have an advantage in that they can be stored completely separately from the entities and the repositories. But we ran into two issues.

First, we couldn’t inject anything into them, either via constructor injection or static injection. It looks like there’s something funky about how they hook into the process that I don’t understand and my guess is that they are instantiated explicitly somewhere along the line. Regardless, it meant we couldn’t inject our UserRetrievalService or a Provider into the class.

The next issue was automating the build. When I try to compile the project with a callback in it, the javac task complained about a missing datastorecallbacks.xml file. This file gets created when you build the project in Eclipse but something about how I was doing it via ant obviously wasn’t right. This also leads me to believe there’s something going on behind the scenes.

Neither of these problems is unsurmountable, I don’t think. There is obviously some way of accessing the current HttpSession somehow because Guice is doing it. And clearly you can compile the application when there’s a callback because Eclipse does it. All the same, both issues remaining unsolved by us, which is a shame because I kind of like this option.

Pass the User to Repository

This is what was suggested in the StackOverflow question I posed on the topic. We have repositories for most of our entities so instead of calling put( appointment ), we’d call put( appointment, userWhoPerformedTheAction ).

I don’t know that I like this solution (as indicated in my comments). To me, passing the current user into the DAO/Repository layer isn’t something the caller should have to worry about. But that’s because in my .NET/NHibernate/SQL Server experience, you can set things up so you don’t have to. Maybe it’s common practice in AppEngine because it’s still relatively new.

(Side note: This question illustrates a number of reasons why I don’t like asking questions on StackOverflow. I usually put a lot of effort into phrasing the question and people often still end up misunderstanding the goal I’m trying to achieve. Which is my fault more than theirs but still means I tend to shy away from SO as a result.)

Add a User property to each Entity

I can’t remember where I saw this suggestion. It’s kind of the opposite of the previous one. Each entity would have a User property (marked as @Transient) and when the object is loaded, this is set to the current user. Then in your repositories, it’s trivial to set the user who modified or deleted. This has the same issue I brought up with the last one in that the caller is responsible for setting the User object.

Also, when new objects are created, we’d need to set the property there as well. If you’re doing this on the client, you may have some issues there since you won’t have access to the HttpSession until you send it off to the server.

Do it yourself

This is our current implementation. In our repositories, we have a prePersist method that is called before the actual “save this to the datastore” method. Each individual repository can override this as necessary. The UserRetrievalService is injected in and we can use it to set the relevant audit fields before saving to the repository.

This works just fine for us and we’ve extended it to perform other domain-specific prePersist actions for certain entities. I’m not entirely happy with it though. Our repositories tend not to favour composition over inheritance and as such, it is easy to forget to make a call to super.prePersist somewhere along the way. Plus there’s the nagging feeling that it should be cleaner and more testable than this.

Related to this is the underlying problem we’re trying to solve: retrieve the user from the session. In AppEngine, the session is really just the datastore (and memcache) with a fancy HttpSession wrapper around it. So when you get the current user from the session, you’re really just getting it from the datastore anyway using a session ID that is passed back and forth from the client. So if we *really* wanted to roll our own here, we’d implement our own session management which would be more easily accessible from our repositories.

So if you’re an AppEngine user, now’s where you speak up and describe if you went with one of these options or something else. Because this is one of the few areas of our app that fall under the category of “It works but…” And I don’t think it should be.

Kyle the Pre-persistent

Microsoft and Open Source

Sebastien Lambla — Thu, 05 Apr 2012 11:44:09 +0000

I have been a vocal, and sometimes harsh, critic of Microsoft’s approach to Open Source Software. I call that activism, some call it whining, ranting or pisser dans la soupe. People are entitled to their opinion, and mine has been formed after working in the Microsoft sphere for more or less 14 years.

Over the years, the various component organizations within Microsoft have had various levels of involvement with Open Source, with different philosophies, and from what I have gathered over the years from post-conference drinks, internal conflicts that can be quite a challenge.

As this post is going to be long, I’m going to sum-up the problem, and you can go to the conclusion bit to read what I believe should be done differently.

Microsoft has no vision when it comes to Open-Source, no strategy and no leadership. Some groups are seen as more progressive, and do tend towards a better approach to OSS. Some people consider those steps as sufficient to stop pressuring Microsoft towards responsible OSS, and I disagree: it’s only some groups, and even within those groups, the advances are still very much one of exporter. Recognizing Microsoft as an OSS-friendly player is like recognizing the People’s Republic of China under the Reagan administration, it’s a step in the direction of normalization, not a recognition of the Human Right lifetime achievements of Mao Tsedung.

You know why I know Microsoft has no leadership and vision when it comes to Open-Source? Because I asked Steve Ballmer when he was in London, and he replied with this: “I don’t know, but we won’t impose any view on our divisions. I’ll come back to you by email though.” Steve, my email is still seb@serialseb.com and I’m looking forward to your views.

Where we come from

The current state of affairs is not bad, considering where Microsoft was years ago. We’re talking about a company that positioned itself as the defender of private code, vilifying OSS licenses, having no visible OSS project going and providing its value as the integrated stack: it’s all MS or none. We have certainly come a long way.

Opening up slowly

I think under the influence of Scott Guthrie, an internal change started deep within the asp.net team ranks, certainly a movement that resulted in the recent announcement of the opening up of the asp.net webstack as a true Open-Source project.

Now that’s commendable, and I applaud the move. Open-Sourcing and opening up to collaborative development for stuff you’ve built is great. Well done.

Not invented in Redmond syndrome

Let’s be clear. The goal of MVC was to build a better Monorail, as opposed to help Monorail productize their framework for inclusion in Visual Studio.

The exact same thing happened with Nuget. With multiple projects with different approaches already in the work, Microsoft started their NPack project (I seem to recall someone at Oredev mentioning it was originally built as part of the WebPages tooling). It wasn’t to be open-source to start with, hence why the existing OSS projects didn’t get the news until much later.

But, I hear you say, Nuget is an open-source project from Outercurve, and they combined their code with another project. The open-sourcing was decided very late in that process, once the investment in code was so large that the ship had left already and nothing would stop it. And on a side-note, there was no merging of the code, the Outercurve foundation’s press release was inaccurate, just look at the history.

What really happens when a new project gets produced by Microsoft? It sucks the air out of a lot of community projects, often including all the best ideas of said projects, and Microsoft gives nothing back. If you’re in that boat, you either redraw your plans based on their new announcements (usually rewriting on top of the new shiny stuff), or you dissolve your group and join their OSS project (the Nuget / Nubular case).

Certain projects (like nhibernate and nunit) continue to happily go forward, and maybe the OSS alternatives that are impacted by Microsoft entering a space just were not good enough. I don’t know, we’ll see in a few years.

Adoption of existing projects

Of course it can be said, and there again, I commend Microsoft for the move, that libraries such as jQuery, modernizr and now Newtonsoft.Json, are being used, supported by Microsoft and shipped alongside the product.

This is also a great departure from time immemorials. Makes no mistake however, Microsoft is being pushed by market pressure to integrate with those frameworks. The change is only that they’re abandoning slowly the idea that they’ll release until they gain marketshare, in those domains where Microsoft has historically been slow at releasing compared to alternative platforms.

This is only an opinion piece, but I’ll suggest that the move to gain web development share has pushed Microsoft towards dropping the projects they had and failed to make popular (the asp.net AJAX stuff that got more or less killed when jQuery was adopted). And I think that’s great.

Pragmatism is also the reason why newtonsoft.json is being used by Web API. The current Json providers on .net are broken, there is no way to provide the customization people want, and fixing them cannot be done in the same release cycles as MVC, as some are part of the core framework (and may or may never be fixed, but certainly not in time for the next release).

It was not chosen because Microsoft wants to adopt more external OSS projects as their core vision. It was chosen because they couldn’t fix what was broken and rebuilding would have been silly.

On a side-note, twitter conversations started going in the direction of why Newtonsoft.Json was chosen over other libraries. I made the assertion that provided it’s an internal implementation with no exposed surface, the choice of *any* json lib would’ve been fine. ScottHa said it was because the library was the most popular, and testing was done on their part. That lead me to the assertion that if the only library under consideration was chosen for popularity, and there is no exposed surface of that lib on public APIs, the choice is one of brand-surfing and gives extra marketing points to their OSS credentials. If some of the API is exposed, it was probably a reasonable choice. I don’t care which library you chose, I care the process you used to chose which library and the reasoning behind it. I question (and by question, I mean discuss) the motives, not the solution.

[Edit: After looking for it (as the twiterverse didn’t reply to the specific of the discussion), one object from json.net *is* exposed on the JsonMediaTypeFormatter, so that is that and I’ll stick to my original analysis, it’s probably a reasonable choice.]

There is no vision, because there is no systemic approach to deciding between building and joining communities.

Asp.net Web API could have quite happily join existing OSS projects that dealt with this problem area years ago, but it wasn’t to be. Nuget funding could have been used on top of existing projects, instead of a pure creation, but it wasn’t to be either. What is the decision process behind the rebuild or join decision?

And can anyone that has had their project get the oxygen sucked out of, with nothing given back, be blamed for distrusting the company?

The corporation

When the Web API project was being kicked off, Glenn Block called me asking me if I thought he should take the job, and if he did, would I help. I told him what I’m telling you now: Yes, you can’t do as bad as the previous attempts, and I’ll happily help to make sure the architecture and model fits the reality of what developers want, we’ll all win. I helped because I was asked. Ideally I’d rather they had asked me to work on OpenRasta and create a new version with us, but it was not to be. In the end, the two frameworks may have very different code, but most of the concepts and conventions are exactly the same, so everyone will feel right at home on both platforms, and that’s in no small measure because the brilliant minds behind Web API respected HTTP as much as I and the other advisors did.

More recently, Drew Miller from the NuGet team has been very active in opening communication channels with OpenWrap, something that has not existed ever, and he should be applauded for it: I can now have a discussion about our interop story with someone that wants to make things better. Again, I’d rather the NuGet team had joined OpenWrap rather than communicate to us their imminent release, way too late to get any positive outcome, but at least it’s a step in the right direction.

Those are individuals, not the company, and out of all the voices out there, they are a minority. You cannot judge the whole company on specific individuals, and teams cannot use the excuse Microsoft is really many companies. That is certainly true, but completely irrelevant to external agents when judging how the company deals with OSS.

How to fix this

Three words, transparency, transparency, transparency.

When you have an idea for a product, and there are existing Open-Source projects out there that have adoption and could really help the resources of a big vendor, try and justify the choice.

If you’re going to rebuild, take the phone, call the leader, explain your plans, and don’t be a douche about it (aka telling someone “we realize we are stepping on toes here but we’re big so it’ll happen” is not particularly a great way to deliver the news). Chances are, they’ll probably understand the reasoning. ISVs have had that communication channel from Microsoft for years, and get to know about competing products way in advance, but it’s not the case for OSS projects.

Be open about the process afterwards too. The policy of not mentioning anyone, or pretending you cannot look at existing projects at all in public, would make anyone think you have something to hide. It probably is against some corporate policy somewhere, but make that change, you did do it about the no-OSS policy.

And finally, value the people on your platform, the loud mouths like me and the small guys that abandon their projects because you started yours. We are on your platform because we like it, and we want to make it better. And maybe if you’re going to step on people’s toes, try to find a band aid for them to sweeten the blow, and find a way to collaborate on something else with them. Don’t lose the people that have been developing the ideas you’re going to be building on, it’s not good business.

If these things had happened in the last few years, I wouldn’t distrust Microsoft-the-company as much as I do now. But the bridges that are being built by bright individuals inside the company do go in the right direction.

Is that going to be enough? I sincerely don’t know, but until we’re there, I’ll continue shouting at the top of my voice for Microsoft to become the business partner my OSS projects wants to be dealing with.

It’s called tough love, silly.

[Edit: added an additional paragraph to project competition]

[Edit 2: added a bit on the json.net choice]

Moving from Action Filters to Message Handlers

johnvpetersen — Wed, 04 Apr 2012 12:57:05 +0000

The updated code for the Web API project can be found here In the last post on Making Your ASP.NET Web API’s Secure, I received some very useful comments. Thanks to Pedro Reys for pointing out my non-use of HttpMessageHandlers. Pedro is absolutely correct in his observation that through the use of handlers, we can detect issues BEFORE processing gets into the controller context. If you are concerned about performance (and we all should be!!) – Http Message Handling is where you want to be. It turned out to be a pretty easier conversion. As you may recall, there were 3 actions filters that enforced:

HTTPS
A valid authorization token
A valid IP Host source

Here are the 3 new handlers:

using System;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;

namespace WebAPI
{
 public class HttpsHandler : DelegatingHandler
 {
  protected override Task SendAsync(HttpRequestMessage request,
   CancellationToken cancellationToken)
  {
   if (!String.Equals(request.RequestUri.Scheme, "https", StringComparison.OrdinalIgnoreCase))
   {
    return Task.Factory.StartNew(() =>
    {
     return new HttpResponseMessage(HttpStatusCode.BadRequest)
     {
      Content = new StringContent("HTTPS Required")
     };
    });
   }
   return base.SendAsync(request, cancellationToken);
  }
 }
}

using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using WebAPI.Models;

namespace WebAPI
{
 public class TokenValidationHandler : DelegatingHandler
 {
  protected override Task SendAsync(HttpRequestMessage request,
   CancellationToken cancellationToken)
  {
   string token;

   try
   {
    token = request.Headers.GetValues("Authorization-Token").FirstOrDefault();
   }
   catch (System.InvalidOperationException)
   {
    return Task.Factory.StartNew(() =>
    {
     return new HttpResponseMessage(HttpStatusCode.BadRequest)
     {
      Content = new StringContent("Missing Authorization-Token")
     };
    });
   }

   try
   {
    var foundUser = AuthorizedUserRepository.GetUsers().FirstOrDefault(x => x.Name == RSAClass.Decrypt(token));
    if (foundUser == null)
     return Task.Factory.StartNew(() =>
     {
      return new HttpResponseMessage(HttpStatusCode.Forbidden)
      {
       Content = new StringContent("Unauthorized User")
      };
     });
   }
   catch (RSAClass.RSAException)
   {
    return Task.Factory.StartNew(() =>
    {
     return new HttpResponseMessage(HttpStatusCode.InternalServerError)
     {
      Content = new StringContent("Error encountered while attempting to process authorization token")
     };
    });
   }
   return base.SendAsync(request, cancellationToken);
  }
 }
}

using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using WebAPI.Models;

namespace WebAPI
{
 public class IPHostValidationHandler : DelegatingHandler
 {
   protected override Task SendAsync(HttpRequestMessage request,
   CancellationToken cancellationToken) {

    var context = request.Properties["MS_HttpContext"] as System.Web.HttpContextBase;
    string userIP = context.Request.UserHostAddress;

    var foundIP = AuthorizedIPRepository.GetAuthorizedIPs().FirstOrDefault(x => x == userIP);
    if (foundIP == null)
     return Task.Factory.StartNew(() =>
     {
      return new HttpResponseMessage(HttpStatusCode.Forbidden)
      {
       Content = new StringContent("Unauthorized IP Address")
      };
     });

    return base.SendAsync(request, cancellationToken);

   }
 }
}

I went ahead and created a specific exception to address situations when the RSA encryption/description process throws the generic bad data exception. Here is the revised RSAClass:

using System;
using System.Linq;
using System.Security.Cryptography;
using System.Text;

namespace WebAPI
{
 public class RSAClass
 {
  private static string _privateKey = "s6lpjspk+3o2GOK5TM7JySARhhxE5gB96e9XLSSRuWY2W9F951MfistKRzVtg0cjJTdSk5mnWAVHLfKOEqp8PszpJx9z4IaRCwQ937KJmn2/2VyjcUsCsor+fdbIHOiJpaxBlsuI9N++4MgF/jb0tOVudiUutDqqDut7rhrB/oc=AQAB

3J2+VWMVWcuLjjnLULe5TmSN7ts0n/TPJqe+bg9avuewu1rDsz+OBfP66/+rpYMs5+JolDceZSiOT+ACW2Neuw==

0HogL5BnWjj9BlfpILQt8ajJnBHYrCiPaJ4npghdD5n/JYV8BNOiOP1T7u1xmvtr2U4mMObE17rZjNOTa1rQpQ==jbXh2dVQlKJznUMwf0PUiy96IDC8R/cnzQu4/ddtEe2fj2lJBe3QG7DRwCA1sJZnFPhQ9svFAXOgnlwlB3D4Gw==evrP6b8BeNONTySkvUoMoDW1WH+elVAH6OsC8IqWexGY1YV8t0wwsfWegZ9IGOifojzbgpVfIPN0SgK1P+r+kQ==LeEoFGI+IOY/J+9SjCPKAKduP280epOTeSKxs115gW1b9CP4glavkUcfQTzkTPe2t21kl1OrnvXEe5Wrzkk8rA==HD0rn0sGtlROPnkcgQsbwmYs+vRki/ZV1DhPboQJ96cuMh5qeLqjAZDUev7V2MWMq6PXceW73OTvfDRcymhLoNvobE4Ekiwc87+TwzS3811mOmt5DJya9SliqU/ro+iEicjO4v3nC+HujdpDh9CVXfUAWebKnd7Vo5p6LwC9nIk=";
  private static string _publicKey = "s6lpjspk+3o2GOK5TM7JySARhhxE5gB96e9XLSSRuWY2W9F951MfistKRzVtg0cjJTdSk5mnWAVHLfKOEqp8PszpJx9z4IaRCwQ937KJmn2/2VyjcUsCsor+fdbIHOiJpaxBlsuI9N++4MgF/jb0tOVudiUutDqqDut7rhrB/oc=AQAB";
  private static UnicodeEncoding _encoder = new UnicodeEncoding();

  public static string Decrypt(string data)
  {

   try
   {
    var rsa = new RSACryptoServiceProvider();
    var dataArray = data.Split(new char[] { ',' });

    byte[] dataByte = new byte[dataArray.Length];
    for (int i = 0; i < dataArray.Length; i++)
    {
     dataByte[i] = Convert.ToByte(dataArray[i]);
    }

    rsa.FromXmlString(_privateKey);
    var decryptedByte = rsa.Decrypt(dataByte, false);
    return _encoder.GetString(decryptedByte);

   }
   catch (Exception)
   {
    throw new RSAException();
   }

  }

  public static string Encrypt(string data)
  {

   try
   {
    var rsa = new RSACryptoServiceProvider();
    rsa.FromXmlString(_publicKey);
    var dataToEncrypt = _encoder.GetBytes(data);
    var encryptedByteArray = rsa.Encrypt(dataToEncrypt, false).ToArray();
    var length = encryptedByteArray.Count();
    var item = 0;
    var sb = new StringBuilder();
    foreach (var x in encryptedByteArray)
    {
     item++;
     sb.Append(x);

     if (item < length)
      sb.Append(",");
    }

    return sb.ToString();

   }
   catch (Exception)
   {
    throw new RSAException();
   }
  }

  public class RSAException : Exception {

   public RSAException() : base("RSA Encryption Error") {}

  }
 }
}

The one difference I did notice between the action filter and handler was in the way unhandled errors were passed back to the client. For example, with an action filter, a missing authorization token would have the following rendered to the client:

On the other and, the message handler would render the same error the client this way:

You may have noticed I changed the linq calls from First to FirstOrDefault. I got flamed a bit for using just First – and then letting the exception deal with the results in the event a value could not be found. I still contend that using an exception or checking the nullity of a value doesn’t make that much of a difference. In the case of checking for the existence of a header, it’s a moot point because if you invoke code like this:

var token =
  request.Headers.GetValues("Authorization-Token").FirstOrDefault();

and the Authorization-Token header does not exist, you never get to the FirstOrDefault() Linq Query…:-) As it turns out, you have to wrap that call in a try catch anyway – at least if you wish to present to the client a predictable and useful message. Always take with a grain of salt when somebody espouses “best practice” or something being a “bad practice” or “bad form”. No question, there are established good/best practices out there. Those practices however, are quantifiable and verifiable through empirical evidence. Anything else is simply a matter of opinion and preference. AND – context means everything. What may be a good/best practice in one scenario may not be so in another scenario. And often, technology has little to nothing to do with the equation. For example, Dependency Injection is a good practice and is often, a best practice. BUT – in a shop where the concept is new, foreign or not understood, if it presents a bar to shipping software, it may not be a best practice for that shop – at that time. One of the many reasons why I quickly brush off notions of what best practices are and are not when asked.. A bit of a rant on my part. I definitely appreciate the comments – but rest assured, I’ll push back a bit when I think conclusions are offered absent a premise. Sorry..that’s the lawyer in me that is now hard wired!!

Making your ASP.NET Web API’s secure

johnvpetersen — Mon, 02 Apr 2012 21:03:22 +0000

Note: Code for this example is on Google Docs.

SecureWebAPI
SecureWebAPITest

NOTE: DO NOT USE THE PUBLIC/PRIVATE KEYS IN THIS EXAMPLE IN YOUR PRODUCTION APPLICATION!!!! BE SURE TO GENERATE YOUR OWN KEYS!! OTHERWISE, YOUR APPLICATION WILL NOT BE SECURE AS THE PRIVATE KEY WILL BE IN THE FIELD!!!!

Recently, I’ve been exploring the new ASP.NET Web API. So far, I’ve been impressed with how easy it is to build RESTful web interfaces. In the examples I’ve published, none have been secure. In the real world – the world that exists beyond the world of samples and demos – security is a matter than cannot be brushed aside. In this post, I squarely tackle that issue by showing you an approach that locks down and secures your ASP.NET Web API.

In research this topic, looking what others have done this far, I came away with a lot of approaches that in my opinion, where too complicated and quite frankly, a pain to setup. It was also questionable what the value proposition was for all that pain. In other words, how was my site made more secure by implementing one approach or another. Two big topics that I won’t discuss today are oAuth and OpenID. Sure, those approaches each solve a segment of the overall problem. oAuth is about authorization. For example, I can use my credentials in Twitter to gain access to another site. The relationship with oAuth is one app authorizing another app. OpenID on the other hand is about authentication. I have an Openid and I use that as the basis for an application to authenticate who I am – without the need for that application to maintain my security details. With OpenID, it’s one security mechanism used for many applications. In any application, you need to be both authenticated and authorized. Once a user gets access to an application, the process still needs to be secure – especially if HIPPA related type data is presented.

For many applications, OpenID or oAuth can work fine. But for a Web API, it really does not work. First, RESTful apis are just that – RESTful – which means they are stateless. To be effective, we cannot make assumptions from one call to the next. We have to supply credentials with each and every request, whether it is a GET, POST, PUT or DELETE. Like Any API, a Web API should, in my opinion be self sufficient as to controlling how people get to the site. Indeed, things like oAuth and OpenID can work. However, I think the process can be made simpler – with techniques that have been around a while and are as relevant as ever – with all due respect to their newer counterparts.

HTTPS

Of all the things that can be done to make your API, more secure, requiring it to run over HTTPS is the easiest thing to implement. This does nothing for authorization and authentication – which I will get to in a moment. HTTPS however, does afford encryption that is not present in regular HTTP traffic. With HTTPS, your traffic is far less likely to be subject to packet sniffing eavesdropping attacks. The key is, how can you ensure that your Web API is only accessed via HTTPS? The answer lies in one of the most useful features in ASP.NET MVC – that has also been implemented in ASP.NET Web API: Action Filters. And in this case, I’ll make the action filter a global action filter – eliminating the need to manually add the filter to every controller action.

Here is the custom action filter

using System;
using System.Linq;
using System.Net.Http;
using System.Web.Http.Filters;
using System.Web.Http.Controllers;

namespace WebAPI
{
 public class CustomHttpsAttribute : ActionFilterAttribute
 {
  public override void OnActionExecuting(HttpActionContext actionContext)
  {
   if (!String.Equals(actionContext.Request.RequestUri.Scheme, "https", StringComparison.OrdinalIgnoreCase))
   {
    actionContext.Response = new HttpResponseMessage(System.Net.HttpStatusCode.BadRequest)
    {
     Content = new StringContent("HTTPS Required")
    };
    return;
   }
  }
 }
}

Nothing very complicated going on here. If https is not present in the URL, a response is fabricated indicating a bad request (error 400). A message is also added to the response content as well.

As the previous figure illustrates, when we attempt to navigate to the Web API with HTTP, as opposed to HTTPS, a response with status code 400 results – along with a friendly message in the response body that describes the error.

Tokens based on Public/Private Keys

Of all the ways to authorize and authenticate, it seems to me that tokens have done a good at this task. A token can be used to authorize, and for the most part, authenticate a user. Of course, it is possible for a key to get passed around. I’ll address that in the next section. That said, it may be perfectly fine to assume that if a visitor has a properly encrypted token, that visitor is who he/she/it claims to be (in cases where its an application, not a person, it is the right pronoun!!)

The idea behind a private and public key pair is simple. A token is encrypted with the public key. In some cases, the public key sits in the field. In cases where data has to be encrypted by the client to be decrypted on the server, the public key needs to be in the field. The server is the only thing that should have the private key. It’s the private key that handles the decryption. To get this working, we need a simple class to handle our encryption needs:

using System;
using System.Linq;
using System.Security.Cryptography;
using System.Text;

namespace WebAPI
{
 public class RSAClass
 {
  private static string _privateKey = "s6lpjspk+3o2GOK5TM7JySARhhxE5gB96e9XLSSRuWY2W9F951MfistKRzVtg0cjJTdSk5mnWAVHLfKOEqp8PszpJx9z4IaRCwQ937KJmn2/2VyjcUsCsor+fdbIHOiJpaxBlsuI9N++4MgF/jb0tOVudiUutDqqDut7rhrB/oc=AQAB3J2+VWMVWcuLjjnLULe5TmSN7ts0n/TPJqe+bg9avuewu1rDsz+OBfP66/+rpYMs5+JolDceZSiOT+ACW2Neuw==
0HogL5BnWjj9BlfpILQt8ajJnBHYrCiPaJ4npghdD5n/JYV8BNOiOP1T7u1xmvtr2U4mMObE17rZjNOTa1rQpQ==jbXh2dVQlKJznUMwf0PUiy96IDC8R/cnzQu4/ddtEe2fj2lJBe3QG7DRwCA1sJZnFPhQ9svFAXOgnlwlB3D4Gw==evrP6b8BeNONTySkvUoMoDW1WH+elVAH6OsC8IqWexGY1YV8t0wwsfWegZ9IGOifojzbgpVfIPN0SgK1P+r+kQ==LeEoFGI+IOY/J+9SjCPKAKduP280epOTeSKxs115gW1b9CP4glavkUcfQTzkTPe2t21kl1OrnvXEe5Wrzkk8rA==HD0rn0sGtlROPnkcgQsbwmYs+vRki/ZV1DhPboQJ96cuMh5qeLqjAZDUev7V2MWMq6PXceW73OTvfDRcymhLoNvobE4Ekiwc87+TwzS3811mOmt5DJya9SliqU/ro+iEicjO4v3nC+HujdpDh9CVXfUAWebKnd7Vo5p6LwC9nIk=";
  private static string _publicKey = "s6lpjspk+3o2GOK5TM7JySARhhxE5gB96e9XLSSRuWY2W9F951MfistKRzVtg0cjJTdSk5mnWAVHLfKOEqp8PszpJx9z4IaRCwQ937KJmn2/2VyjcUsCsor+fdbIHOiJpaxBlsuI9N++4MgF/jb0tOVudiUutDqqDut7rhrB/oc=AQAB";
  private static UnicodeEncoding _encoder = new UnicodeEncoding();

  public static string Decrypt(string data)
  {
   var rsa = new RSACryptoServiceProvider();
   var dataArray = data.Split(new char[] { ',' });
   byte[] dataByte = new byte[dataArray.Length];
   for (int i = 0; i < dataArray.Length; i++)
   {
    dataByte[i] = Convert.ToByte(dataArray[i]);
   }

   rsa.FromXmlString(_privateKey);
   var decryptedByte = rsa.Decrypt(dataByte, false);
   return _encoder.GetString(decryptedByte);
  }

  public static string Encrypt(string data)
  {
   var rsa = new RSACryptoServiceProvider();
   rsa.FromXmlString(_publicKey);
   var dataToEncrypt = _encoder.GetBytes(data);
   var encryptedByteArray = rsa.Encrypt(dataToEncrypt, false).ToArray();
   var length = encryptedByteArray.Count();
   var item = 0;
   var sb = new StringBuilder();
   foreach (var x in encryptedByteArray)
   {
    item++;
    sb.Append(x);

    if (item < length)
     sb.Append(",");
   }

   return sb.ToString();
  }
 }
}

In this simple RSAClass, I have embedded the public/private RSA Parameters. To create and export new parameters, you would issue the following code outlined in the next image:

[Test]
 public void RSAParameters() {

  var rsa = new RSACryptoServiceProvider();
  var privateParameters = rsa.ExportParameters(true);
  var publicParameters = rsa.ExportParameters(false);

  //Export private parameter XML representation of privateParameters
  //object created above
  Debug.WriteLine(rsa.ToXmlString(true));

  //Export private parameter XML representation of publicParameters
  //object created above
  Debug.WriteLine(rsa.ToXmlString(false));
 }

The following is a simple test that illustrates how to encrypt and decrypt a string with the public/private key pair:

[Test]
  public void VerifyToken() {

   var token = "User1";
   var encryptedToken = RSAClass.Encrypt(token);
   var decryptedToken = RSAClass.Decrypt(encryptedToken);

   Assert.AreEqual(token,decryptedToken);

  }

In this case, the encrypted 128 byte token that results for “User1″ is:

57,46,60,70,93,230,85,33,98,19,10,46,84,91,218,43,207,42,159,
167,5,25,157,4,224,142,235,8,160,199,123,100,107,58,37,204,133,
81,138,196,237,190,56,119,158,7,224,89,84,85,208,169,44,179,102,
218,55,60,76,134,144,22,208,230,165,179,83,125,86,57,224,42,
29,58,188,45,73,33,160,87,165,105,131,139,132,137,209,67,92,36,
168,73,176,205,251,48,240,228,14,39,197,36,42,21,216,242,172,
4,160,234,138,77,156,28,191,63,111,207,221,31,103,213,58,62,
186,123,221,230

You may wish to modify this by having a smaller encrypted token. This is but one approach of many that are possible.

The next thing we need is an action filter to make sure the token is present in each and every request:

using System;
using System.Linq;
using System.Net.Http;
using System.Web.Http.Filters;
using System.Web.Http.Controllers;
using WebAPI.Models;

namespace WebAPI
{
 public class TokenValidationAttribute : ActionFilterAttribute
 {
  public override void OnActionExecuting(HttpActionContext actionContext)
  {
   string token;

   try
   {
    token = actionContext.Request.Headers.GetValues("Authorization-Token").First();
   }
   catch (Exception)
   {
    actionContext.Response = new HttpResponseMessage(System.Net.HttpStatusCode.BadRequest)
    {
     Content = new StringContent("Missing Authorization-Token")
    };
    return;
   }

   try
   {
    AuthorizedUserRepository.GetUsers().First(x => x.Name == RSAClass.Decrypt(token));
    base.OnActionExecuting(actionContext);
   }
   catch (Exception)
   {
    actionContext.Response = new HttpResponseMessage(System.Net.HttpStatusCode.Forbidden)
    {
     Content = new StringContent("Unauthorized User")
    };
    return;
   }
  }
 }
}

There are two levels of checking here. The first is whether or not the authorization token is present in the header. If not, a bad request response (error 400) is returned. If the token was provided but does not resolve to an authorized user, then an unauthorized response (error 401) is returned. The following illustrates how to setup Fiddler with the Authorization-Token in the header:

The following illustrates the results of the call in the previous image:

For purposes of this demo, I established a simple authorized user repository that has a list of pre-defined users:

using System;
using System.Collections.Generic;
using System.Linq;

namespace WebAPI.Models
{
 public class AuthorizedUserRepository
 {
   public static IQueryable GetUsers() {

      IList users = new List();
      users.Add(new User("User1"));
      users.Add(new User("User2"));
      users.Add(new User("User3"));
      users.Add(new User("Administrator"));

      return users.AsQueryable();
   }
 }
}

Within this approach, there are all sorts of variants you could employ. For example, you could swap out the public/private keys on some periodic basis. Of course, that means you will have to issue new tokens to users who have access to your API!!

IP Filtering

Up to this point, while the system is pretty secure, it can be locked down further. Right now, tokens could be moved from location to location. It may be that you only want to authorize traffic from certain IP addresses and hosts. For purposes of this demo, I created a simple Authorized IP Repository:

using System;
using System.Collections.Generic;
using System.Linq;

namespace WebAPI.Models
{
 public class AuthorizedIPRepository
 {
    public static IQueryable GetAuthorizedIPs() {

      var ips = new List();

      ips.Add("127.0.0.1");
      ips.Add("::1");

      return ips.AsQueryable();
    }
 }
}

In actual practice, I might take the extra step of linking specific tokens to a specific IP range. I’ll leave that exercise to you! The following is the action filter that restricts traffic from a defined IP list:

using System;
using System.Linq;
using System.Web.Http.Controllers;
using System.Web.Http.Filters;
using System.Net.Http;
using WebAPI.Models;

namespace WebAPI
{
 public class IPHostValidationAttribute : ActionFilterAttribute
 {
  public override void OnActionExecuting(HttpActionContext actionContext)
  {

   var context = actionContext.Request.Properties["MS_HttpContext"] as System.Web.HttpContextBase;
   string userIP = context.Request.UserHostAddress;
   try
   {
    AuthorizedIPRepository.GetAuthorizedIPs().First(x => x == userIP);
   }
   catch (Exception)
   {
    actionContext.Response =
       new HttpResponseMessage(System.Net.HttpStatusCode.Forbidden)
       {
        Content = new StringContent("Unauthorized IP Address")
       };
    return;
   }
  }
 }
}

To make these action filters global, the following code in the Global.asax Application_Start() will do the trick:

var config = GlobalConfiguration.Configuration;
config.Filters.Add(new TokenValidationAttribute());
config.Filters.Add(new CustomHttpsAttribute());
config.Filters.Add(new IPHostValidationAttribute());

If you wished to initiate a request from your C# code, you might have a utility method like this:

  WebRequest getRequest(string method, string contentType, string endPoint)
  {
   var request = WebRequest.Create(endPoint);
   request.Method = method;
   request.ContentType = contentType;

   ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };

   request.Headers.Add("Authorization-Token", "57,46,60,70,93,230,85,33,98,19,10,46,84,91,218,43,207,42,159,167,5,25,157,4,224,142,235,8,160,199,123,100,107,58,37,204,133,81,138,196,237,190,56,119,158,7,224,89,84,85,208,169,44,179,102,218,55,60,76,134,144,22,208,230,165,179,83,125,86,57,224,42,29,58,188,45,73,33,160,87,165,105,131,139,132,137,209,67,92,36,168,73,176,205,251,48,240,228,14,39,197,36,42,21,216,242,172,4,160,234,138,77,156,28,191,63,111,207,221,31,103,213,58,62,186,123,221,230");

   return request;
  }

This request has the necessary information to make an API request

That’s pretty much it. Now we have a ASP.NET Web API that requires requests to be under the HTTPS protocol, requires an encrypted authorization token and requires traffic to only come from a predefined population of IP addresses. There are any number of ways you can vary this implementation. My goal with this post was to make it easy to get started. With this approach, you don’t have to worry about cumbersome query string parameters, etc. In this example, I am putting authentication details where they belong – in the header – which means it will work for anything. No need to deal with clumsy query string parameters either.

Enjoy!!

Continuous Testing

gregyoung — Mon, 02 Apr 2012 11:53:22 +0000

On my build times post from “Don”

And you want to run your tests _every time_ you hit ctrl+s !?

Actually I have run into this viewpoint quite a bit. I see it most commonly from people who are not doing TDD. The reasoning seems to be that in the process of working on a feature they may save 20 times over a few hours then try to build. The “Save” mode is 100% for TDD workflows. If you are not doing TDD workflows then it probably is not for you. That said after having this discussion many times there are other options that better match non-TDD workflows. Let’s go through all four workflows.

Realtime / Saving modes

The realtime (as you type) and saving (when you save) modes are the default modes and are focused on people who are interested in very short TDD feedback cycles. Both automated both your build and will figure out which tests to run and run them after the build is completed. If you type (or save over the top) existing runs are fully or partially cancelled depending where they are in the run.

Auto

Auto will no longer automate your builds. Basically it waits for you to build in Visual Studio and when it succeeds will automate the finding and running of the necessary unit tests. This uses the build in VS as the “commit”, not the saving of file.

Manual

The last mode that is available is Manual mode. In this mode no automation will occur (except for us seeing your builds and keeping our graphs updated). The software will not automate any builds or tests.

Many people have asked “Why does manual mode exist?”.

Manual mode actually has a fairly powerful workflow associated with it that may be beneficial in some scenarios that are not good candidates for continuous testing. In this mode Mighty Moose works just like most manual test runners with a small twist there is a feature called “Run Related Tests” (ctrl+shift+y,r).

Run Related Tests is quite interesting for those who are not in a situation to benefit from continuous testing. Let’s try a quick walk through.

type:

[Test]
public void MyFirstTest() {
Assert.AreEqual(Foo.Bar(3));
}

static class Foo {
public static int Bar(int x) { throw new UnimplementedException(); }
}

Now hit ctrl+shift+y,u with the cursor inside of the test. It will tell you that the test has run (failed).

With cursor on “Bar” f12 (goto definition)… replace code with

return 3;

from current cursor location ctrl+shift+y,r (run related tests). 1 test run one test passes. Imagining they are in different files now use ctrl+f6 and back in your tests. You can just keep flipping back and forth like this.

REST Resources

Howard Dierking — Tue, 27 Mar 2012 19:11:52 +0000

I got a question recently about my recent Pluralsight REST Fundamentals course. Several times throughout the course, I say something to the effect of “for more on X, check out the references section at the end of the module”. The question was – where are the references. The short answer is that each module has a PDF of the slides used – and the last slide is for references. However, to save you the trouble of going through each of the PDFs to extract the references, I’ll just provide a consolidated list for the entire course here. Hope this is helpful whether or not you check out the course – though I would love to get your thoughts on that as well!

Fielding, Roy Thomas. Architectural Styles and the Design of Network-based Software Architectures. Doctoral dissertation, University of California, Irvine, 2000.
HTTP/1.1 Specification
Rotem-Gal-Oz. Fallacies of Distributed Computing Explained.
Amundsen, Mike. Building Hypermedia APIs with HTML5 and Node. O’Reilly Media, 2011.
URI Template Specification Draft
Fiddler
cURL
HAL Media Type Specification
RESTBugs Sample Application
http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven
Varia, Jinesh. Architecting for the Cloud: Best Practices.

Charge for bad code?

gregyoung — Tue, 27 Mar 2012 12:14:04 +0000

So let’s throw out a quick idea that we have batted around a bit internally. What if Mighty Moose were to be free? What if we were only to charge you for two features inside of the software that are not needed if you are maintaining a decent code base?

The basic idea is if you write good code, you will never need these features and thus Mighty Moose will be free for you (We will even throw in support!). If however you need these two areas you will pay for them.

The two areas in question are the minimizer (the thing that figures out which tests to run) and the cascading + incremental compilation build providers. If your tests are fast and your builds small, mighty moose is painless to use without these features. You would still get graphs, sequence diagrams, all the unit testing frameworks, gary etc. Its just it would build your solution and run “all” your tests (you could still use ignore + ignore categories etc)

We could look at this as being similar to a cigarette tax (cost shifting). We provide people with good code the tool for free by charging people who need the features that help them in bad situations. There is also a large support cost difference we have measured (quite similar to smokers using up more medical resources). We would be helping push you towards doing the right thing. We would of course also provide you guidance about how to fix these issues in your system. If however you choose to take the easy way out and have the tool alleviate your pain, there would be a cost associated with that decision.

From a managerial perspective we are also giving you a bit of ammunition. I can make the case very easily as to why these two problems are very expensive within an organization. This would provide setting a cost to one solution that can then be used in further decision making. From an options pricing model perspective we are pricing the option of not resolving the issues now.

I understand that this is a fairly innovative idea and may seem odd to some people at first so I wanted to drop it up here as a RFC to get some feedback.

How to make your build faster

gregyoung — Mon, 26 Mar 2012 13:59:43 +0000

I did an informal survey a while ago and the results were frankly abysmal. I cannot understand how people work in such environments. I am ADD, if I have a feedback cycle more than a few seconds I have already hit ctrl+right arrow and am reading my email or sitting in twitter. This however has a very serious affect on my work. I am constantly pushing and popping my mental stack of what was going on (more often than not having cache misses as well). The sample size was pretty decent as well with over 700 respondents.

This post is going to look at some ways we can make our build times faster. There are loads of ways aside from buying SSDs. I work in a VM on a non-SSD drive on a laptop and rarely see a build over 5 seconds mostly because I focus very heavily on not having solutions with 20,000 projects in them but there are some other ways as well.

Project Structures

To start with we can use smarter project layouts. Patrick Smacchia has some good posts on it:

Advices on partitioning code through .NET assemblies

Hints on how to componentize existing code

Keeping well factored projects can go a long way and making a larger project will be faster than building many smaller ones (why? as an example they have a tendency of reusing many of the same references etc that need to be loaded repeatedly causing far more i/o).

Parallel Builds

These are good reading to start but there are many more options that are available. Let’s start with one of the comments from Ben “I think unless you go down the parallel build/test route (like NCrunch) then this issue of build times is not going to go away.”

What a great selling point. Luckily for builds everyone can already do this. Did you know that Visual Studio and MSBuild already do parallel builds? All you have to do is drop into your configuration and you will see an options screen like the one here

Put in your max number of cores that you want. Work with msbuild? just use /maxcpucount:4. Of course this is still rather limiting as if you have project 1 that references project 2 which references project 3 you are in a serial situation anyways. The maxcpucount represents the number of projects that can be built concurrently (e.g.: providing no dependencies). Both MM and nCrunch support this option as well. This can get your build times to be quicker in some scenarios though I tend to see only a +- 20-25% increase on average.

Mighty Moose

We did a quick survey on Mighty Moose users about a week ago and I was amazed to see that very few people were using this feature (about 10% of a survey size +- 100-150 people). Mighty Moose has a smart build system inside of it that is actually much better than what msbuild can do because it has more information. I know dotnetdemon from red gate has something similar to it.

Basically since I know what is changing and I know how that affects the dependency graph I can be much more intelligent with my builds. I also have further knowledge through static analysis about what the affects of that change was (are public contracts changed etc?). Of course its a bit more complicated than this in practice (lots of edge cases) but it can make a huge difference in your build times for your typical builds in your TDD cycle.

[youtube: 05Z5sPrGEeo]

You can see the difference in the video above. MSBuild on an incremental build in AutoTest.Net about 4.5 seconds. MM for the same build + locate tests to run + run the tests about 2.5-3 seconds. AutoTest.Net itself is not a tiny piece of code either. It has a reasonable amount of code in it, even the incremental build is doing pretty well when you consider my dev machine is a VM on a non-SSD drive. This is because we have more information than msbuild does as we stay running compared to it running then dying. You can enable the setting on the first screen in config (Build Setup) here.

Some people have asked if its so much faster why do we make you opt in to using it? There are a few reasons for this. The largest is compatibility mode. Most people do not generally build their projects directly and often have reasons why they don’t actually build successfully. Building the solution gives the highest level of compatibility with existing VS projects. As such we make you opt in for it, but its worth opting in for!