$.get("http://twitter.com/status/user_timeline/SCREEN_NAME.json",
  function(tweets){
    // work with tweets
   },
  "jsonp");

(For the purposes of this example, I’m going to assume there is no way to get the tweets for multiple users in a single call)

To get the tweets for three users, you would need to make three separate $.get calls to the user_timeline endpoint. Since each call is executed asynchronously, with no guarantee which would return first, the code to coordinate the response for all three users would likely be a mess of shared state and/or nested callbacks.

As of jQuery 1.5, the solution is much simpler. Each of the ajax functions were changed to return a Deferred object which manages the callbacks for a call. (The Deferred object is beyond the scope of this post, but I encourage you to read the documentation for a more thorough explanation.) The power of Deferred objects become apparent when used with the new jquery.when utility function. jquery.when accepts any number of Deferred objects, and allows you to assign callbacks that will be invoked when all of the Deferred objects have completed. The Deferred objects returned by the ajax functions are complete when the responses are received. This may sound confusing, but it should be much clearer when you see it applied to the example scenario:

$.when( getTweets('austintexasgov'),
        getTweets('greenling_com'),
        getTweets('themomandpops')
      ).done(function(atxArgs, greenlingArgs, momandpopsArgs){
    var allTweets = [].concat(atxArgs[0]).concat(greenlingArgs[0]).concat(momandpopsArgs[0]);
    var sortedTweets = sortTweets(allTweets);
    showTweets(sortedTweets);
      });

});

var getTweets = function(user){
    var url='http://twitter.com/status/user_timeline/' + user + '.json';
    return $.get(url, {count:5}, null, 'jsonp');
}

// see it in action at http://jsfiddle.net/94PGy/4/

• I have a helper method, getTweets, which returns the return value of a call to $.get. This will be a Deferred object representing that call to the twitter server.
• I call $.when, passing it the three Deferred objects from the three ajax calls.
• The done() function is chained off of $.when to declare the code to run when all three ajax calls have completed successfully.
• The done() function receives an argument for each of the ajax calls. Each argument holds an array of the arguments that would be passed to that ajax call’s success callback. The $.get success callback gets three arguments: data, textStatus, and jqXHR. Therefore, the data argument from the call for @greenling_com tweets is available in greenlingArgs[0]. Similarly, the textStatus argument for the call for @austintexasgov tweets would be in atxArgs[1].
• The fifth line creates the allTweets array combining the tweets (the first, or data, argument) from all three calls to twitter.

It it that last point that is interesting to me. I’m able to work with a single collection containing data from three separate ajax requests, without writing any awkward synchronization code.

Play with the example on jsFiddle

Fortunately, Robert Moore and Miguel Madero have already taken the code, fixed it to work with the latest libraries, and released it as NQUnit. They’ve even packaged it up for easy consumption via Nuget:

PM> Install-Package NQUnit

My team recently dumped our implementation in favor of NQUnit. Thanks, and great job guys!

Some Nuget packages include executables in their tools folder. It is very easy to use these tools within Visual Studio because Nuget makes them available in the path of the Package Manager Console. However, they are very difficult to use outside of Visual Studio, especially in a build script. The problem is the name of the folder containing the installed package includes the version number of the package. If you install NUnit 2.5.1, the nunit-console.exe will be in packages\NUnit.2.5.1\tools. However, if you later upgrade to NUnit.2.5.2, the path to nunit-console.exe will change to packages\NUnit.2.5.2\tools. You will need to change your build scripts every time you upgrade your version of NUnit. That is unacceptable.

The solution is to create a helper that can figure out where the tool lives. If you are using rake for build automation, it is fairly straightforward:

# Usage: nunit_path = tool "NUnit", "nunit-console.exe"

def tool(package, tool)
  File.join(Dir.glob(File.join(package_root,"#{package}.*")).sort.last, "tools", tool)
end

If not, you may want to create a batch file in the root of your project that calls your tool. You can create a tool specific batch file:

@ECHO OFF
SETLOCAL
REM You might prefer specific scripts for each tool. Makes the usage a bit easier. This is an example for NUnit.

FOR /R %~dp0\source\packages %%G IN (nunit-console.exe) DO (
  IF EXIST %%G (
    SET TOOLPATH=%%G
    GOTO FOUND
  )
)
IF '%TOOLPATH%'=='' GOTO NOTFOUND

:FOUND
%TOOLPATH% %*
GOTO :EOF

:NOTFOUND
ECHO nunit-console not found.
EXIT /B 1

Or, if you have lots of tools from different packages, you might just want a generic batch file that allows you to specify the executable name:

@ECHO OFF
SETLOCAL
REM This can be used for any .exe installed by a nuget package
REM Example usage: nuget_tool.bat nunit-console.exe myproject.tests.dll
SET TOOL=%1
FOR /R %~dp0\source\packages %%G IN (%TOOL%) DO (
  IF EXIST %%G (
    SET TOOLPATH=%%G
    GOTO FOUND
  )
)
IF '%TOOLPATH%'=='' GOTO NOTFOUND

:FOUND
%TOOLPATH% %2 %3 %4 %5 %6 %7 %8 %9
GOTO :EOF

:NOTFOUND
ECHO %TOOL% not found.
EXIT /B 1

I’ve spent a lot of time over the past couple weeks building and consuming Nuget packages on real projects. I’ve picked up a few tips that, while not mind-blowing, should help you get beyond the simple scenarios you see in most demos.  I’ll refer to an example .nuspec from one of my projects, so you may want to keep it open in your browser.

Do not store the version number in your nuspec file

I assume you already had a build script before Nuget came. You already have a way to generate and store the version of your build output. For example, I tend to store the first three components of the version in a VERSION.txt file in my code repository. It gets manually incremented when appropriate (see semver). The fourth component is the build number generated by my continuous integration server. This information is already used to generate my AssemblyInfo.cs file, artifacts package name, etc. It doesn’t make any sense for me to repeat that information in my .nuspec file, so set the version tag to 0.0.0 (see line 5) to remind yourself that it is not used. Use the -Version option of the nuget.exe pack command to provide the actual version.

Prefer the <files> element approach to adding files to your package

Again, you already have a build script, and it likely copies all of your build output to a known location. It doesn’t make sense to copy the files around even more to build the structure required by Nuget’s “conventional folders” approach. Instead, use the <files> section of your .nuspec (see line 20) to refer to the files in your build output folder. The powerful wildcard support makes it easy to include everything you want, including source code, as suggested by the next tip…

Publish symbols packages for your Nugets

Nuget has integrated support for building and publishing symbol packages to symbolsource.org. These symbol packages will greatly enhance the debugging experience for anyone that uses your library, and since its so easy, there is no reason for you to not publish them (assuming your library is open source).

1. Include .pdb files for the files in the package’s lib folder (see line 22)
2. Include all of the corresponding source files in the package’s src folder. Using the filespec wildcard support, this can be one line (see line 23).
3. Pass the -Symbols flag to the nuget.exe pack command when building your package. It will create two files: yourpackage.nupkg and yourpackage.symbols.nupkg.
4. (optional, but recommended) Register at symbolsource.org and associate your Nuget.org API key with your account.
5. A single call to nuget.exe push yourpackage.nupkg will upload yourpackage.nupkg to nuget.org and yourpackage.symbols.nupkg to symbolsource.org.

SymbolSource has good instructions on how package consumers can configure Visual Studio to use the symbol packages.

Provide multiple packages that target different usage scenarios

A single source code repository may produce many build artifacts, but if they aren’t always used together, they shouldn’t be distributed via Nuget together. Instead, create a single Nuget package that targets each scenario. A perfect example of what not to do is the NUnit.2.5.10.11092 package (I don’t mean to pick on NUnit, it was probably created quickly by a volunteer just to get it out there. That’s cool, I get it. If anyone from the NUnit team wants my help fixing the packages, just contact me). When you install NUnit via Nuget, you get nunit.framework.dll, nunit.mocks.dll and pnunit.framework.dll added to your project. I would guess that 90% of users just want nunit.framework.dll. Similarly, the tools folder contains a number of executables, but I bet most people just want nunit-console.exe. I would break this up into an NUnit package (nunit.framework.dll in lib and all files needed to run nunit-console.exe in tools), an NUnit.Mocks package (nunit.mocks.dll in lib and has the NUnit package as a dependency), and an NUnit.Parallel package (all of the pnunit files with a dependency on NUnit if needed). The bottles.nuspec in my example is just one of many packages produced by the bottles source code repository.

Create core packages that work outside of Visual Studio

The powershell scripts and web.config modifications are cool, but if they are not absolutely necessary to use your library, provide a “core” package that does not include them. You can always provide a “quickstart” package with these extras, and have it depend on your core package to bring in the meat.

Automate building and publishing your packages

If you follow my previous two tips, you’ll have multiple .nuspec files to deal with. Fortunately, it is easy to add a single step to your build process to create all of your packages. I have a gist with a couple examples to get you started (feel free to fork to improve or add more examples). It should be fairly straightforward to make new scripts to publish your packages based on those examples.

I feel pretty strongly about only pushing “public CI builds” to the official Nuget feed, for traceability. This makes the publishing step a little more complex, but nothing that can’t be handled in a rake script. I’ve got an example that downloads the .nupkg files built on http://teamcity.codebetter.com and then publishes them.

Create an icon for your package

The icon will show up on nuget.org and in the Visual Studio Nuget dialog. Not many people do it (right now), so your packages will stand out. Let GitHub serve your icon file – just browse to the icon in your repo on GitHub, and then copy the link for the “raw” url and include it as your <iconUrl> metadata. Might as well do the same for your <licenseUrl>.

Install Nuget Package Explorer

This probably should have been the first tip, its that useful. Makes it much easier to view the contents and metadata in your local .nupkg files, and download packages from nuget.org. Invaluable when designing your packages. It even lets you edit your packages, though I’ve never tried that.

I recently started getting to know Microsoft’s Nuget package management tool. I’ll admit I was going into it expecting to be disappointed, still annoyed that it had effectively killed the nu project and marginalized OpenWrap – two projects in the same problem space, but from the community, that I thought had promise. However, after an hour or so of playing with it – pulling packages into a project, authoring my own package – I came away impressed. It worked and it was easy.

Of course, it wasn’t too long before I ran into a pain point. I wanted to create a nuget package as part of my continuous integration build. A nuget package needs a version number that increases with every release. So it stands to reason that every time you build the package, you are going to give it a new version number. The problem was that the version number could only be specified in the .nuspec file. The .nuspec file describes how to build the package, mostly static information that should be checked into source control. The idea that I would have to modify this file every time I built seemed absurd (because it was). I had found it! A reason to complain about nuget! Microsoft just doesn’t get it! Wait until twitter hears about this!

A New Hope

And then I remembered there was something very different about the nuget project: while it is primarily maintained by Microsoft employees, it accepts contributions from the community. Suddenly, the barbs I was about to throw wouldn’t apply just to the big bully Microsoft, but my own .NET developer community. As a core member of many open source projects over the years, I know how limited resources are, and that feature prioritization is usually based on the needs of the people willing to do the work. If I thought it was important, it was time to step up. So I logged an issue. It got a few nods of approval, so I submitted a pull request. And then the strangest, most wonderful, the big-ship-is-turning thing happened: they accepted it. Within two days of discovering this twitter-rant worthy, how-could-they-not-think-of-this-scenario, nuget-is-doomed problem, it was fixed. Huh.

Fast forward a few weeks. Old prejudices die hard. I’m trying to take advantage of the very cool symbolsource.org integration (I think the lack of an approachable public symbol server until now has been as big a hole as not having a package manager). It becomes painfully clear that the nuget support for symbol package creation is only viable for the most simplest scenarios. I was annoyed and exhausted from a long night of fighting the tool, so was a little quicker on the twitter trigger (although still much tamer than I felt):

Then, remembering my last experience, I figured I would at least start a discussion before giving up for the night. To my surprise, the next day it was turned into an issue – this isn’t just another Microsoft Connect black hole. After hashing out a few details, I went to work on a solution and submitted a pull request. It was accepted within a few days. Aha! This is open source. This is how its supposed to work. This works.

The Nuget project is the most exciting thing to come out of Microsoft DevDiv since the .NET Framework was released. I think it is very important to the .NET open source community that it succeeds. Not just for the obvious benefit of simplifying distribution for all of the non-Microsoft open source solutions. But for the potential it has to change minds internally at Microsoft about how community collaboration can work. And for the potential it has to change minds in the community, about how collaboration with Microsoft can work.

Epilogue: The alternative

Ever read the What If…? comic books? From wikipedia:

What If stories usually began with [the narrator] briefly recapping a notable event in the mainstream Marvel Universe, then indicating a particular point of divergence in that event. He would then demonstrate, by way of looking into a parallel reality, what could have happened if events had taken a different course from that point.

I’ve been thinking a lot lately about “What if… Microsoft had launched ASP.NET MVC as a collaborative project that accepted community contributions the same way that Nuget does?”

Would FubuMVC exist? I’m not sure it would, and I’m not sure that would be a bad thing. We, the FubuMVC team, started out building a new product on ASP.NET MVC, adopting Microsoft’s framework before it even went 1.0. We inevitably ran into some pain points that we had to work around. Of course, the workarounds had to be implemented in our application codebase because Microsoft did not want them and/or could not take them. As we built up more and more workarounds, we relied less and less on the out-of-box functionality. It starts with a custom action invoker. Which leads to custom model binding. Then custom HTML helpers. Route creation and URL resolution. After a year and a half of this, it became clear we were building our own framework. During a few days of a long Christmas vacation, Chad Myers went the last mile and extracted our changes into a google code project that eventually became FubuMVC.

Now consider what might have happened if Microsoft accepted contributions to MVC. Remember, we didn’t decide from day 1 to write a new framework – we had a product to build. As each issue came up, we could have submitted it as a patch. Each patch on its own wouldn’t have been a revolutionary change. But the motivations behind them may have started to sneak into the consciousness of everyone working on the codebase. You see, we, and anyone else that may have contributed, had a major advantage over the Microsoft contributors: we were building real applications with this framework.

Now, I don’t think the end result would have been a framework with all of FubuMVC’s traits (requiring an inversion of control container would have been DOA). But certainly some of the core ideals (composition over inheritance, leveraging the strengths of the type system) could have snuck in. And instead of going our own way, we could have benefited from all of Microsoft’s work: Visual Studio tooling, razor view engine support, documentation, and an overall thoroughness you get from paid employees working on it full time. And of course its not just us – ideas and code from projects like OpenRasta, NancyFX, and OWIN could have snuck in too.

I think that was a major missed opportunity. The Nuget project is proving there is a better way.

> Install-Package HtmlTags

HtmlTags provides a simple object model for generating HTML from .NET code. The main premise is that you are much better off building a model of the HTML elements you want to produce (and serializing it to a string when needed), rather than building a string directly. It was created by Jeremy Miller as part of FubuMVC to support the awesome conventional HTML features I’ve talked about before. I then pulled it out of FubuMVC and tightened up the API for a more general release. I strongly believe that regardless of which .NET web framework you are using, if you are generating snippets of HTML (think ASP.NET MVC HtmlHelper extensions), you should use HtmlTags.

Check out the readme for an overview and example usage of HtmlTags.

Big thanks to Sharon Cichelli for helping me put the final touches on the codebase for the 1.0 release, and to Robert Greyling for the sharp new logo!

The code is on github: https://github.com/joshuaflanagan/adnug_Feb2011

The master branch has the state of the code before I started the talk. The live branch has the code after I was done.

If you aren’t familiar with git, you can download the two branches as zip files:

master: https://github.com/joshuaflanagan/adnug_Feb2011/archives/master

live: https://github.com/joshuaflanagan/adnug_Feb2011/archives/live

The sample application is built on ASP.NET MVC 3. It demonstrates a simplified version of the convention-based view helpers from FubuMVC. The view helpers make use of the HtmlTags library (now available via nuget), which I recommend you use whenever building HtmlHelper extensions (for ASP.NET MVC or FubuMVC).

If you are curious, I also have a version of the same sample application that was built on FubuMVC:

https://github.com/joshuaflanagan/leveraging-conventions

That code includes a readme with basic setup instructions, as well as notes.txt which contains an outline of the steps I followed during the presentation.

As always, feel free to contact me with any questions or feedback.

In our application, we have standard CRUD endpoints for each of our top level entities: New/Create/View. The view endpoint for a case is case/GUID. To properly implement the sensitive cases feature, we had to make sure that a user without the "View Sensitive Cases" would get an access denied error if they attempted to load the page for a sensitive case.

A side note about FubuMVC authorization

FubuMVC has a very powerful configuration model for declaring and decorating endpoints (routes/web pages). I’m not going to dive deep into the configuration, but you can read Jeremy Miller’s introduction. When configuring an endpoint, if you attach any authorization policies (IAuthorizationPolicy), an authorization behavior will be added to the endpoint to execute your policies at runtime. When a request comes in to the endpoint, each policy gets a chance to vote on whether the request can continue. If the policies do not vote to Allow, the request is interrupted and an HTTP 403 is returned to the client.

RestrictedDataAuthorizationPolicy

We already have a class that defines whether a user can view a case – SensitiveCaseDataRestriction, which I introduced in my post about filtering at the data access level. In the context of a web request to view a Case, we have a single Case instance. If we could execute the data restrictions against the given Case, we could determine if the request should continue or return a 403. That is exactly what the RestrictedDataAuthorizationPolicy does:

public class RestrictedDataAuthorizationPolicy<T> : IAuthorizationPolicy where T : DomainEntity
{
    private readonly IEnumerable<IDataRestriction<T>> _dataRestrictions;

    public RestrictedDataAuthorizationPolicy(IEnumerable<IDataRestriction<T>> dataRestrictions)
    {
        _dataRestrictions = dataRestrictions;
    }

    public AuthorizationRight RightsFor(IFubuRequest request)
    {
        var entityFilter = new SingleEntityFilter<T>(request.Get<T>());
        _dataRestrictions.Each(entityFilter.ApplyRestriction);
        return entityFilter.CanView ? AuthorizationRight.None : AuthorizationRight.Deny;
    }
}

It takes advantage of the fact that IDataRestrictions operate against the IDataSourceFilter<T> interface. In the data access context, the IDataSourceFilter implementation would build a SQL WHERE clause. In this context, we already have an entity instance (the call to request.Get<T>() in line 12 above), so we just need to see if it "passes" all of the rules. The SingleEntityFilter class is an implementation of IDataSourceFilter that keeps track of a single boolean value: CanView. It is set to true initially, but successive calls to ApplyRestriction() could set it to false if the entity does not meet the requirements of an IDataRestriction. After all of the data restrictions are applied, the CanView property tells us if we need to deny access to the current web request.

Applying the policy

The final step is to wire it all up together. We need to attach the RestricedDataAuthorizationPolicy to the appropriate endpoints (the "View" routes for our entities). FubuMVC allows us to define a convention so that all of the appropriate endpoints have the policy applied automatically. The implementation details of a FubuMVC convention are beyond the scope of this post, but I’ll include the code to give you a hint at the possibilities. It isn’t the most succinct code, but it should give you an idea of how powerful it is to be able to query and modify the composition of your endpoints at startup time.

public class AuthorizeByDataRestrictionsConvention : IConfigurationAction
{
    public void Configure(BehaviorGraph endpoints)
    {
        // "view" actions are declared on classes that implement CrudController interface
        // ex: CaseController : CrudController<Case>
        var viewEntityActions = endpoints.Behaviors.Select(x => x.FirstCall())
            .Where(x => x.HandlerType.IsCrudController())
            .Where(x => x.Method.Name == "View");

        foreach (var action in viewEntityActions)
        {
            var entityType = action.HandlerType.FindInterfaceThatCloses(typeof(CrudController<>)).GetGenericArguments()[0];
            var endpoint = action.ParentChain();

            var policyType = typeof(RestrictedDataAuthorizationPolicy<>).MakeGenericType(entityType);
            endpoint.Authorization.AddPolicy(policyType);
        }
    }
}

ToStartOfTimeZonesDayInUtc

The name is clunky because its really hard to describe succinctly. Given a time (in UTC) and a time zone, I want to know the time (in UTC) that the day started in that time zone. This is useful, for example, when you want to display a list of events, grouped by day, in the user’s time zone. If an event happened at 3:05am UTC on February 4th, it should be grouped in Feburary 4th when displayed to a user in Western European Time, but grouped in February 3rd for a user in Central Time (US).

/// <summary>
/// The time in UTC that the day started in the given time zone for a specific UTC time
/// </summary>
/// <param name="utcTime">A point in time, specified in UTC</param>
/// <param name="timezone">The time zone that determines when the day started</param>
/// <returns></returns>
public static DateTime ToStartOfTimeZonesDayInUtc(this DateTime utcTime, TimeZoneInfo timezone)
{
    var startOfDayInGivenTimeZone = utcTime.ToLocalTime(timezone).Date;
    return startOfDayInGivenTimeZone.ToUniversalTime(timezone);
}

StartOfTodayInUtc

A convenience method for determining the day, right now, in a given time zone.

/// <summary>
/// The time in UTC that the current day started in the given time zone
/// </summary>
/// <param name="timezone">The time zone that determines when the day started</param>
/// <returns></returns>
public static DateTime StartOfTodayInUtc(this TimeZoneInfo timezone)
{
    return DateTime.UtcNow.ToStartOfTimeZonesDayInUtc(timezone);
}

ToLocalTime and ToUniversalTime

These are pretty straightforward but I’ll include them because the methods above depend on them. DateTime already has ToLocalTime and ToUniversalTime methods, but they are flawed because they do not let you specify a time zone. I’m not sure why they were not added as overloads when the TimeZoneInfo type was introduced.

/// <summary>
/// Converts a UTC time to a time in the given time zone
/// </summary>
/// <param name="targetTimeZone">The time zone to convert to</param>
/// <param name="utcTime">The UTC time</param>
/// <returns></returns>
public static DateTime ToLocalTime(this DateTime utcTime, TimeZoneInfo targetTimeZone)
{
    return TimeZoneInfo.ConvertTimeFromUtc(utcTime, targetTimeZone);
}

/// <summary>
/// Converts a local time to a UTC time
/// </summary>
/// <param name="sourceTimeZone">The time zone of the local time</param>
/// <param name="localTime">The local time</param>
/// <returns></returns>
public static DateTime ToUniversalTime(this DateTime localTime, TimeZoneInfo sourceTimeZone)
{
    return TimeZoneInfo.ConvertTimeToUtc(localTime, sourceTimeZone);
}

The key to its flexibility is the IDataSourceFilter<T> parameter passed to the data restriction at runtime. For a quick reminder, look at the implementation of the "sensitive case" data restriction:

public class SensitiveCaseDataRestriction : IDataRestriction<Case>
{
    public void Apply(IDataSourceFilter<Case> filter)
    {
        if (!PrincipalRoles.IsInRole(DomainActions.ViewSensitiveCases))
        {
            filter.WhereEqual(x => x.IsSensitive, false);
        }
    }
}

The data restriction performs actions on the filter passed to it. What effect that filter ultimately has on the system is not known, nor a concern of the data restriction itself. That is the role of the data restriction consumer, who determines which type of implementation of IDataSourceFilter<T> to provide. Think of it like LINQ – there are a set of operations (the extension methods on IEnumerable) which can be transparently applied to a number of sources (the LINQ data providers). In fact, we considered passing in an IQueryable<T> as the parameter to the Apply() method, but quickly realized that opened up way more operations than we intended to support. Like LINQ, we have one implementation of IDataSourceFilter that alters SQL queries (as I demonstrated in the last post), and we have another implementation that simply operates on objects in memory (which I intend to demonstrate in a future post).

The bottom line is that data restrictions represent behavior of the domain (like "only users with the "view sensistive cases" privilege should see cases marked as sensitive"), and live in the domain. The fact that we were able to easily leverage these rules in our data access code is a pleasant bonus.