Since then, it was included on Sengrid’s library list and so it means we’re famous right ..? Also, people wrote about it on stack overflow so we’re def famous!

Sendgrid Webhooks

Sendgrid is a 3rd party service that provides email delivery on your behalf, providing you with an API. On top of that it can give you a functionality, where every user action can result in a callback to your endpoint. This effectively means that every time an email is processed, send, open, clicked, spam reported, unsubscribed or bounced - it will call your servers with additional details. I blogged about creating reporting from the sendgrid webhooks earlier on.

If you’re serious about your channel this is super important, because you get info such as clicked url, IP address or the user-agent of the client. Plus, you can pass custom arguments along with the API calls as email meta-data that make it to the callbacks as well.

Parse the Webhook Callbacks

I wanted to provide a library that would take the JSON in the callback and turned that into a polymorphic list of Sendgrid Webhook Events. Sendgrid tends to aggregate the callbacks into batches so you usually receive more than 1 event in a single callback.

[
      {
        "email": "john.doe@sendgrid.com",
        "timestamp": 1337197600,
        "smtp-id": "<4FB4041F.6080505@sendgrid.com>",
        "event": "processed"
      },
      {
        "email": "john.doe@sendgrid.com",
        "timestamp": 1337966815,
        "category": "newuser",
        "event": "click",
        "url": "https://sendgrid.com"
      }
    ]

Using the library

The library allows you to easily deseralise the callbacks and create a list of typed events of a common base. All custom fields, date and list conversions are handled for you.

var parser = new WebhookParser();
    var events = parser.ParseEvents(json);

    var webhookEvent = events[0];

    // Shared base properties
    webhookEvent.EventType; // Enum - type of the event as enum
    webhookEvent.Categories; // IList<string> - list of categories assigned ot the event
    webhookEvent.TimeStamp; // DateTime - datetime of the event converted from Unix time
    webhookEvent.UniqueParameters; // IDictionary<string, string> - map of key-value unique parameters

    // Event-specific properties
    var clickEvent = webhookEvent as ClickEvent; // Cast to the parent based on EventType
    clickEvent.Url; // string - URL on what the user has clicked

Serialisation

Since we’re dealing with JSON - I pulled in a single dependency on the project. It is the Newtonsoft.JSON library. ServiceStack.Text is my preferred option with a lower footprint and marginally higher performance, however, in this instance Newtonsoft.JSON is the one that is used more widely.

The custom serialisation used the type attribute of each event to determine the underlying event type. Each event was based of the same base WebhookEventBase.

namespace Sendgrid.Webhooks.Events
  {
      public abstract class WebhookEventBase
      {
          public WebhookEventBase()
          {
              UniqueParameters = new Dictionary<string, string>();
          }

          [JsonProperty("event"), JsonConverter(typeof(StringEnumConverter))]
          public WebhookEventType EventType { get; set; }

          [JsonProperty("email")]
          public string Email { get; set; }

          [JsonProperty("category"), JsonConverter(typeof(WebhookCategoryConverter))]
          public IList<string> Category { get; set; }

          [JsonProperty("timestamp"), JsonConverter(typeof(EpochToDateTimeConverter))]
          public DateTime Timestamp { get; set; }

          public IDictionary<string, string> UniqueParameters { get; set; }
      }
  }

I have also added couple of custom serialisers that deal with custom date conversion or csv to array. That way you have access to a proper DateTime date and List of categories. You can also delcare the WebhookParser with your own version of the JsonConverter, allowing for virtually any changes possible.

The custom converters are the WebhookCategoryConverter

namespace Sendgrid.Webhooks.Converters
    {
        public abstract class GenericListCreationJsonConverter<T> : JsonConverter
        {

            public override bool CanConvert(Type objectType)
            {
                return true;
            }

            public override bool CanRead
            {
                get { return true; }
            }

            public override bool CanWrite
            {
                get { return false; }
            }

            public override object ReadJson(JsonReader reader, Type objectType, object existingValue,
                JsonSerializer serializer)
            {
                if (reader.TokenType == JsonToken.StartArray)
                {
                    return serializer.Deserialize<List<T>>(reader);
                }
                else
                {
                    T t = serializer.Deserialize<T>(reader);
                    return new List<T>(new[] {t});
                }
            }

            public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
            {
                throw new NotImplementedException();
            }
        }
    }

and EpochToDateTimeConverter

namespace Sendgrid.Webhooks.Converters
    {
        public class EpochToDateTimeConverter : JsonConverter
        {
            private static readonly DateTime EpochDate = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

            public override bool CanConvert(Type objectType)
            {
                return objectType == typeof(DateTime);
            }

            public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
            {
                if (value == null)
                    return;

                var date = (DateTime) value;
                var diff = date - EpochDate;

                var secondsSinceEpoch = (int) diff.TotalSeconds;
                serializer.Serialize(writer, secondsSinceEpoch);
            }

            public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
            {
                var timestamp = (long) reader.Value;

                return EpochDate.AddSeconds(timestamp);
            }
        }
    }

Code

Check out the code at https://github.com/mirajavora/sendgrid-webhooks or pull down from nuget using

Install-Package Sendgrid.Webhooks

Contributions

A surprising number of people started using the code and we’ve had few contributions already, which is great! Thanks to Andy McCready, vanillajonathan, brianp101 and Petteroe.

One last piece of the puzzle is monitoring of the health of the nodes themselves. This typically includes CPU and Memory usage, disk space monitoring and application health monitoring. Diamond collectors are great exactly for that.

Monitoring your server health

I’ve seen instances where a server has gone down purely because it ran out of disk-space. It’s also not uncommon to have a rogue process that suddenly takes up all the CPU in spikes that are easy to miss if you don’t have the right visibility. It’s also a good idea to measure the available memory on your servers - in case you hit a slow memory leak that manifests itself over-time.

All these metrics should be part of your business metrics that you verify when doing releases. A common approach is to have a canary node(s) that you deploy the latest version. Once deployed, you can verify these metrics compared to the rest of the cluster before deploying on all nodes. In an ideal world, these checks should be automated and be part of promoting your builds.

Diamond collectors

There’s a variety of ways to get these metrics from unix boxes to something like graphite. My choice has always been Diamond - mainly for it’s simplicity to set up.

Diamond

Diamond is a python daemon that collects system metrics and publishes them to Graphite or other supported handlers. It is capable of collecting cpu, memory, network, i/o, load and disk metrics. Additionally, it features an API for implementing custom collectors for gathering metrics from almost any source.

It was originally made by BrightCove, but now it’s maintained mainly by devs from python-diamond.

You can find it on Github.

Collectors

Diamond contains a ton of different collectors. While you have a ton to choose from, you are unlikely to need them all. The active collectors are defined via config. Here are some that I would like to highlight

• Memory collector
• CPU collector
• Available disk space collector

Setting up diamond

There are multiple ways to get diamond set-up. The repo illustrates few manual steps to get it running on a box. However, I prefer to have them running as part of my ansible scripts so that it can be repeated and automated effortlessly.

Install Diamond via ansible

Depening on how your code is structured, I tend to add a diamond role that takes care of installing all the dependencies, adding the configuration and any custom collectors I may have.

Your dir structure of the role should also look something along these lines

- tasks
  - main.yml
- files
  - collectors
    - SomeCustomCollector1.py
    - SomeCustomCollector2.py
- templates
   - diamond.conf

Configuration

The role relies on a diamond configuration file. This effectively specifies what handler we’re going to use (in this instance graphite), what collectors are enabled and their settings and the intervals in which they are collected. The entire file can be found here, but I can pick out few key properties to watch out for.

# Specify that metrics are sent via graphite handler
handlers = diamond.handler.graphite.GraphiteHandler

....

# Interval when the metrics are collected and sent off
collectors_reload_interval = 3600

....

# Graphite server host
host = 

# Port to send metrics to
port = 2003

...
# Specify path prefix or suffix
path_prefix = your-metrics
path_suffix = suff

Building custom collectors

While Diamond comes with a long list of in-built collectors, it also allows you to build your custom collectors. We’ve done this recently as we needed to monitor an endpoint on localhost and push that as a metric intro graphite. Rather than building a cron job, custom collector was perfect for that use-case. I conviniently lifted this from the original repo.

Your collector needs to inherit from the diamond.collector.Collector and implement the collect method. You can check out the base class here for more info on what’s exposed to you while writing the collector.

There is also an example collector on the github page of the project.

The bottom line is, the more email you send, the more you want to rely on a 3rd party to take that burden from your hands.

Sendgrid Reporting

Sendgrid comes with pretty comprehensive reporting, where you can drill down on your email performance in detail over time. It has a concept of email categories where you can group your communication types and report on those.

It builds reports based on geo location of the users, email ISP and device type. However, what if you want a little bit more?

Building more detailed reporting

One of the things we always missed was URL click reports - looking at what each user has clicked. We also pass specific meta-data along with the message that we would need to report on - things such as user country for the user or their specific id.

Sendgrid webhooks

Sendgrid is more than ready for this scenario, providing their customers with webhooks functionality. This is triggered on every action (processed, send, open, click, spam report etc). There are a few libraries that will help you along when developing in the Sendgrid library index. Our endpoint was for various reasons in C# so we ended up adding to the library index :-)

Data flows

The quick diagram shows the data flows when dealing with webhooks.

Elastic Search with Kibana seemed like an ideal candidate for storage for the use-case - time-series data that does not change over time and that we need to visualise and query easily. It also gives us advantage when curating the data - we can index based on months and remove indexes as they become obsolete. On top of it, utilising existing tools and parts of infx, it let us deliver this end-to-end with less than 2k lines of code including tests and cloudformation scripts for AWS.

If you don’t need to expose the callback data within the organisation as a stream, you can simplify the architecture by going directly from the endpoint to Elastic Search. However, you would miss the opportunity for your entire organisation to benefit from the data-set!

Aggregate your callbacks in Elastic Search

If you haven’t used Elastic Search or Lucene before, find more about it here.

Effectively, it’s a real-time distributed search and analytics engine based on Lucene. I’ve talked more around what ES is and how you can deploy it on AWS using cloudformation and ansible.

Visualise using Kibana

Kibana is a great way of visualising your data and the release of v4 earlier this year has brought a whole bunch of new features.

If you need to visualise your Elastic Search data, Kibana is what you use. Let’s you create visualisations, searches and dashboards. You can use bar charts, line charts, pie charts, tables - everything that you’d expect from a grown up visualisation framework. It has filters that you can pin and ad-hoc queries that you can then filter your messages by.

Using the data, you can build out dashboards for your email volumes

Your spam reports, split by country

User timeline based on action

And of course, pick a choice of your time-frame

Cloudformation and ansible scripts can make this tedious job super simple.

Elastic Search

I’ve used Lucene several times in the past do to free-text search, but it was tough to built it with scalability and resilience in mind (possible, but tougher than it needed to be). Elastic Search abstracts all that work away, giving you a beautiful API with great tools on top of it.

If you haven’t used it yet, check it out @elastic.

It is essentially a distributed lucene index over multiple nodes. Each index that you create within the elastic search consists of multiple shards that you define at index creation (default is 5). Each shard can then have a specific number of replicas - these are copies of the data and should not live on the same node as the primary shard.

A shard is basically a single lucene index. When a request is made to an elastic search index, ES delegates this to each shard and aggregates the results.

Plugins

There are also numerous plugins that make running and administering Elastic Search cluster easy. There are so many so I wanted to call out only couple that I use always.

Kopf

I’m not quite sure what I would do without the Kopf plugin. It that handles everything from visibility of overall indexes, performance of the nodes and doc size to changing various cluster settings. It can also run backups, snapshots, ad-hoc index queries or queries to change aliases. Did I mention it’s awesome?

AWS Plugin

You cannot run Elastic Search on AWS without this plugin. It’s as simple as that. It enabled unicast discovery and S3 storage (for backups and recovery).

Check it out at the Github repo

Building the cluster using Cloudformation on AWS

If you don’t use AWS, feel free to skip this. However, if you do, this may decrease the time you spend creating the cluster - using Cloudformation. Cloudformation essentially automates the creation of all your AWS resources starting from roles, EC2 instances, security groups and much more.

By using CF, you can be up and running within seconds and recreating the cluster repeatedly isn’t a tedious job. Some may argue how hard is it to build a two-three node cluster, but there’s always scope for human error and it’s not exactly time well spent - especially if it can be automated.

Cloudformation script for auto-scaling group

I created a fairly simple Cloudformation script that I’ve been re-using. It creates a multi-node auto-scaling group of ubuntu machines, security groups and ES role for the instances.

Please change the security groups based on what you need - at the moment they allow access from any IP. You may also need to tweak to rules so that instances have visibility of each other depending on your setup.

Creating the stack

Using the provided script, you can upload it to AWS Cloudformation. Based on your script, you will need to fill in the parameters defined in the script.

After that, AWS will go through each resources within the CF script and create it.

Deploying Elastic Search

Once your infrastructure is setup, you’re going to need to actually deploy elastic search and all the plugins to the nodes. That’s where ansible can help you out massively. If you’ve never used ansible, it’s a neat way to run scripts on remote machines. Check it out here

There is ton of open source scripts that help you run different ansible roles. The one I’ve been using is under Traackr repo - ansible-elasticsearch. George Stathis, if you’re ever in town, I owe you a beer!

It has support for pretty much all config you will need, will install plugins and will install the required java dependency.

Pull it down & get going git@github.com:Traackr/ansible-elasticsearch.git and add it as a role in your ansible scripts.

Your Ansible script

Your mainl.yml script should look something alo the lines.

- name: "Setup the Elastic Search on nodes."
  hosts: elasticsearchnodes
  remote_user: ubuntu
  roles:
    - { role: ansible-elasticsearch, sudo: yes }
  vars_files:
    - roles/ansible-elasticsearch/defaults/main.yml

When you have added the ansible-elasticsearch as a folder in your roles, you should update your machines based on the boxes you created.

I also added cluster name and security group to the inventory vars below. You can also define specific node-names per instance.

[aws-es-01]
ec2-54-88-217-187.compute-1.amazonaws.com

[aws-es-01:vars]
elasticsearch_node_name=clever-node-name1

[aws-es-02]
ec2-54-164-77-248.compute-1.amazonaws.com

[elasticsearchnodes:children]
aws-es-01
aws-es-02

[elasticsearchnodes:vars]
elasticsearch_cluster_name=elasticsearch.cluster
elasticsearch_plugin_aws_ec2_groups=Elastic-Search-ElasticSearchSecurityGroup-1E0LR18NU18JG

Of course, if you are running deployments on multiple machines this can be cumbersome. There are scripts that auto-generate inventories based on specific AWS tags - so you can deploy to all instances that have a tag that you’ve defined at creation time.

Changing variables

You should then open roles\ansible-elasticsearch\vars\main.yml and edit variables accordingly.

Make sure you give ES enough HEAP_SIZE - the default 1GB is almost never enough for any use-case and pick your versions. You can also specify any plugins you want to install along the way. The sample below includes both the kopf and AWS plugins mentioned above.

elasticsearch_user: "ubuntu"
elasticsearch_group: "ubuntu"
elasticsearch_version: 1.7.2
elasticsearch_heap_size: 2g
elasticsearch_max_open_files: 65535
elasticsearch_timezone: "Europe/London"
elasticsearch_install_java: yes

elasticsearch_plugin_aws_version: 2.7.1
elasticsearch_plugins:
  - { name: 'lmenezes/elasticsearch-kopf/1.5.7' }

elasticsearch_service_startonboot: yes

Running the script

Finally, once you have your vars in place, run the main.yml ansible script (sample above)

ansible-playbook --private-key=[path-to-your-private-key] -i aws main.yml

This will run the deployment on the machines specified in your inventory. At the end your cluster will be setup!

If you don’t track it, you can’t measure it. Realtime service and business metrics should be part of any production application. Knowing how is the app performing is as important as measuring whether the product impact of your changes.

A good set of service metrics lets you effectively monitor the impact of your changes on the app performance. Has your change to multi-threading really achieved the times x throughput? Has the last dependency injection change caused a slow memory leak? Are there levels of response codes you’re not expecting in your app?

There are a variety of ways to deal with application level metrics. The world of .NET offers you perf counters while unix has a variety of possibilities. Usuaully, I would go for the simplest, most friction-less option. That’s why StatsD with Graphite is so appealing.

StatsD

Etsy talks about measuring everything and anaything. Their real obsession with metrics made them release an awesome library that took great traction - because it’s simple and effective.

In essence, StatsD server listens on a UDP/TCP port and collects metrics. These are then aggregated and in intervals passed to a back-end of your choice - most likely graphite.

The advantage of the community is that there is a massive list of client implementations so integrating StatsD into your app is super-straightforward. There are multiple clients for node, python, java, ruby, php, .net, go …. and more. Check out the entire list. Also, you will find a bunch of server implementations beyond the original Node.js. If you fancy StatsD on windows machines, check out statsD.net.

Integrate StatsD client into your JVM app

If you’re looking to add statsD into your JVM application, the java client by tim group seemed like the best choice. It has zero dependencies and it’s pretty straight-forward.

Add your dependency to mvn or gradle

MVN

    <dependency>
        <groupId>com.timgroup</groupId>
        <artifactId>java-statsd-client</artifactId>
        <version>3.0.1</version>
    </dependency>

Gradle
    'com.timgroup:java-statsd-client:3.1.0'

And init the statsd client with the prefix, host of the statD server and the port. StatsD has a concept of namespaces, where you can group your metrics - that allows for better visualisation and keeps them neat. The choice of the namespace is yours, depending on what suits you. In bigger deployments you might go for something like “application-name.data-centre.box-name.counter-name”.

import com.timgroup.statsd.StatsDClient;
    import com.timgroup.statsd.NonBlockingStatsDClient;
    
    public class DiagnosticsService {
        private static final StatsDClient statsd;
    
        public DiagnosticsService(String host, int portNumber) {
            statsd = = new NonBlockingStatsDClient("your.custom.prefix", host, portNumber);
        }
        
        .....
    }

I tend to have a single statsD client within the app as a singleton wrapped by a diagnostics service.

StatsD Metric Types

StatsD supports a range of metric types. These you fit 99% of your metric logging scenarios. It also has a concept of a flush interval, where the data is sent off to back-ends.

Counters

Basic counters that are incremented each time you log against the counter. These are reset to 0 at flush. You can also set a sampling interval to tell StatsD you’re only sending part of the data-set.

your.namespace.counter:1|c

Timers

These are great for monitoring response times of any kind. You tell statsD how long an action took. It then automatically works out percentiles, average (mean), standard deviation, sum, and min/max. Really awesome.

your.namespace.response_time:300|ms

Gauges

Gauges are single values that can be incremented or decremented or set to a specific value. Unlike counters, gauges aren’t reset to zero at flush time

Sets

These count unique set of occurrences between flushes.

Logging metrics using the JVM client

Using the JAVA implementation of the StatsD client is then pretty straight-forward.

import com.timgroup.statsd.NonBlockingStatsDClient;
    import com.timgroup.statsd.StatsDClient;


    public final class StatsDPerformanceService implements DiagnosticsService {

        private static StatsDClient statsd = null;
        private static DiagnosticsConfig config;

        public StatsDPerformanceService(DiagnosticsConfig configuration) {
            config = configuration;
            statsd = new NonBlockingStatsDClient(
                    getPrefix(), config.getHost(), config.getPort());
        }

        private String getPrefix() {
            return String.format("yourprefix.%s", config.getBoxName()).toLowerCase();
        }

        @Override
        public void incrementCounter(String counterName) {
            if(config.getEnableMetrics())
                statsd.incrementCounter(counterName);
        }

        @Override
        public void decrementCounter(String counterName) {
            if(config.getEnableMetrics())
                statsd.decrementCounter(counterName);
        }

        @Override
        public void gauge(String gaugeName, long value) {
            if(config.getEnableMetrics())
                statsd.gauge(gaugeName, value);
        }

        @Override
        public void recordExecutionTime(String timerName, long value) {
            if(config.getEnableMetrics())
                statsd.recordExecutionTime(timerName, value);
        }
    }

you can then also consider helper methods using runnable and callable to wrap timings around the methods

@Override
        public <T> T executeWithTimer(Callable<T> callable, String counterName) {
            if(callable == null || counterName == null)
                return null;

            T result = null;
            long startTime = System.nanoTime();
            try {
                result = callable.call();
            } catch (Exception e) {
                e.printStackTrace();
            }
            long endTime = System.nanoTime();

            long duration = (endTime - startTime)/1000000;

            recordExecutionTime(counterName, duration);
            return result;
        }

    ... and execute like

    String result = diagnosticsService.executeWithTimer(
    () -> randomService.getResult(someVar), "SuperAwesomeNameOfTheCounter");

Enjoy! StatsD is great - I’ll look at configuring StatD and graphite in my next post.

Every decent app produces some kind of logging. Traditionally, this has been achieved by wrapper such as log4net, slf4j log4j and many others. The information it produces is invaluable as it’s usually the only source of information when troubleshooting issues.

As you scale and increase the amount of boxes and systems that you manage, this all becomes a lot more challenging and it’s easy to lose visibility. If you manage 30 boxes, you’re not going to check every single one of them for logs - it’s cumbersome and too painful. The chance of missing a defect or a misbehaving service also grows.

Logstash

This is where Logstash comes in, it’s an open-source framework to manage events and logs. In a nutshell, Logstash can take a variety of inputs, apply filters and transformations on the data that comes in and then push them forward to another store.

As an open source project, it contains a massive amount of plugins. Just to name a few

• Inputs - collectd, raw files, elasticsearch, gelf, unix, s3, redis, rabbitmq
• Outputs - cloudwatch, boundary, elasticsearch, gelf, graphite, sdout
• Codecs - JSON, dots, multiline, fluent, graphite and many more

To check out a full list, go to the logstash documentation.

Configuring Logstash

Each Logstash instance can be configured with multiple inputs and outputs and the topology very much depends on your use cases. For example, you may chose to run a single centralised instance of logstash with configs pointing to each separate box or on the contrary, run logstash on every instance.

Did I meantion Logstash has the best logo in the universe?

Log4net/Log4j and Logstash

In my scenario, I was looking to collect multiple logs from multiple apps on a large number of boxes across multiple data-centres. Each of the apps already had logging, which was mostly based on log4net.

Rather than having a separate configuration for each logstash instance, it was much simpler to use the UDP plugin from logstash. This means the existing apps just added a new appender that would log to an UDP port and logstash would just pick that up.

This made the whole logstash config really simple

input { 
        udp { 
          port => 5960 
          codec => plain { 
            charset => "UTF-8" 
          } 
          type => "log4net" 
        }
    }

Changes to the Log4net appender were straight-forward too.

... 
        <appender name="UdpAppender" type="log4net.Appender.UdpAppender">
            <RemoteAddress value="127.0.0.1" /> <!-- set to 127.0.0.1 and host name mapped to this on my machine (port 80) -->
            <RemotePort value="5960" />
            <layout type="log4net.Layout.PatternLayout">
                <conversionPattern value="%date [%thread] %-5level - %property{log4net:HostName} - MyApplication - %message%newline" />
            </layout>
        </appender>
....
add the appender to the root loggers
....
        <root>
            <level value="ERROR" />
            <appender-ref ref="OutputDebugStringAppender" />
            <appender-ref ref="TraceAppender" />
            <appender-ref ref="ErrorFileAppender" />
            <appender-ref ref="UdpAppender" />
        </root>

Adding Graylog2 into the mix

Graylog2 is one of my favourite tools. The sole purpose is to aggregate and analyse logs in real time. With elastic search sitting underneath, it let’s you do complex queries on the data and create custom dashboards.

If you have an instance of Graylog2 running somewhere, it’s easy to use the gelf output to channel all the incoming logs into Logstash into Graylog2. The full logstash config could then look something like this:

input { 
    udp { 
      port => 5960 
      codec => plain { 
        charset => "UTF-8" 
      } 
      type => "log4net" 
    }
}


filter {
  if [type] == "log4net" {
    grok {
      remove_field => message
      match => { message => "%{TIMESTAMP_ISO8601:sourceTimestamp} \[%{NUMBER:threadid}\] %{LOGLEVEL:loglevel} +- %{IPORHOST:tempHost} - %{GREEDYDATA:tempMessage}" }
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "message" , "%{tempMessage}" ]
        replace => [ "host" , "%{tempHost}" ]
      }
    }
    mutate {
      remove_field => [ "tempMessage" ]
      remove_field => [ "tempHost" ]
    }
  }
}

output {
  gelf {
     host => "your-host-name"
     custom_fields => ["environment", "PROD", "service", "BestServiceInTheWorld"]
     }
  stdout { }
}

A result will be a centralised stream of logs that you can easily analyse and create dashboards from. The beauty of logstash is that you can easily add further outputs (eg graphite).

Looking For a Light-Weight Blogging Platform

When I fist started blogging, I went for the Orchard Project as the blogging platform. I was hoping that it would be light, MVC-based platform that would be easy to manage and maintain. I was a bit dissapointed with Orchard - it was pretty heavy, development was cumbersome, the performance was low and overall felt like a bit of an overkill.

Looking at a new framework, I had several requirements. It had to be fast, simple and be able to handle

• posts
• archive & monthly archive
• widgets to load latest & custom
• ability to generate custom content pages

I also wanted move away from an IIS based hosting over to Nginx. That meant I had few open-source CMS options to look at. There is Django CMS, Joomla and of course Drupal.

And then I stumbled upon Octopress

Octopress

Unlike the traditional CMS projects, octopress is a blog-aware static content site generator. It runs on top of Jekyll which is the engine behind GitHub pages.

The content is generated from the markdown files you create. Jekyll alone does not have a great deal of html templates and css/js - that’s where Octopress comes in. It gives you an awesome framework that you can take forward and tweak to your liking, directly in the code.

The fact that all the content is static makes your site super-fast - as demonstrated in the graph below.

Living with Octopress

Having used Octopress for few weeks now, the experience has been great. The ability to change html/css directly is great as well as no need to have data-store in place. The deployment is simple and straightforwards as well as backups. In short, exactly what I a was after.

Installation

Installing Octopress is super-easy. The two main dependencies are Ruby and Git, which most devs will have anyway.

git clone git://github.com/imathis/octopress.git octopress
cd octopress

You will then need to install couple of other gem dependencies and install octopress - to setup dirs and structure

gem install bundler
rbenv rehash    # If you use rbenv, rehash to be able to run the bundle command
bundle install

rake install

Octopress has a concept of templates/layouts and includes which you can include to avoid repetition. You can find them in /source/_layouts and /source/_includes

Writing Posts

Once you have Octopress up and running, you can start creating posts. There are rake methods to do that as a shortcut.

To create a new post, run

rake new_post["A new post Title"]

This will create a new markdown file in /source/_posts. Each post contains a set of variables and the actual content. You can also add your own on top of the pre-defined.

---
layout: post
title: "SASS with Visual Studio Part 1 Introduction"
date: 2011-09-11 15:01:30 +0000
comments: true
categories: [SASS, CSS, Visual Studio]
customVariable: "Custom content"
image: /images/posts/sass/sass_thumb.jpg
---

This is a start of the content ...

Once you’re happy with the content, you can run another rake task to produce the content.

rake generate

The generated files will end up in /public

Hacking Octopress

Octopress is an ideal choice for any developer. In fact, there was very little that it couldn’t do out of the box or I couldn’t change right away.

It was super easy to change any html and tweak any shared components. There are lots of plugins available, the only additional bit that I was interested was a monthly archive which I borrowed from here.

Back ups

One of the nicest things about Octopress is the ease of back-ups. With no dependencies on DBs or any other storage, I tend to simply push the entire blog onto a public github account.

That means I have a permanent backup and I can clone and start updating the content anywhere. Changing hosting providers is a matter of minutes rather than hours.

Deployment

All that octopress needs is an http server in front. It supports a simple push to something like GitHub pages or Rsync to your own server. You can also easily write your own rake tasks to push the content anywhere.

Any questions, give me a shout @mirajavora

Real World Polyglot Persistence

by Jimmy Board

A man behind the AutoMapper is not only a great speaker, but unlike so many other talks, he brought along a real-world example. I could really identify myself with his main view of – use the right tool for the job, rather than try to basterdise something that was not build for the purpose. That means, be prepared to use lots of different storage mechanisms in a large app and lots of different ways to put the systems together.

He showed off fake e-commerce app that would

• Store the main catalogue in relational DB (changes little, mostly read, caching)
• Store the cart info in Key-Value Store instead of Session (auto-expiry, transient data)
• Store order information in Document DB (typically a different system, when catalogue changes, order does not)
• Use Graph DB for suggestion engine

Each component worked together in a way that was also closer to the real-world scenario – HTTP post between components, Messaging Bus and AJAX on front-end to different systems. You can check out the slides and the code here.

Clean Architecture and Design

by Robert C. Martin

I enjoyed Uncle Bob talking about clean architecture. The abstract was a bit vague so I feared the talk will be a bit too generic. I was wrong. I really enjoyed it, in particular, because he was describing an architectural anti-pattern that I’ve seen so many times in MVC.

He stressed the importance of Web being a presentation logic and that your presentation logic (or framework) should not dictate the core business model. I’ve seen so many times where the ViewModel logic has grown beyond manageable (calling services or data access layer) or view logic grown out of proportion. He stressed the importance of keeping ViewModels and Models separate.

A good talk, great speaker.

Under the Covers with ASP.NET SignalR

by Davian Fowlwards

The authors of SignalR David Fowler and Damian Edwards came in and stole the show. The most entertaining and enjoyable talk by a long mile. After a quick demo of what is SignalR they went on to and wrote a simpler version of SignalR - on stage – live – re-using only one helper class and a message bus class.

This was both very ballsy and impressive but also gave a great inside to what SignalR does and how. Starting with the negotiation to the different transport types. It also wasn’t all about the code – whenever something needed to be explained, they would go in a bit of detail.

It was a great session – fun, full of content and great QA at the end. Awesome guys!

C# 5

by Jon Skeet

I went to couple of talks by Jon at NDC. This session could have also been called async – await :-) I would recommend the session if you never used the async-await before. Jon gives a great explanation of the async-await pattern using an example of a cardboard box that contains something the result inside.

He also went on to cover the intricacies of unit-testing the awaitable functions – really worth checking it out.

Backbone

by Kim Bekkelund and Hans Inderberg

I was glad to see Backbone on the list of topics – especially now that I work with it day-to-day. The guys showed off some good tips and patterns to use when using backbone such as

• using subviews and pushing the logic from view to model
• avoid adding too much logic in initialize
• create base-classes for each component to share logic (baseviews, basemodels), 1 level of inheritance is usually enough
• don’t be shy to use vanilla JS

Overall, good tips and issues we’ve come across as well. They also suggested grouping Backbone components (view, model) together in feature folders. I guess that was the only suggestion I would have a problem with, given you would want to re-use some of the models.

There were plenty of other talks that I enjoyed, but I won’t bore you with details. In particular both sessions by Dominick Baier who gave good overview of Web API security and OAuth2. I also enjoyed Dan North talking about Effective Teams & Jon Skeet on Abusing C#. All in all, great conference.

This problem is traditionally solved by Authentication and Authorization and your credentials are passed along with the request to the server. An alternative is to exchange the username and password for a short-lived access token and use this access token to perform the protected actions. This solution does not fit every scenario, however, it means that if the access token gets exposed, the user credentials are not revealed.

Authentication and Authorisation

There is a key difference between authentication and authorisation. Authentication is regarded as identifying the user – confirming that you are, who you say you are. Authorisation is a secondary step that happens once the authenticity was established. Authentication is all about does this person have access to access this resource or perform certain action.

Creating an Access Token

The first thing the user needs to do is exchange the user credentials for an access token. This token is stored on the client side and verified every time by the service API.

In the action below, username and password is captured by the login model and passed down to the authentication service. If the auth service returns a valid user, we can create a short-lived auth token for the user to use. The token is then set in a cookie and returned as part of the response to the user.

[HttpPost]
public ActionResult Index(LoginModel model)
{
    if(!ModelState.IsValid)
        return View(model);
 
    var result = _authenticationService.Authenticate(model);
    if (!result.IsAuthenticated)
        return View(model);
 
    var token = new AccessToken(result.User.Id);
    _accessTokenRepository.Save(token);
    Response.Cookies.Add(new HttpCookie("token", token.Id) { Expires = token.Expires, Path = "/" });
 
    return RedirectToAction("Index", "Security");
}

Authentication Handler to check the Access Token

Once you have created your access token, it will be sent to the server every time as part of the cookie collection. The server should then check the access token cookie on every request create an appropriate IPrincipal based on the access token.

The best way to ensure access token is processed on every request, you can create a custom handler for authentication by inheriting from DelegatingHandler class.

protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
    var accessToken = request.Headers.GetCookies("token");
    if (accessToken.Count == 0)
        return base.SendAsync(request, cancellationToken);
 
    var tokenValue = accessToken[0]["token"].Value;
    var token = _accessTokenRepository.FindById(tokenValue);
    if(token == null)
        return base.SendAsync(request, cancellationToken);
 
    var user = _userRepository.FindById(token.UserId);
 
    var identity = new GenericIdentity(user.Username, "Basic");
    var principal = new GenericPrincipal(identity, user.Roles.ToArray());
    Thread.CurrentPrincipal = principal;
 
    return base.SendAsync(request, cancellationToken);
}

The handler gets the access token from the cookie and attempts to find the user based on the access token. If the user is found, new GenericIdentity and GenericPrincipal are created based on the user and user’s roles. You can then assign the GenericPrincipal to the current thread.

Remember to add your AuthenticationHandler to the MessageHandlers in the Web API GlobalConfiguration

....
GlobalConfiguration.Configuration.MessageHandlers.Add(
    new AuthenticationHandler(Container.Resolve<IAccessTokenRepository>(),
                              Container.Resolve<IUserRepository>()));
...

Protect API Actions with Authorize Attribute

Once the user gets authenticated and the user roles are stored on the thread’s IPrincipal, you can you use the in-built Web API Authorize attribute to check whether the user is authenticated. You can even specify roles that user needs to be in to perform a specific action.

[Authorize]
public override IEnumerable<Contact> Get()
{
    return base.Get();
}
 
[Authorize(Roles = "Administrator")]
public override System.Net.Http.HttpResponseMessage Delete(Guid id)
{
    return base.Delete(id);
}

If the user is not authenticated or does not have the correct permissions, the server will return 401 HTTP status code.

Code Sample

You can check out all the the above in the code sample on GitHub. Run the solution and navigate to /security - have a look on the SecurityController, SecureContactController and AuthenticationHandler controller. If you have any questions give me a shout @mirajavora

An important part of Web API is resource content negotiation. The HTTP protocol RFC defines content negotiation as the process of selecting the best representation for a given response when there are multiple representations available. In practise the same resource can be represented in a variety of different ways – lets say a contact information resource can be shown in JSON representation, but also in XML or even as a PNG QR code containing the same content.

The content negotiation can be either server or client driven. In the first instance, the server decides what content to send down based on the various headers sent from the client. The latter requires an additional call to the server to get the list of available representations. Web API provides you with an easy way to send different representations of the same content.

Creating new MediaTypeFormatter to Deliver QR Code

The Web API comes with MediaTypeFormatter base class. If you want to create your own formatter, simply inherit from MediaTypeFormatter. If we take the existing scenario of contact management, lets say we want to create a QR code that contains the contact information of each contact stored in the database.

Our QrMediaFormatter can only be used for writing the type of Contact. Therefore we add logic to the CanWriteType and override the WriteToStreamAsync member.

public class QrMediaFormatter : MediaTypeFormatter
{
    private const string ApiEndpoint = "http://chart.apis.google.com/chart";
 
    public override bool CanReadType(Type type)
    {
        return false;
    }
 
    public override bool CanWriteType(Type type)
    {
        if(type == null)
        {
            throw new ArgumentNullException("type");
        }
 
        return type == typeof(Contact);
    }
 
    public override Task WriteToStreamAsync(Type type, object value, Stream stream, System.Net.Http.HttpContent content, System.Net.TransportContext transportContext)
    {
        return Task.Factory.StartNew(() => WriteQrStream(type, value, stream, content.Headers));
    }
 
    private void WriteQrStream(Type type, object value, Stream stream, HttpContentHeaders contentHeaders)
    {
        var contact = value as Contact;
        var values = new Dictionary<string, string>()
                         {
                             {"cht", "qr"},
                             {"chs", "300x300"},
                             {"chld", "H|0"},
                         };
        values.Add("chl", String.Format("{0}%0D%0A{1}", HttpUtility.UrlEncode(contact.Name), HttpUtility.UrlEncode(contact.Email)));
        var endPointUrl = BuildUrl(values);
 
        using(var client = new WebClient())
        {
            var data = client.DownloadData(endPointUrl);
            stream.Write(data, 0, data.Length);
        }
    }
 
    private string BuildUrl(IDictionary<string, string> values)
    {
        return string.Concat(ApiEndpoint, QueryString(values));
    }
 
    private string QueryString(IDictionary<string, string> queryStringItems)
    {
        var stringBuilder = new StringBuilder();
        var joinCharacter = "?";
        foreach (var key in queryStringItems.Keys)
        {
            stringBuilder.AppendFormat("{0}{1}={2}", joinCharacter, key, queryStringItems[key]);
            joinCharacter = "&";
        }
 
        return stringBuilder.ToString();
    }
}

The WriteQrStream method simply takes the Contact, builds up a simple string of the contact name and email and sends the string to the Google QR chart generation API. The stream coming from the API is then directly written to the response stream.

Wire Up Your Custom Media Type Formatter

The SupportedMediaTypes property contains a collection of MediaTypeHeaderValues. These define which media types the formatter can handle. In our case we are happy just to map the QR formatter to a single type –> image/png. However, it is possible to have the same media formatter handle a variety of MediaTypeHeaderValues. For example image/png and image/jpeg.

public QrMediaFormatter()
{
    SupportedMediaTypes.Add(new MediaTypeHeaderValue("image/png"));   
}

Once we have the media type in the SupportedMediaTypes, the client can use the Content-Type request header to request the image representation of the resource. You can try it out in fiddler, accessing the contact resource with “Content-Type: image/png” header.

Map Extension to the Media Type Formatter

The idea is that the same resource should not change URI based on representation. However, it is difficult when some clients are unable to make requests for specific content type. For example, even if you use your contact resource as a path to an image /api/contact/a85ad33c-b61f-4503-8d75-861f3701efe3, the browser client does not send a content-type request for image and the service will the return a default representation of the resource, which in our case isn’t an image.

The way around it is to map a particular extension to the Media Type Formatter. It means adding an extension and therefore changing the URI, but it means it can be used by clients without specifying the requested content-type. First, you need to make sure your routes support extensions.

config.Routes.MapHttpRoute(
    name: "IdWithExt",
    routeTemplate: "api/{controller}/{id}.{ext}");

Then you need to add UriPathExtensionMapping to the MediaTypeMappings.

public QrMediaFormatter()
{
    MediaTypeMappings.Add(new UriPathExtensionMapping("png", "image/png"));
    SupportedMediaTypes.Add(new MediaTypeHeaderValue("image/png"));   
}

Once you have both in place, you should be able to request the resource on /api/contact/a85ad33c-b61f-4503-8d75-861f3701efe3.png URI.

Change the Configuration to Add the QR Formatter

Finally, it is important to wire up all custom media formatters on app startup.

GlobalConfiguration.Configuration.Formatters.Add(new QrMediaFormatter());

Code Sample

You can check out all the the above in the code sample on GitHub. If you have any questions give me a shout @mirajavora

The code sample for the article can be found on GitHub. You can find more details at the end of the article.

Creating a RESTful API with ASP.NET

There are several options you can go for when deciding what technology you adopt for your RESTful API. In fact, you’ve got lots to chose from. If you’re old-school (and enjoy living in the past), you might want to go for WCF. Another popular open source framework for creating RESTful APIs with C# is OpenRasta. A creation of Sebastien Lambda has been around for a while, however, it hasn’t had that much traction in the last 18 months.

Smaller frameworks such as NancyFx or Simple.Web are also an option.

Why Should You Chose Web API Over Anything Else

Obviously, ASP.NET Web API isn’t going to be fit for all. However, here are few reasons why you should consider Web API

Simplicity, Convention Based, Designed with Rest in mind

Gives you a direct access to manipulate HTTP requests and responses and uses conventions to map HTTP methods to actions. It is simple and easy to get up and running, no XML-based setup required.

Content Negotiation And Filters

It supports JSON and XML out of the box and lets you create your own formatters for whatever media type you specify through the content-type headers. It also provides filters that can be applied to each state of the action. It also comes with lots of goodies such as Model State validation.

Routing

Web API uses the same mechanism to deliver resources as the rest of the ASP.NET stack – routing. It keeps the API endpoint implementations separated from the Uris. It also lets you use add constraints in the urls.

IOC Support and Dependency Resolver

Comes with support for IOC out of the box. In fact, has been written so that almost every part (including the default content formatters) can be replaced.

Create a New Web API Endpoint

In order to get started, you can either select Web API Project from Visual Studio create project dialog or you can install the Web API Nuget Package package. This will pull down further dependencies such as (Microsoft.AspNet.WebApi.WebHost, Microsoft.AspNet.WebApi.Core, Microsoft.AspNet.WebApi.Client and Newtonsoft.Json).

Create your API endpoint by inheriting from ApiController.aspx).

public class ContactController : ApiController
{
        public IEnumerable<Contact> Get()
        {
            ...
        }
 
        public Contact Get(Guid id)
        {
            ...
        }
 
        public HttpResponseMessage Post(Contact entity)
        {
            ...
        }
 
        public HttpResponseMessage Put(Guid id, Contact entity)
        {
            ...
        }
 
        public HttpResponseMessage Delete(Guid id)
        {
            ...
        }
}

The ApiController class uses conventions to map HTTP methods to actions. As long as the action name starts with the HTTP method name, it will be mapped correctly. For example, for HTTP DELETE method to remove Contact resource you should name the action DeleteContact or just Delete. However, if you don’t want to follow the pattern, you can also decorate each method with HttpGet, HttpPut, HttpPost, or HttpDelete.

The automatic content negotiation will then work out which formatter to use and present the same resource using the requested format. One thing to note is that Web API uses Newtonsoft.Json as a default JSON serializer!

HttpResponseMessage and HttpResponseException

Web API gives you access to the raw HTTP response via the HttpResponseMessage and the HttpRequestMessageExtensions. The extension class contains methods for creating responses beyond the simple 200 with a resource. You can access the extensions by calling Request.ExtensionMethod.

For example, when a PUT method is called, HTTP standard dictates that on a successful creation of the resource, the server should respond with 201 Created and Uri of the created resource in the Location header.

... validation and creation of the resource
 
HttpResponseMessage response = Request.CreateResponse<Contact>(HttpStatusCode.Created, resource);
response.Headers.Location = GetLocation(resource.Id);
return response;

In a similar fashion, at any point you can call the HttpResponseException and encapsulate the HttpResponseMessage within it.

public Contact Get(Guid id)
{
    var entity = _repository.FindById(id);
    if (entity == null)
    {
        throw new HttpResponseException(new HttpResponseMessage(HttpStatusCode.NotFound));
    }
 
    return entity;
}

Model Validation and ModelState

One of the awesome features of MVC is the model validation. Web API follows the same pattern and you can use the same annotation to use model validation in Web API. You can use the in-built validators or create your own custom validators. Calling ModelState.IsValid in your method will then evaluate the model.

public class Contact
{
    //...other fields
 
    [Required]
    public string Name { get; set; }
 
}
 
public HttpResponseMessage Post(Contact contact)
{
    if (ModelState.IsValid)
    {
        //.. handle successful creation of the contact resource
    }
    return Request.CreateResponse(HttpStatusCode.BadRequest);
}

Set Up Route To The API Endpoint

Although the routing in Web API follows the same pattern as routing in MVC, it is important to understand they are not the same. You are calling MappHttpRoute on HttpRouteCollection with a routeTemplate.

public static void Register(HttpConfiguration config)
{
    config.Routes.MapHttpRoute(
        name: "DefaultApi",
        routeTemplate: "api/{controller}/{id}",
        defaults: new { id = RouteParameter.Optional }
    );
}

If you want to read more on Web API routing, this article by Mike Wasson is a good start.

Dependency Injection

You can inject your Web API controllers with your own services using your own IDependencyResolver implementation. For more details, you can check out the code sample (Castle Windsor implementation). The only thing you need to do is set your own DependencyResolver implementation on App Start.

GlobalConfiguration.Configuration.DependencyResolver =
                     new WindsorDependencyResolver(Container);

Code Example

You can check out all the the above in the code sample on GitHub. It covers a basic Contat Management API, Dependency Injection with Castle Windsor, Model Validation, Basic Routing setup and front-end based on Twitter Bootstrap and JQuery.

How It Went Downhill

All was going well. We agreed on the terms, I put together a detailed outline, which got reviewed and then eventually approved and then … then things went sour. It started with the schedule. It was based on the outline a consisted of a simple formula between days and pages. No questions asked about holidays, week ends apparently weren’t a thing anymore and nobody celebrates Christmas these days. At this point the alarm bells were ringing but after a bit of back-and-forward they were happy to accept my schedule.

The final straw was the contract. A complete one-sided, you cover all indemnity kind of style which I expected. However, I had a lawyer look at it and we decided to ask for few changes mainly around revisions, payment terms and the ability to terminate contract from my side. The answer was no, not even let’s talk about it – straight up take it or leave it. My advice is if you get approached by Packt … run!

Why Have I Decided To Self-Publish

Never before it was this easy to self-publish a book. The traditional publishers must be pooping themselves as we speak. I still have a lot of material, outline and lot of examples. Creating a book out of it has to be easy right? Here are just a few reasons why I chose to self-publish over going traditional.

Royalties

Writing a book isn’t about the big bucks (unless you sell thousands) but still, 16% royalties is just taking the piss. When you self publish you know that the biggest piece of cake is going to the person who’s done most of the work. That works both ways, for authors and people who buy the book.

Schedule

Having a complete control over your own schedule is a big win. You don’t have to worry about touch deadlines – missing them usually has an effect on your money.

Content and Revisions

The publisher usually has the right to change anything and in some cases let you iterate the content as many time as they see fit. When you self-publish, you can publish when you think it’s right.

Self-Publishing With Leanpub

I’ve heard about Leanpub a while back from Scott Hanselman. It lets you write a book using markdown and publish it in a agile way. It generates .epub, .mobi and .pdf formats for you and your readers have access to the latest iteration as soon as it’s published. You can get feedback as you go along and improve the content.

It seemed like a perfect fit for what I was trying to do. You can check out the book page by clicking on the link below. You can check out the content and the progress. Also, you can sign up to be notified once the book gets released.

SignalR was built with scalability in mind. Even though you will be able to run a fair number of concurrent connections on a single server instance, there will come a point where a single node will not be able to handle the load. The maximum threshold of concurrent connections per node depends on quite a few factors such as server spec, client transport type or the amount of work alongside message processing.

The solution is to increase the number of server nodes and run SignalR over load balancer or in web-garden. However, these separate instances will need to pass data between each other to ensure every client gets the same data. This is done through a backplane.

SignalR will support one of the following backplanes to pass messages between the web nodes: Service Bus, Redis and SQL Server. This article will look at implementing Redis as the backplane. However, you can also use the Service Bus for Windows Server that just went 1.0 and the SignalR team are also promising an SQL backplane implementation soon.

Running Redis on Windows

Redis is an in-memory key-value store that persists the data to the disc from time-to-time. It is the in-memory element that gives Redis the amazing performance. It was written in ANSI C and therefore works well on Linux or OSX. However, you can now run it on Windows using the MSOpenTech redis port. I’m not 100% sure if it is fit for production, but for local dev with Redis it is more than good.

Getting up and running with Redis from MS OpenTech

You can start by cloning the repository from https://github.com/MSOpenTech/redis and building the solution from /msvs/RedisServer.sln. It is written in C++ so you should be able to build it in Visual Studio in Release mode and pick up resulting files from /msvs/Release/.

You can then copy the Release files to your desired directory, for example C:\Redis\bin\ and copy the redis.conf configuration file to the same directory (from the root of the repository).

Run the Redis Server locally

Once you have your redis server built and the config file in place, you can open up C:\Redis\bin\ folder in command prompt and run:

redis-server.exe C:\redis\bin\redis.conf

The server will start up and give you a heart-beat ever five seconds with the number of connected clients. It’s very important that you specify the config file. If you don’t redis will use the default config and SignalR will not be able use the queues.

Now that you’ve got the redis server running, you can connect with your clients. To test your redis server, you can connect with a console client. Open up another command prompt, navigate to C:\Redis\bin\ and type in

redis-cli -h localhost -p 6379

You should get a successful connect message redis localhost:6379>

If you are not after a local Redis server on Windows, you can get a free instance from http://redistogo.com/ to test things out.

Wire Up SignalR and Redis

Wiring up SignalR and Redis is really, really simple and takes only two steps. First of all you should pull down the Microsoft Asp.Net SignalR Redis package. It will pull down various dependencies, including BookSleeve library that looks after the communication with Redis.

Then, you could call UseRedis extension method on SignalR DependencyResolver to register Redis as your backplane in your global.asax.

//... your existing code
 
RouteTable.Routes.MapConnection<DistributedConnection>("echo", "echo/{*operation}");
GlobalHost.DependencyResolver.UseRedis("localhost", 6379, "", new [] {"signalr.key"});

You can push the setting to your web.config and use a settings class instead of hard-coding them. Since I’m running the server locally, I use “localhost” as my host, the default 6379 port and I pass in an empty string as my authentication pass. Also, you have to specify event key name. You change the port number and password by editing the redis.conf file.

Testing it all locally

Testing it all locally can be a bit tricky. The best thing to do is to test it in full IIS with web-garden. You should set up your local app as you normally would and then increase the number of web-processes. You will find the setting in your App Pools –> Select your app pool and Click on Advanced Settings –> Set the Maximum Number Of Worker Processes to higher than 1. (I’ve set it up to 10 just to be sure).

When you spin up few web-clients, you should see the number of worker processes rising for your app pool. You should also see the number of clients rising on your local Redis server. When you broadcast the message over SignalR, it should get delivered to all worker processes. Simple!

If you have any questions, give me a shout @mirajavora

Related Links

Redis Documentation http://redis.io/documentation
Free Redis Instance http://redistogo.com/
MS Open Tech Windows Redis Port https://github.com/MSOpenTech/redis

Related Articles

SignalR – Introduction To SignalR
SignalR – Publish Data Using IHubContext
SignalR – Publish Data Using Proxies
SignalR – Dependency Injection

You may consider storing the content in a resource file and embed it in a class library. Or perhaps do a clever virtual directory mapping in your IIS setup. However, the best solution is simply to compile the views into a class library using Razor Generator. Razor Generator is tool that allows processing Razor files at design time instead of runtime, allowing them to be built into an assembly.

Install Razor Generator Extension

In order to get your views compiled, you will need to install a tiny extension. The latest version can be found in the Visual Studio Extensions Gallery. Once installed, you will need to re-start Visual Studio.

The extension will enable RazorGenerator Custom Tool. Custom tools in Visual Studio can be attached to various files to generate automatic classes. For example, the ResXFileCodeGenerator tool looks after auto-generating classes for resource files. Visual Studio comes with several tools built in, however, you can also build your own.

Create Your Shared Project

First, you will need to create you shared class library project. You can use the standard class library.

Next thing you need to need to do is include the RazorGenerator.MVC package via Nuget. This is going to bring down couple of dependencies and will enable serving of the auto-generated views once they are compiled in the dll.

Add Re-Usable Views

Once you have your common project setup, you can create the same directory structure as you would in a normal MVC project. For example, if you wanted your re-usable partial view to be in shared, you would place it in ~/Views/Shared/YourPartial.cshtml or even ~/Views/Shared/DisplayTemplates/YourPartial.cshtml.

Finally, you need to make sure you specify RazorGenerator as a custom tool for the Views. Simply right-click on the views and select properties. You will a field for the Custom Tool which you should set to RazorGenerator.

Once you re-build the project, you should see an arrow below your views, indicating there is a linked .cs file.

Include a Reference to Your Shared Project

Finally, you should reference the class library into your projects. All you need is a reference to the dll produced when the class library is built. You can then call up any views as if they were present in the solution. The example below assumes you have SharedPartial.cshtml in your ~/View/Shared/ folder in the Example.Common.

... your view content
<div class="container">
    @Html.Partial("SharedPartial")
</div>
... your view content

Have fun! If you have any questions, give me a shout @mirajavora.

Related Links

Razor Generator Extension for Visual Studio
Razor Generator Project On Codeplex

You can add browser definition files to your solution and specify capabilities for the latest devices. It is however a quick road to hell.

Instead, there are several open source libraries that help you improve the device detection and override the BrowserCapabilities. 51Degrees.mobi is widely recognised and I had very good experience using it in the past. It contains an amazing list of the devices which gets updated on regular basis. It is a freemium product, however, in most cases you will only need the functionality that is in the free version.

Add 51Degrees to your solution

blog-51Degrees-Nuget

The easiest way to integrate 51Degrees is by adding a nuget package. The package name is 51Degrees.mobi and it brings down the FiftyOne.Foundation.dll, 51Degrees.mobi.config config file and /Mobile/Default.aspx page.

If you run .NET 4, the dll automatically registers its own MobileCapabilitiesProvider (HttpCapabilitiesBase.BrowserCapabilitiesProvider) using PreApplicationStartMethod attribute and registers the RedirectModule. If you are on 2.0 or 3.5, you will need to register the detection module yourself in your web.config.

<httpModules>
    <add name="Detector" type="FiftyOne.Foundation.Mobile.Detection.DetectorModule, FiftyOne.Foundation"/>
</httpModules>

OR

<system.webServer>
    <modules>
    <remove name="Detector"/>
    <add name="Detector" type="FiftyOne.Foundation.Mobile.Detection.DetectorModule, FiftyOne.Foundation"/>
    </modules>
</system.webServer>

Use Detector To Improve Mobile Detection

If all you want is improvement of your mobile detection, then you can safely delete the 51Degrees.mobi.config and the Default.aspx page. The 51Degrees MobileCapabilitiesProvider will still run and override your IsMobileDevice property. If you use mobile views, you will notice a significant improvement in which devices get the mobile view.

You should not forget that 51Degrees will regard all tablets as mobile devices. This will result in IPad getting served the mobile views. If that’s something that you don’t want, you will need to write extra filter that will reset the IsMobileDevice to false for all tablet devices.

Redirect Your Mobile Traffic

51Degrees can also be configured to redirect your mobile traffic using the 51Degrees.mobi.config file in the root of your app. It lets you define either page-by-page redirects or redirects based on patterns or regular expressions.

An example of the redirect setting

<redirect firstRequestOnly="true" timeout="20" devicesFile="~/App_Data/Devices.dat" mobilePagesRegex="/(Mobile|Tablet)/">
    <locations>
        <location name="tablet" url="~/Tablet/Default.aspx">
        <add property="IsTablet" matchExpression="true"/>
        <add property="IsMobile" matchExpression="true"/>
        </location>
        <location name="mobile" url="~/Mobile/Default.aspx">
        <add property="IsMobile" matchExpression="true"/>
        </location>
    </locations>
</redirect>

The sample above will catch the first request of the device and redirect it to either /Mobile/Default.aspx or /Tablet/Default.aspx based on the device features. If you are on premium, you can optionally specify your devices file. You can find out more about the redirect settings in the 51Degrees user guide.

What You Get For Your Money

If you decide to buy 51Degrees.mobi, it works out around $1 per day per site instance. You pay a subscription for a year, which gives you fully featured detection and regular updates. And what else do you get? Lots of detail – and I mean lots of it.

Here’s a list of all the capabilities that 51Degrees offers – the premium only features are in red.

Related Links

51Degrees on Codeplex
What’s New In Asp.NET MVC 4 Mobile Development

The Old School – XML And XSLT or Token Replacement

A common way to implement email templates was to use XML in combination with XSLT. It used to make sense: you separate the data from the markup, are able to create multiple XSLT tranforms based on HTML vs PlainText using the same data, you could use complex logic and loops and you could re-use the templates cross-systems. However, editing and maintaining the templates was never easy. Simpler systems may even use token replacement techniques.

Razor Engine To Generate Your Email Templates

Razor engine became the new default view engine for ASP.NET MVC projects. In fact, there are at least 4 major view engines and lots of others to choose from. A view engine takes a model and generates HTML. However, a view engine is also a perfect tool for creating email templates.

RazorEngine project is built on Microsoft’s Razor parsing technology. The key thing is that you don’t need web-context dependencies to parse the templates or any other related razor dependencies.

Why You Should Consider Razor Engine For Your Email Templates

Out of the box you get:EmailTemplate

• Ability to use strongly typed or anonymous models
• Conditional statements
• Loops
• Clean Razor syntax and IntelliSense

On top of that, if you use Razor as your view engine in MVC, it gives your front-end devs similar environment to create the templates, rather than learning a new system/syntax (you can always write a bit of code to enable in-browser preview). Plus you get intellisense.

Parsing Your Templates

First of all, create a model for your template

public class EmailModelBase
{
    public string Name { get; set; }
    public string UnsubscribeUrl { get; set; }
}

You can then read the content of your template and use your model to parse the resulting HTML by calling Razor.Parse()

public static class EmailFactory
{
     public static string ParseTemplate<T>(T model, string pathToTemplates, EmailType emailType)
     {
         var templatePath = Path.Combine(pathToTemplates, string.Format("{0}.cshtml", emailType));
         var content = templatePath.ReadTemplateContent();
 
         return Razor.Parse(content, model);
     }
}

Editing your HTML Templates becomes a doddle – just use the @Model syntax

... more HTML
<p align="left" class="article-title">Dear @Model.Name</p>
... more HTML

If you need to, you can use loops or conditional statements

@foreach(var image in Model.Images)
{
   <td class="w130" width="130" valign="top">
       <table class="w130" width="130" cellpadding="0" cellspacing="0" border="0">
           <tbody><tr>
                      <td class="w130" width="130"><img src="@image" editable="true" label="Image" class="w130" width="130" border="0"></td>
                  </tr>
           </tbody></table>
   </td> 
}

Building New Email Templates

Building new html email templates can be very tedious. All the style needs to be inline and getting things working well cross device can be a pain. Campaign monitor has a simple email builder that you can base your emails on - Campaign Monitor Email Builder. If you want test cross-device, Litmus is a place to go. You can send a test email to litmus and it gives you a preview on huge variety of devices.

Testing Your Email Templates Locally

Rather than setting up SMTP on your local machine, I suggest you use something like SMTP 4 Dev. It runs in the tray, listens to the port 25 and catches all your outgoing emails. It also lets you preview each email. This means your local system can send out emails without you worrying they will be delivered and you can preview them easily (providing you use System.Net.Mail and your Smtp client or web.config host is set to localhost).

Sample App

The sample app illustrates couple of examples, gives you the ability to generate, preview and send Razor Email templates via SMTP client. You can download the app from the link below.

Any questions, give me a shout @mirajavora

Setting up dependency injection with SignalR is pretty straightforward. SignalR uses DefaultDependencyResolver to resolve its own services (IConnectionManager, IConfigurationManager, IJsonSerializer, ITransportManager and many more) and hub extensions (IHubManager, IJavaScriptProxyGenerator and many more). Using GlobalHost, you can set your own dependency resolver.

Extending the DefaultDependencyResolver

When it comes to writing your own dependency resolver for SignalR, you have two options: - You can implement IDependencyResolver and write the entire logic yourself (for the brave with lots of time) - You can extend the DefaultDependencyResolver and override members with your own logic (the suggested route)

The DefaultDependencyResolver Members

public virtual object GetService(Type serviceType);
public virtual IEnumerable<object> GetServices(Type serviceType);
public virtual void Register(Type serviceType, Func<object> activator);
public virtual void Register(Type serviceType, IEnumerable<Func<object>> activators);

All you need to do when extending the DefaultDependencyResolver is to override the GetService and GetServices methods. In the example below, I’m using Castle.Windsor container, but similar pattern can be used with any other DI containers. You can find few existing implementations below.

public class SignalrDependencyResolver : DefaultDependencyResolver
{
    private readonly IKernel _kernel;
 
    public SignalrDependencyResolver(IKernel kernel)
    {
        _kernel = kernel;
    }
 
    public override object GetService(System.Type serviceType)
    {
        //check if component exists in your container, if not use base to resolve
        return _kernel.HasComponent(serviceType) ? _kernel.Resolve(serviceType) : base.GetService(serviceType);
    }
 
    public override IEnumerable<object> GetServices(Type serviceType)
    {
        var objects = _kernel.HasComponent(serviceType) ? _kernel.ResolveAll(serviceType).Cast<object>() : new object[] { };
        return objects.Concat(base.GetServices(serviceType));
    }
}

In order to wire up your custom DependencyResolver, you need to set the DependencyResolver via GlobalHost.

protected void Application_Start()
{
    var container = CreateContainer();    
    //container init code ...
 
    //create dependency resolver
    var signalrDependency = new SignalrDependencyResolver(container.Kernel);
    GlobalHost.DependencyResolver = signalrDependency;
    RouteTable.Routes.MapHubs(signalrDependency);
 
    //further startup code
    //...
}

With the setup above, you can register all your hubs in your container.

public class HubsInstaller : IWindsorInstaller
{
    public void Install(IWindsorContainer container, IConfigurationStore store)
    {
        container.Register(
            AllTypes.FromAssembly(typeof(MvcApplication).Assembly)
                .BasedOn<Hub>().Configure(
                    x => x.Named(x.Implementation.FullName.ToLower()).LifeStyle.Transient));
    }
}

Your container will then auto-resolve any services that are already registered.

public class ChatHub : Hub
{
    private readonly IRepository _repository;
 
    public ChatHub(IRepository repository)
    {
        _repository = repository;
    }
 
    //methods ...
}

Under The Hood Of DefaultDependecyResolver

You can check out the default dependency resolver at GitHub. You can find all the services that are registered – the resolver keeps then in a private readonly dictionary Dictionary<Type, IList<Func