While this is great, I wanted to see how far I can get trying to get Visual Studio to connect and remote-debug a native app running inside a Linux Docker container. As it turns out there's an informative blog post on this topic as well. The basic idea is that we build a Docker image with all the development tools we need along with an SSH server and then we spin it up and remote debug from Visual Studio like how we do with normal Linux servers. Visual Studio remains oblivious to the fact that the program is running inside a Docker container.

Secure computing mode policies

When I attempted to follow along with what's in the blog post however I ran into the following error when I tried to launch the debugger:

Unable to start debugging. Unexpected GDB output from command "-interpreter-exec console "target remote localhost:18358"". Remote connection closed  

After much googling I stumbled upon an image in Docker hub which included some instructions as to how we are to use that image to debug from Visual Studio. Here's the Docker run command that was being proposed that we use:

docker run -d -p 12345:22 --security-opt seccomp:unconfined ducatel/visual-studio-linux-build-box  

The specific option of interest is --security-opt seccomp:unconfined. I use Docker for Windows as my local Docker installation. Docker for Windows makes it drop-dead easy to get going with a fully functional Docker environment. You download and install an MSI (or a DMG on macOS to get Docker setup on your Mac) and everything pretty much just works. As it turns out this results in the creation of a Hyper-V virtual machine (VM) on my Windows host running a MobyLinux distribution. The Linux kernel in the VM appears to have seccomp enabled. Secure computing mode or seccomp is an execution mode on Linux that restricts the set of system calls a given process is allowed to make. One can define a "seccomp profile" to be explicit about exactly which system calls are allowed and which aren't.

Docker now has built in support for seccomp, i.e., it spins up containers with a default seccomp profile enabled. While this default profile tends to work well for a vast majority of the use cases out there, it so happens that it disables some system calls that are necessary for enabling remote debugging from Visual Studio. Passing --security-opt seccomp:unconfined while running a container essentially causes Docker to disable seccomp restrictions on the container.

After disabling the seccomp profile Visual Studio was able to connect and remote debug perfectly. Visual Studio also includes a built-in Linux Console which is essentially a shell connected to your program's standard input and output (yep, input too!). This means you get to see the console output right there within Visual Studio and you can interactively respond to your program as well.

Generating Dockerfiles for your applications

After having figured out how to get remote debugging working I wanted to be able to automate the creation of Dockerfile definitions for my needs. I put together a small Node.js app to implement this automation. The source is hosted up on GitHub here:

https://github.com/avranju/docker-linux-dev-image

To run the app do the following:

• Install a recent version of Node.js

• Install the Yarn package manager (if you prefer npm however then that should work too - just replace running yarn below with npm install)

• Clone the repo

The repo includes a file called Dockerfile.template which you can customize to fit your needs. Just make sure you don’t remove the instructions already there for setting up an SSH server. Now open a terminal/command prompt and run the following from the folder where you cloned the repo:

$ yarn
$ node app.js

This will produce output that looks like this:

$ node app.js
[*] Reading Dockerfile template.
[*] Creating output folder C:\code\docker-linux-dev-image\output\HyoI9qTUe.
[*] Generating new keypair.
[*] Saving private key file.
[*] Saving public key file.
[*] Generating Dockerfile.

> The SSH keys and Dockerfile are in the folder C:\code\docker-linux-dev-image\output\HyoI9qTUe.

The generated Dockerfile along with SSH keys is dropped into the output folder. CD into the output folder and build the Docker image the usual way:

$ cd output/HyoI9qTUe
$ docker build -t vsdebug .

If everything goes well you'll find the newly minted image in your Docker engine. Now to run a container from this image you'd run the following command:

docker run -d -p 2222:22 \  
           --security-opt seccomp:unconfined \
           vsdebug

This will spin a container up with an SSH server listening on local port 2222. You can test this out by SSHing to this server like so:

$ cd output/HyoI9qTUe
$ ssh -i id_rsa -p 2222 root@localhost

That’s it! Now you should be able to remote debug your apps to your heart’s content. Combining this with the support for volume mounting Windows folders into a Linux Docker container using Docker for Windows, I am able to get the perfect setup! You first enable sharing your Windows drives with Docker from the Settings screen in Docker for Windows.

And then you spin your container up like so and you find yourself lost in debugging nirvana!

$ docker run -d -p 2222:22 \
             -v c:/code/my-linux-app:/code/my-linux-app \
             --security-opt seccomp:unconfined \
             vsdebug

Hope this helps you get a jump-start with your own Visual Studio/Windows/Linux development setup.

So what is Docker Swarm?

Swarm basically allows you to treat a cluster of nodes, i.e. a collection of physical/virtual machines, as one giant Docker host. It attempts to abstract away from you the fact that there are a cluster of individual Docker hosts that are actually running your containers for you. Simple enough, isn't it? The basic idea is that you stick a bunch of VMs (or physical machines) behind a swarm manager service which then sets about providing the Docker host REST API for you. The swarm manager's implementation of the Docker host API will simply delegate the actual work to one or more of the worker nodes that it has access to.

Since the swarm manager's API is basically exactly equivalent to the Docker host API, it means that in order to manage the cluster, you can use pretty much the exact same tooling that you use today when you deal with a single Docker host. You can, for instance, use the Docker CLI to deal with a swarm manager host just as you would a regular Docker host. For those who prefer looking at a picture instead of reading a bunch of text (well, any more than you already have), here's a graphic that shows how it works. Image credits: Docker Inc. on SlideShare.

The swarm software itself is available as a public Docker image on Docker Hub (apart from being open source that is).

Pluggable Schedulers

So when you ask a swarm manager to spin up a container how exactly does it know which of the possibly 100s of nodes you've got it configured with to use? Swarm comes with a built-in scheduler that can figure things out by itself but is also designed to make the scheduling process pluggable - meaning, you can replace it's built-in scheduler with another one of your choice if you so feel like it. For example, you can use the Apache Mesos project's scheduler and hook it up with Docker Swarm so that Mesos takes care of picking out the best node to spin up a given container.

Pluggable node discovery

Swarm supports multiple mechanisms for associating nodes (which can be physical or virtual machines) with an instance of the swarm manager. As with scheduling, there's a bulit-in hosted discovery service provided by docker.com which you can choose to use or set one up yourself using etcd, consul, zookeeper or just a plain text file containing host names and IP addresses.

My demo for my talk

So there, now you know what Docker Swarm is all about (kind of). For my talks though I needed a way of easily spinning up and tearing down Docker Swarm clusters on Azure. Remember, this was in March of 2015 and support for Docker Swarm was just beginning to show up in Docker Machine which is a tool that lets you easily create and manage Docker host VMs and Swarm clusters. So I quickly put together a few bash scripts to automate provisioning of the VMs for my Swarm cluster. This post is about those scripts and how you can use them for your own needs.

Firstly, the bash scripts are open source and hosted on GitHub here:

https://github.com/avranju/azure-swarm

The main two script files in question are the following:

• swarm-up.sh - this brings up the cluster for you
• swarm-down.sh - this tears down the cluster you created using swarm-up.sh

Setting up your PC

Before you can run the scripts you'll need to do the following:

1. If you're on Windows, then install Git so that you get the Git Bash console. If you're on Mac/Linux, well, you already have bash.
2. Install Node.js using your favorite method. I myself like Node Version Manager (NVM) to manage my node.js versions (there's a Windows version available too).
3. Install Git if you don't have it already.
4. Install json from NPM from a terminal like so: npm install -g json
5. Install the Azure CLI like so: npm install -g azure-cli. Configure the Azure CLI with a valid Azure Subscription. If you don't know how to do that then this handy guide should help.
6. Clone the repo like so:

git clone https://github.com/avranju/azure-swarm.git

Running the scripts

Running the script isn't very hard. To setup a cluster with default options (1 small master VM and 2 small worker node VMs located in "West US") just run this:

./swarm-up.sh

This will do the following:

1. Generate new SSH keys
2. Create a new storage account and container
3. Create a new Azure virtual network
4. Spin up a VM to run the Swarm Manager service in the virtual network created in step 3
5. Spin up as many worker node VMs as needed (again, in the same virtual network)
6. Create a bunch of files in a folder called output.

If everything goes well you should have a Docker Swarm cluster of your own with everything hooked up.

Output files

Each run of the script is identified by a randomly generated 8 character long hex string. For e.g. you might get this: 35f8fa98. A file containing this ID is produced in the output folder. For instance, for the ID 35f8fa98, the file would be called swarm-35f8fa98.deployment. You'll see in a bit why this is important.

Another file that you'll be interested in is a file containing SSH cofiguration information. For the same deployment ID as before, this file will be called ssh-35f8fa98.config. You can use this file to SSH into any of the VMs. For example, to SSH into the swarm-master VM, you'd run the following command:

ssh -F output/ssh-35f8fa98.config swarm-master  

The same command will work for any of the worker node VMs (just change swarm-master to swarm-00 or swarm-01 and so forth).

Tearing down the cluster

The whole deployment ID shebang that I described above pays off when it comes to tearing down everything because having a deployment ID allows us to cleanly delete the deployment. Continuing with the same deployment ID as before, bringing a cluster down involves running the following script:

./swarm-down.sh output/swarm-35f8fa98.deployment

swarm-down.sh will attempt to delete everything that swarm-up.sh created - virtual network, cloud service, VMs and storage account. This will work even with partially deployed clusters (for e.g. you started running the script and then stopped it mid-way because, well, let's just say you had your reasons) because in that case the script will simply attempt to delete something that doesn't exist which is, well, harmless.

Customize your deployment

There are a few options that you can customize by editing the value of various variables in the options.sh file. Here're the ones you're likely to be interested in:

• VNET_LOCATION - The Azure data center where your VMs will be provisioned. This is "West US" by default.
• VM_SIZE - The size of the VMs. Accepts any valid size string that designates a VM size. This is "Small" by default.
• VM_IMAGE - This is the name of the Linux VM image to use. By default this is Ubuntu 14.04 LTS. Ubuntu 15 doesn't work with this script at this point since Ubuntu has switched to systemd for running system services from v15 onwards while the script relies on it being upstart.
• VM_USER_NAME - The SSH user name. "avranju" by default.
• SWARM_WORKER_NODES - The number of worker VMs to spin up. This is 2 by default.

Running your containers

To run your containers you'll want to SSH into the swarm-master VM and set your environment up so that it points to the Swarm Manager service which is itself running as a container listening on port 2377 on the VM. Using the output files generated by swarm-up, you'd do the following:

$ ssh -F output/ssh-35f8fa98.config swarm-master
avranju@swarm-master:~$ export DOCKER_HOST=0.0.0.0:2377
avranju@swarm-master:~$ docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 851c91a
OS/Arch (client): linux/amd64
Server version: swarm/0.4.0
Server API version: 1.16
Go version (server): go1.4.2
Git commit (server): d647d82
OS/Arch (server): linux/amd64

As you can tell from the text in yellow highlight the CLI is talking to a Docker Swarm host. Now you can go ahead and start spinning up containers willy nilly and Docker Swarm should dutifully schedule them on your worker node VMs.

Finis

That's pretty much it. As always, please feel free to fork, modify, send pull requests etc on these scripts and/or sound out in the comments below.

1. Replace all instances of . and _ with a single white space.
2. Title case each white space delimited word.

Simple enough. I figured it might be an interesting exercise implementing this in the various programming languages that I have varying levels of familiarity with. Here goes.

Java

For various reasons we needed to support Java 6 and up for the plugin. I am fairly new to the Java world so at first it seemed like I was going to have to implement this by hand till I discovered the immensely useful Google Guava library. With Google Guava this turns out to be a function that looks like this:

private String scrubString(String name) {
  // replace all instances of . and _ with white space
  CharMatcher matcher = CharMatcher.anyOf("._");
  name = matcher.replaceFrom(name, ' ');

  // split the string into a sequence delimited by white space
  Iterable<String> tokens = Splitter.on(' ').split(name);

  // this function, given a string returns a title cased
  // version of it
  Function<String, String> makeTitleCase =
      new Function<String, String>() {
        @Override
        public String apply(String str) {
          return Character.toUpperCase(str.charAt(0)) +
              str.substring(1);
        }
      };

  // transform the tokens into their title-cased counterparts
  Iterable<String> titleCaseTransformer = Iterables.transform(
      tokens, makeTitleCase);

  // re-join the title-cased scrubbed strings using white space
  return Joiner.on(' ').join(titleCaseTransformer);
}

That's, well, verbose. If I wanted a terser version of this, I could do this:

private String scrubString(String name) {
  return Joiner.on(' ').
      join(Iterables.transform(
          Splitter.on(' ').split(
              CharMatcher.anyOf("._").
                  replaceFrom(name, ' ')),
          new Function<String, String>() {
            @Override
            public String apply(String str) {
              return Character.toUpperCase(
                  str.charAt(0)) + str.substring(1);
            }
          }));
}

But that's of course, far less readable. With Java 8 lambda syntax however this can be simplified somewhat.

private String scrubString(String name) {
  return Joiner.on(' ').
      join(Iterables.transform(
          Splitter.on(' ').split(
              CharMatcher.anyOf("._").
                  replaceFrom(name, ' ')),
          str -> Character.toUpperCase(str.charAt(0)) +
                    str.substring(1)));
}

Though the only piece of code that was replaced is the callback routine that transforms regular strings to their title-cased counterparts, it does however declutter the code a fair bit.

C#

With C#'s support for LINQ this turns out to be far terser.

private string ScrubString(string str)
{
  return String.Join(" ",
    from p in new Regex (@"[._]").Replace(str, " ").Split(' ')
    select Thread.CurrentThread.
            CurrentCulture.TextInfo.ToTitleCase(p));
}

I wrote that first and then realized that given that we have the ToTileCase method it's a bit of an overkill to split and join the string. Here's a simpler version:

private string ScrubString (string str)
{
  return Thread.
         CurrentThread.
         CurrentCulture.
         TextInfo.
         ToTitleCase (new Regex (@"[._]").Replace (str, " "));
}

Python

With Python's support for list comprehension this ends up being even terser than C#.

import string
import re

def string_scrub(str):
  return string.join([s.title() for s in \
      string.split(re.sub('[._]', ' ', str))])

C++ 11

Here's my take on this using C++ 11 capabilities:

string scrub(const string& input) {
  regex re { "[._]" };
  string str = regex_replace(input, re, " ");

  vector<string> tokens;
  split(str, ' ', tokens);

  transform(tokens.begin(), tokens.end(), tokens.begin(),
      [](const string& s) {
        return title_case(s);
      });

  return join(tokens, ' ');
}

string title_case(const string& str) {
  return string(1, toupper(str[0])) + str.substr(1);
}

vector<string>& split(
    const string& str,
    char delimiter,
    vector<string>& tokens) {
  string item;
  stringstream ss(str);
  while(getline(ss, item, delimiter))
    tokens.push_back(item);
  return tokens;
}

string join(const vector<string>& tokens, char delimiter ) {
  ostringstream ss;
  bool first = true;
  for_each(tokens.begin(),
      tokens.end(),
      [&ss, &first, &delimiter](const string& s) {
        if(first) {
          first = false;
        } else {
          ss << delimiter;
        }
        ss << s;
      });

  return ss.str();
}

JavaScript (of course!)

Here's the JavaScript version (using ES2015 syntax).

function scrubString(str) {
  return str.
    replace(/[._]/g, ' ').
    split(' ').
    map(s => `${s.charAt(0).toUpperCase()}${s.substr(1)}`).
    join(' ');
}

I really like the nice fluent manner in which we are able to translate the requirements into an implementation in JS.

Common Lisp

It's been a while since I have dabbled in Common Lisp, but after some fervent searching here's what I came up with. Note that this does use a library that is not part of the standard Common Lisp distribution called CL-PPCRE which appears to be a fairly popular regular expression library for Common Lisp.

(load "~/quicklisp/setup.lisp")

(ql:quickload :cl-ppcre)

(defun scrub_string (str)
  (string-capitalize (cl-ppcre:regex-replace-all "[._]" str " ")))

There. If I had to pick a favorite I'd have to say I like the JavaScript version the best. The Common Lisp and the C# versions aren't too bad either. What do you think? Sound off in the comments!

... some meta data tags here ...
<Rows>
    <TableRow>
      <Fields>
        <Field>Blah blah blah</Field>
        <Field>Stuff here</Field>
        <Field>3</Field>
        <Field>More stuff</Field>
      </Fields>
    </TableRow>
    <TableRow>
      <Fields>
        <Field>Blah blah blah and blah</Field>
        <Field>Stuff here</Field>
        <Field>2</Field>
        <Field>More stuff</Field>
      </Fields>
    </TableRow>
</Rows>

Here's the processing that needed to be done:

1. Extract the text contents from the first Field tag from under Fields under each TableRow.
2. Filter out rows that didn't match a specific regular expression.
3. Extract a specific sub-string from the text.
4. Group the data on the sub-string and compute the number of times that string occurs.
5. Produce the output as a comma separated value (CSV) file.

Extracting text from the Field tag

For step 1, since we're dealing with extremely large XML files, using a DOM based parser was out of the question since that wouldn't be very memory efficient. I wrote a small utility in C++ (called get-msg) using the XmlLite parser that's been shipping in Windows since Vista days! XmlLite is a native component modeled on .NET's XmlReader and XmlWriter types. It is a forward only, stream processing pull parser which means that it has extremely low memory footprint and can deal with XML inputs of arbitrary size. On the flip side, the programming model isn't quite as convenient as a DOM parser.

The following snippet shows how you can load up an XML document using XmlLite. TableReader is a simple class I put together to make working with XmlLite easier. The variable _reader below is a member instance of type CComPtr<IXmlReader> and _fileStream is another member of type CComPtr<IStream>.

bool TableReader::Load(wstring file)
{
    // free up current reader and stream
    _reader.Release();
    _fileStream.Release();

    // load up file
    HRESULT hr = SHCreateStreamOnFile(
        file.c_str(),
        STGM_READ,
        &_fileStream);
    if (FAILED(hr)) {
        return false;
    }
    hr = CreateXmlReader(
        __uuidof(IXmlReader),
        (void **) &_reader,
        nullptr);
    if (FAILED(hr)) {
        return false;
    }
    hr = _reader->SetInput(_fileStream);
    if (FAILED(hr)) {
        return false;
    }

    // move to the "Rows" element
    if (MoveToElement(L"Rows") == false) {
        return false;
    }

    return true;
}

The code should be fairly self-explanatory. The MoveToElement method right at the end of the method is a member method of the TableReader class that's intended to make the job of navigating the node tree easier. Here's what this method looks like:

bool TableReader::MoveToElement(wstring elementName)
{
    HRESULT hr;
    XmlNodeType nodeType;
    LPCWSTR wszLocalName = nullptr;

    while ((hr = _reader->Read(&nodeType)) == S_OK) {
        switch (nodeType) {
            case XmlNodeType_Element:
            {
                hr = _reader->GetLocalName(&wszLocalName, nullptr);
                if (FAILED(hr)) {
                    return false;
                }

                // check if the local name is the same as
                // "elementName" and if yes, then we're
                // done 
                if (elementName.compare(wszLocalName) == 0) {
                    return true;
                }
                break;
            }
        }
    }

    return SUCCEEDED(hr);
}

As you can tell, all it does is to keep walking the nodes in the XML document till it encounters an element whose name matches elementName. With this method handy, looking for the specific Field XML tag in question becomes fairly straightforward. Here's the method that does the job:

bool TableReader::ReadMessage(LPCWSTR *ppwszMsg)
{
    HRESULT hr;

    // move to next "TableRow" element
    if (!MoveToElement(L"TableRow")) {
        return false;
    }

    // move to first "Field" element
    if (!MoveToElement(L"Field")) {
        return false;
    }

    // move reader to the "text" part of the element
    XmlNodeType nodeType;
    hr = _reader->Read(&nodeType);
    if (nodeType != XmlNodeType_Text &&
        nodeType != XmlNodeType_EndElement) {
        return false;
    }

    // retrieve the message
    *ppwszMsg = nullptr;
    hr = _reader->GetValue(ppwszMsg, nullptr);
    return SUCCEEDED(hr);
}

The final program is then basically a tight loop that keeps calling ReadMessage till it returns false. Here are the relevant bits.

wstring fileName{ argv[1] };  
TableReader reader;  
if (reader.Load(fileName) == false) {  
    wcout << L"Attempt to load the XML file failed." << endl;
    return 1;
}

// read and print all the messages
LPCWSTR pwszMsg;  
while (reader.ReadMessage(&pwszMsg)) {  
    // we use wprintf instead of wcout because wcout seems to have
    // trouble dealing with embedded byte order mark byte sequences
    // for some reason
    wprintf(L"%s\n", pwszMsg);
}

Getting the tools - GnuWin

Now that we have a way of rapidly extracting the Field element that we're interested in from the source XML the rest of the text processing work turns out to be fairly straightforward when we have the right tools handy. The first thing to do is to install the GnuWin package via Chocolatey. If you don't know what is Chocolatey and you're a Windows user then you really should get to know it! Briefly, Chocolatey is a command line package manager for Windows - apt-get for Windows if you will. GnuWin is a package that basically installs Win32 ports of all the key Unix/Linux tools without having to rely on a heavyweight "environment" like Cygwin. Installing GnuWin is a simple matter of running the following from a command prompt:

cinst GnuWin

That's it. It does take a while to pull in all the files and get setup though.

Processing the text

The tools we're going to use to get the job done are essentially - grep, sed, sort and uniq. Here are the commands I used.

Filter out rows that didn't match a specific regular expression:

grep "Creating OSDisk from OSImage\:.*"

Extract a specific sub-string from the text:

sed -n "s/.*Creating OSDisk from OSImage:\(.*\).*/\1/p"

Group the data on the sub-string and compute the number of times that string occurs:

sort | uniq -c

Produce the output as a comma separated value (CSV) file:

sed -n "s/ *\([0-9]*\) \(.*\)/\2,\1/p"

What we do is to basically pipe everything together like so:

get-msg input.xml |
  grep "Creating OSDisk from OSImage\:.*" |
  sed -n "s/.*Creating OSDisk from OSImage:\(.*\).*/\1/p" |
  sort | uniq -c |
  sed -n "s/ *\([0-9]*\) \(.*\)/\2,\1/p" 

And finally output redirect everything to a .csv file. That's pretty much it! Processing a 14 GB XML document through this pipeline on my quad core Intel i7 2014 Lenovo Carbon with 8 GB of RAM (and a truly horrendous keyboard) takes about 5 minutes. Not bad eh?

If you need to convert files from one markup format into another, Pandoc is your swiss-army knife.

Pandoc works by connecting readers and writers via an intermediate JSON based abstract syntax tree (AST) representation of the document.

This allows one to independently code up readers and writers targeting just the AST and you are automatically able to convert between pretty much all supported formats. A comprehensive library of readers and writers is supported by the tool out-of-the-box including, happily for me, a reader that can parse HTML and a writer that can produce markdown. You convert a HTML document into markdown by running the tool from the command line like so:

 pandoc -f html input.html -t markdown_strict -o output.md

Here, -f indicates that the source format is HTML, -t indicates that the target format is markdown and -o signifies the name of the the output file. If you don't supply a file name via the -f and -o options then the tool will simply read from and write to standard input and output respectively.

Manipulating the AST

The real power of the tool, in my opinion, lies in the fact that you can have the tool output the AST representation of the source document as JSON which you can then programmatically manipulate any way you like. Here's how you generate the JSON for a given HTML document:

pandoc -f html input.html -t json -o output.json

Pandoc also supports reading from and writing to standard input and output streams which when combined with some input/output redirection you can build some really nifty document processing pipelines. In my case, the markdown produced by Pandoc for some of my posts had some problems. For example, in some cases the markdown output would include markup like this - **** - which basically represents the presence of empty <b></b> or <strong></strong> tags in the source HTML. I wrote up a little node.js app to fix up this issue. I had the node app read JSON from the standard input stream and output the processed JSON to standard output. Once I had it working the way I wanted, I was able to put together a command such as the following:

pandoc -f html input.html -t json | node app.js | pandoc --no-wrap -f json -t markdown_strict -o output.md

The first part of the command converts the input HTML document into an AST representation which it then writes out to standard output as JSON. The JSON output is piped to the node.js app which loads it up into memory, walks the tree, fixes up document nodes that contain empty Strong elements by removing them and then outputs the modified AST as JSON to standard output. The modified AST JSON is then piped back to pandoc which proceeds to convert it to markdown. Easy-peasy!

Pandoc filters

The node.js app that removed the redundant Strong elements is an example of a filter app. The source for the filters I used to scrub the posts from my blog is available on Github. First I implemented the filters I needed as an array of functions in filters.js. Here for example is the filter that removes empty Strong elements:

{
    name: "Remove empty bold/strong tags",
    nodeType: "Strong",
    apply: function(content) {
        // if content's c array is empty then remove the
        // node
        if(content &&
           content["c"] &&
           Util.isArray(content.c) &&
           content.c.length === 0 ) {
               return null;
        }

        return content;
    }
}

In app.js I first load up the JSON from standard input like so:

function loadJsonAsync() {
    var deferred = Q.defer();
    var json = "";
    process.stdin.setEncoding("utf8");
    process.stdin.on("readable", function() {
        var chunk = process.stdin.read();
        if(chunk !== null) {
            json += chunk;
        }
    });
    process.stdin.on("end", function() {
        deferred.resolve(json);
    });

    return deferred.promise;
}

The filters are initialized via require like this:

var Filters = require("./filters.js").Filters;

And then the app proceeds to process the JSON like so:

// load up the json from stdin
loadJsonAsync().done(function(json) {
    var ast = JSON.parse(json);
    // apply all of our filters
    Filters.forEach(function(filter) {
        ast[1] = walk(ast[1], filter);
    });
    console.log(JSON.stringify(ast));
});

walk is a helper function I wrote to recursively visit each node in the syntax tree and apply the supplied filter on it. Here's the full function implementation:

function walk(content, filter) {
    if(typeof(content) !== "object") {
        return content;
    }
    if(Util.isArray(content)) {
        return content.map(function(item) {
            return walk(item, filter);
        }).filter(function(item) {
            return (item !== null);
        });
    } else {
        if(filter.nodeType === "*" ||
           filter.nodeType === content.t) {
            // If a filter's `apply` method returns null then, it is
            // removed from the AST. If `apply` returns an object
            // then the node is replaced with what's returned from
            // the filter. If `apply` returns an array then the
            // array is spliced in.
            var node = filter.apply(content);
            if(node) {
                // splice array if its an array
                if(Util.isArray(node)) {
                    throw new Error("Not implemented yet.");
                } else {
                    content.t = node.t;
                    content.c = node.c;
                }
            } else {
                return null;
            }
        }

        if(content.c) {
            return {
                t: content.t,
                c: walk(content.c, filter)
            };
        }
    }

    // I don't think we should ever get here.
    return content;
}

And that's about it. I wrote a little driver program to iterate through all my 70 odd posts and then run them through my pandoc based document processing pipeline to be left with pristine markdown that I can then take and load up into Ghost! There were still a few manual tweaks required for some of the posts but those were few and far between and definitely doable by hand. Yay pandoc!

Nerdworks reloaded

An aesthetic refresh was overdue and I also wanted to move away from my home-brew blog engine to a more contemporary implementation. After much consideration, hard work and toil I present to you ladies, gentlemen, boys and girls - this 2nd version of Nerdworks Blogorama! In keeping with the design zeitgeist of the day, the blog now features a spare clean typography driven design and I've kept the features down to a bare minimum (all there is is a search box really) and have let the content take center stage. I hope you like this!

So what is this built on?

I hope to provide some insight into how this new incarnation of the blog works over a series of blog posts the next few days. Right now though, here's some information on what some of the key pieces are:

1. The site runs on the Ghost blogging platform which is an ExpressJS based blogging framework built on Node.JS. I forked the Ghost Github repository and made a whole bunch of tweaks and changes to get it to work the way I wanted.
2. I used the excellent pandoc tool to convert all of the HTML from my old blog into markdown that I can use with Ghost. I had to do a little pre and post processing on the content to scrub the posts a bit. More on that in a separate post.
3. I integrated the Disqus commenting system and wrote a couple of utilities to export all the comments from the old blog into the custom XML format that Disqus requires, in order to import them into the new blog.
4. One of the requirements I set myself when starting on this project was to make sure that all the existing links continue to work just fine. A good part of the customization I did on the base Ghost code base was in enabling additional routing logic for my old .aspx URLs (you don't want to end up losing all the good Google and Bing karma that you've built up over the years).
5. I used a forked copy of the Ghost Vapor theme for the layout and visual styling. Again, had to make a set of changes to incorporate everything I wanted in the UX.
6. I setup a virtual machine on Azure to host an instance of the absolutely fantastic open source search and analytics system Elastic Search. I then wrote a little program to submit all of my existing blog content to the search engine and had it indexed. And finally, I modified the Vapor theme and Ghost to add support for search. A full treatment of how all this works warrants a separate post in its own right.
7. And finally I also modified the Grunt task to add support for automating building and deployment of the site to Azure websites.

So there you have it. If you want to drop a note to me you can use my spiffy new contact form to write an email. Or you can leave a comment below of course. So long!

 template<int index, typename... Ts>
 struct print_tuple {
     void operator() (tuple<Ts...>& t) {
         cout << get<index>(t) << " ";
         print_tuple<index - 1, Ts...>{}(t);
     }
 };

Clearly, we’ll need to define a base case to break out of the compile time recursion for the print_tuple call in the function call operator overload – which in this case would be when the non-type template parameter index is equal to zero. So I went ahead and defined the following partial specialization of print_tuple specializing on the value zero for the non-type template parameter index.

 template<typename... Ts>
 struct print_tuple<0, Ts...> {
     void operator() (tuple<Ts...>& t) {
         cout << get<0>(t) << " ";
     }
 };

The reason I had to use a variadic template struct instead of a regular variadic template function is that it is not possible to partially specialize a template function in C++. Using a struct/class allows us to do so. Now that we have this setup, we can write a utility print routine to wrap calls to print_tuple and we should be done. Here’s a complete example:

 #include <iostream>
 #include <tuple>

 using namespace std;

 template<int index, typename... Ts>
 struct print_tuple {
     void operator() (tuple<Ts...>& t) {
         cout << get<index>(t) << " ";
         print_tuple<index - 1, Ts...>{}(t);
     }
 };

 template<typename... Ts>
 struct print_tuple<0, Ts...> {
     void operator() (tuple<Ts...>& t) {
         cout << get<0>(t) << " ";
     }
 };

 template<typename... Ts>
 void print(tuple<Ts...>& t) {
     const auto size = tuple_size<tuple<Ts...>>::value;
     print_tuple<size - 1, Ts...>{}(t);
 }

 int main() {
     auto t = make_tuple(1, 2, "abc", "def", 4.0f);
     print(t);

     return 0;
 }

All that print does is to first determine the size of the tuple via tuple_size and then instantiates print_tuple and invokes the function call operator passing the tuple object in question. As you might have noticed we are essentially working our way backwards till we hit the base case where index is zero – i.e. it’ll print the tuple members in the reverse order. Here’s the output this produces:

 4 def abc 2 1

Iterate in order?

I figured, implementing a version that iterates over the tuple members in the order that they are specified should be fairly straightforward. We should just need to define a different base case (for the last item in the tuple instead of the first) and the recursive implementation should simply increment the index instead of decrementing it. Here’s what I came up with:

 template<int index, typename... Ts>
 struct print_tuple {
     void operator() (tuple<Ts...>& t) {
         cout << get<index>(t) << " ";
         print_tuple<index + 1, Ts...>{}(t);
     }
 };

 template<typename... Ts>
 struct print_tuple<tuple_size<tuple<Ts...>>::value - 1, Ts...> {
     void operator() (tuple<Ts...>& t) {
         cout << get<tuple_size<tuple<Ts...>>::value - 1>(t) << " ";
     }
 };

The code shown in bold above are the changes of interest. In particular, turns out, the base case definition which handles the situation when print_tuple is being instantiated with an index that is equal to the size of the tuple minus one, is not really valid C++. Non-type template specialization in C++ can only be done using “simple identifiers”. The expression tuple_size<tuple<Ts...>>::value - 1 is a compile time constant for sure and ideally the compiler should be able to compute that value (which in fact it does for the code in the body of that method definition) but, well, it doesn’t! So, we’re kind of out of luck there.

Generalized iteration?

One might imagine that it should be possible to generalize this iteration (even if it can be done in reverse order only) so that we are able to supply arbitrary callbacks for processing tuple members. Turns out this again, is not possible which I think is reasonable. Because at that point one really needs to question whether using a tuple is the correct choice – a std::vector or std::list or some of the other containers maybe a more appropriate option. Having said that, you might be thinking that we should still be able to generalize this by adding another template parameter for a callback routine and passing in a template function for that parameter. Maybe something like this?

 #include <iostream>
 #include <tuple>

 using namespace std;

 template<int index, typename TCallback, typename... Ts>
 struct iterate_tuple {
     void operator() (tuple<Ts...>& t, TCallback callback) {
         callback(get<index>(t));
         iterate_tuple<index - 1, TCallback, Ts...>{}(t, callback);
     }
 };

 template<typename TCallback, typename... Ts>
 struct iterate_tuple<0, TCallback, Ts...> {
     void operator() (tuple<Ts...>& t, TCallback callback) {
         callback(get<0>(t));
     }
 };

 template<typename TCallback, typename... Ts>
 void for_each(tuple<Ts...>& t, TCallback callback) {
     iterate_tuple<tuple_size<tuple<Ts...>>::value - 1, TCallback, Ts...> it;
     it(t, callback);
 }

 template<typename T>
 void print(T v) {
     cout << v << " ";
 }

 int main() {
     auto t = make_tuple(1, 2, "abc", "def", 4.0f);
     for_each(t, print);

     return 0;
 }

This won’t work because the compiler needs to know the type of TCallback when it is instantiating the for_each template function which in this case it doesn’t and neither do we know the type because we want a different version of print to be used for each unique type in the tuple. If there is some way of telling the compiler to postpone resolution of TCallback till it is actually used then this might have worked. As far as I know, that isn’t possible. But then I might be wrong. If you know a way of doing that, it’ll be great if you could please let me know in the comments.

https://github.com/avranju/AudioPlayerWithCustomStream

Playing media streams from memory

The primary requirement proved to be fairly straightforward to accomplish. Turns out, there already exists an SDK sample showing exactly this. The sample shows how to achieve media playback from memory streams using the Windows.Media.Core.MediaStreamSource object. Briefly, here are the steps:

1. First you go fetch some metadata from the media stream. In case of audio content, this turns out to be the sample rate, encoding bit rate, duration and number of channels. For file based audio sources, the Windows.Storage.StorageFile object has the ability to extract this information from the file directly via Windows.Storage.StorageFile.Properties.RetrievePropertiesAsync. Here's an example function that accepts a StorageFile object as input and then extracts and returns the said metadata from it.

   function loadProps(file) {
       var props = {
           fileName: "",
           sampleRate: 0,
           bitRate: 0,
           channelCount: 0,
           duration: 0
       };
   
   
       // save file name
       props.fileName = file.name;
       return file.properties.getMusicPropertiesAsync().then(
        function (musicProps) {
           // save duration
           props.duration = musicProps.duration;
   
   
           var encProps = [
               "System.Audio.SampleRate",
               "System.Audio.ChannelCount",
               "System.Audio.EncodingBitrate"
           ];
   
   
           return file.properties.
               retrievePropertiesAsync(encProps);
       }).then(function (encProps) {
           // save encoding properties
           props.sampleRate =
              encProps["System.Audio.SampleRate"];
           props.bitRate =
              encProps["System.Audio.EncodingBitrate"];
           props.channelCount =
              encProps["System.Audio.ChannelCount"];
   
   
           return props;
       });
   }
   

2. Wrap the metadata gathered in step 1 in a Windows.Media.MediaProperties.AudioEncodingProperties object which in turn is then wrapped in a Windows.Media.Core.AudioStreamDescriptor object.

3. Use the AudioStreamDescriptor object to initialize a MediaStreamSource instance and setup event handlers for the MediaStreamSource's Starting, SampleRequested and Closed events. As you might imagine, the idea is to respond to these events by handing out audio data to the MediaStreamSourcewhich then proceeds to play that content.

This is all fine and dandy, but how do we get this to work when the audio content is stored in memory in an Windows.Storage.Streams.InMemoryRandomAccessStream object? The challenge of course is in extracting the metadata we need to setup a MediaStreamSource object.

StorageFile can read from arbitrary streams?

As it happens, the StorageFile object has direct support for having it powered by an arbitrary stream (or pretty much anything really). I figured I'll hook up a StorageFile with an InMemoryRandomAccessStream object and have it extract the metadata that I needed. Here's how you connect a StorageFile with data fetched from any arbitrary source - in this case, just a string constant. You create a StorageFile object by calling StorageFile.CreateStreamedFileAsync. CreateStreamedFileAsync requires that you pass a reference to a callback routine which is expected to supply the data the StorageFile object needs when it is first accessed. Here's a brief example:

function init() {
    var reader;
    var size = 0;

    Windows.Storage.StorageFile.createStreamedFileAsync(
           "foo.txt", generateData, null).then(
       function (file) {
        // open a stream on the file and read the data;
        // this will cause the StorageFile object to
        // invoke the "generateData" function
        return file.openReadAsync();
    }).then(function (stream) {
        var inputStream = stream.getInputStreamAt(0);
        reader = new Windows.Storage.Streams.DataReader(inputStream);
        size = stream.size;
        return reader.loadAsync(size);
    }).then(function () {
        var str = reader.readString(size);
        console.log(str);
    });
}

function generateData(stream) {
    var writer = new Windows.Storage.Streams.DataWriter();
    writer.writeString("Some arbit random data.");

    var buffer = writer.detachBuffer();
    writer.close();

    stream.writeAsync(buffer).then(function () {
        return stream.flushAsync();
    }).done(function () {
        stream.close();
    });
}

The problem however, as I ended up discovering, is that StorageFile objects that work off of a stream created in this fashion do not support retrieval of file properties via StorageFile.Properties.RetrievePropertiesAsync or for that matter StorageFile.Properties.GetMusicPropertiesAsync. So clearly, this approach is not going to work. Having said that its useful to know that this technique is possible at all with StorageFile objects as it allows you to defer performing the actual work of producing the data represented by the StorageFile object till it is actually needed. And being a bona fide Windows Runtime object you can confidently pass this around wherever a StorageFile object is accepted - for instance when implementing a share source contract you might hand out a StorageFile object created in this manner via Windows.ApplicationModel.DataTransfer.DataPackage.SetStorageItems.

Reading music metadata using the Microsoft Media Foundation

After a bit of research I discovered that there is another API that can be used for fetching metadata from media streams (among other things) called the Microsoft Media Foundation. In particular, the API features an object called the source reader that can be used to get the data we are after. The trouble though is that this is a COM based API and cannot therefore be directly invoked from JavaScript. I decided to write a little wrapper Windows Runtime component in C++ and then use that from the JS app. After non-trivial help from my colleague Chris Guzak and others directly from the Media Foundation team at Microsoft (perks of working for Microsoft I guess!) we managed to put together a small component that allows us to read the required meta data from an InMemoryRandomAccessStream object. Here's relevant snippet that does the main job (stripped out all the error handling code to de-clutter the code):

MFAttributesHelper(InMemoryRandomAccessStream^ stream, String^ mimeType)
{
    MFStartup(MF_VERSION);

    // create an IMFByteStream from "stream"
    ComPtr<IMFByteStream> byteStream;
    MFCreateMFByteStreamOnStreamEx(
           reinterpret_cast<IUnknown*>(stream),
           &byteStream);

    // assign mime type to the attributes on this byte stream
    ComPtr<IMFAttributes> attributes;
    byteStream.As(&attributes);
    attributes->SetString(
           MF_BYTESTREAM_CONTENT_TYPE,
           mimeType->Data());

    // create a source reader from the byte stream
    ComPtr<IMFSourceReader> sourceReader;
    MFCreateSourceReaderFromByteStream(
           byteStream.Get(),
           nullptr,
           &sourceReader);

    // get current media type
    ComPtr<IMFMediaType> mediaType;
    sourceReader->GetCurrentMediaType(
           MF_SOURCE_READER_FIRST_AUDIO_STREAM,
           &mediaType);

    // get all the data we're looking for
    PROPVARIANT prop;
    sourceReader->GetPresentationAttribute(
           MF_SOURCE_READER_MEDIASOURCE,
           MF_PD_DURATION,
           &prop);
    Duration = prop.uhVal.QuadPart;

    UINT32 data;
    sourceReader->GetPresentationAttribute(
           MF_SOURCE_READER_MEDIASOURCE,
           MF_PD_AUDIO_ENCODING_BITRATE,
           &prop);
    BitRate = prop.ulVal;

    mediaType->GetUINT32(
           MF_MT_AUDIO_SAMPLES_PER_SECOND,
           &data);
    SampleRate = data;

    mediaType->GetUINT32(
           MF_MT_AUDIO_NUM_CHANNELS,
           &data);
    ChannelCount = data;
}

This is the implementation of the constructor on the MFAttributesHelper ref class. As you can tell, the constructor accepts a reference to an instance of an InMemoryRandomAccessStream object and the MIME type of the content in question and proceeds to extract the duration, encoding bitrate, sample rate and channel count from it. It does this by first creating an IMFByteStream object via the convenient MFCreateMFByteStreamOnStreamEx function which basically wraps an IRandomAccessStream object (which InMemoryRandomAccessStream implements) and returns an IMFByteStream instance. The object returned by MFCreateMFByteStreamOnStreamEx also implements IMFAttributes which we then QueryInterface for (via ComPtr::As) and assign the MIME type value to it. Next we instantiate an object that implements IMFSourceReader via MFCreateSourceReaderFromByteStream and use that instance to fetch the duration and encoding bitrate values via the GetPresentationAttribute method. And finally, we retrieve an object that implements the IMFMediaType interface via IMFSourceReader::GetCurrentMediaType and use that object to fetch the sample rate and the channel count values. Once you know how to do all this, it seems quite trivial of course but getting here, believe me, took some doing!

Now that we have this component, reading the metadata from JavaScript proves to be fairly straightforward. Here's an example. In the code below, memoryStream is an InMemoryRandomAccessStream instance and mimeType is a string with the MIME type of the content:

var helper = MFUtils.MFAttributesHelper.create(memoryStream, mimeType);

// now, helper's sampleRate, bitRate, duration and channelCount
// properties contain the data we are looking for

Now with the metadata handy, we simply follow the steps as outlined earlier in this post to commence playback. As mentioned before the sample is hosted up on Github here:

https://github.com/avranju/AudioPlayerWithCustomStream

For the sake of the sample, I took a plain MP3 file and applied a XOR cipher on it and then loaded it up and played back from memory applying another XOR transform on the bits before playback. It all works rather well together and again, hat-tip to Chris Guzak for all his help in whittling down the WinRT component down to its essence and really cleaning up its interface!

Turns out, this is fairly straightforward to accomplish using IIS Express and some registry tweaks. For those of you who don't know, IIS Express is this light weight, self-contained version of IIS meant to be used for developing, debugging and testing web apps. When you use Visual Studio, the web apps themselves run inside IIS Express when you hit F5. Being self-contained, we are able to run a web server pretty much anywhere from a command prompt. Documentation on how to do this is available here. You can download and install IIS Express either via the Web Platform Installer or from here (IIS Express version 8.0 at the time of writing). If your web app files are located in say, D:\Code\Web\Foo then you'd run a web server from that location like so:

"C:\Program Files (x86)\IIS Express\iisexpress.exe" /path:"D:\Code\Web\Foo" /port:8080 /systray:true

The path to iisexpress.exe might be different if you're running on a 32-bit system. It'll just be "Program Files" instead of "Program Files (x86)". Once you've run the command, the web server starts up and you can load your web app in your favorite browser by navigating to http://localhost:8080/. The next step is to integrate this into the Explorer shell so you can run this from wherever you want directly from Explorer. Phil Haack has written up a post on how to do this with the web server that Visual Studio 2008 used to ship with way back in, well, 2008. I adapted the basic steps described there to make it work with IIS Express. Now, setting this up involves editing the Windows Registry, so please be careful with what you do. This works on my machine and that's about all I am willing to say!

If you're on a 64-bit installation of Windows, here're the changes you need to do to your registry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Directory\shell\IISExpressWebServer]
@="Web Sever Here"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Directory\shell\IISExpressWebServer\command]
@="C:\\Program Files (x86)\\IIS Express\\iisexpress.exe /path:\"%1\" /port:8080 /systray:true"

And if you're on a 32-bit installation, then this is what you need to do:

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Directory\shell\IISExpressWebServer]
@="Web Sever Here"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\Directory\shell\IISExpressWebServer\command]
@="C:\\Program Files\\IIS Express\\iisexpress.exe /path:\"%1\" /port:8080 /systray:true"

If you need .reg files so you can just double-click to import them into your registry then they are available here. You might want to edit the .reg files in case your installation paths are different from what's given there. That's pretty much it!

vector<int> nums { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
auto evens = count_if(begin(nums), end(nums), [](int num) {
    return (num % 2) == 0;
});

The third parameter passed to the standard count_if function is a predicate that is expected to return true if the value passed to it satisfies the condition and false otherwise. In the snippet above we simply count the number of instances of even numbers in the collection. Search for "C++ lambdas" on your favorite search engine and you should get plenty of material out there talking about this feature. What follows in this post are some notes on certain aspects of C++ lambdas that I happened to notice as I was learning about them listed in no particular order.

1. You can pass a lambda object around as you would pretty much anything else. Here's a made up example showing how you can pass a lambda as an argument to another function.

   #include <iostream>
   
   using namespace std;
   
   template <typename T>
   void call(T);
   
   int main() {
     auto fn = []() { cout<<"Lambda"<<endl; };
     call(fn);
     return 0;
   }
   
   template <typename T>
   void call(T fn) {
     fn();
   }
   

2. You can return lambdas from functions like any another object which makes for some interesting possibilities such as the following:

   #include <iostream>
   #include <functional>
   
   using namespace std;
   
   template<typename T>
   function<T()> makeAccumulator(T& val, T by) {
       return [=,&val]() {
           return (val += by);
       };
   }
   
   int main() {
       int val = 10;
       auto add5 = makeAccumulator(val, 5);
       cout<<add5()<<endl;
       cout<<add5()<<endl;
       cout<<add5()<<endl;
       cout<<endl;
   
       val = 100;
       auto add10 = makeAccumulator(val, 10);
       cout<<add10()<<endl;
       cout<<add10()<<endl;
       cout<<add10()<<endl;
   
       return 0;
   }
   

   Which produces the following output:

   15
   20
   25
   
   110
   120
   130
   

   The key thing to remember here is that it is your responsibility to make sure that the values you capture in a lambda remain in memory for the lifetime of the lambda itself. The compiler will not for instance, prevent you from capturing local variables by reference in a lambda and if you continue to access a variable that is no longer available, well, then the behavior is undefined.

3. Simply defining a lambda causes all variables captured by value by the lambda to be copy constructed. You don't really have to have any code that invokes the lambda in order for the variables to be copied. This is consistent with the idea that creating a lambda function essentially creates a function object which has as instance members the variables that have been captured in the lambda. Here's an example:

   #include <iostream>
   
   using namespace std;
   
   class Foo {
   public:
     Foo() {
       cout<<"Foo::Foo()"<<endl;
     }
   
     Foo(const Foo& f) {
       cout<<"Foo::Foo(const Foo&)"<<endl;
     }
   
     ~Foo() {
       cout<<"Foo~Foo()"<<endl;
     }
   };
   
   int main() {
     Foo f;
     auto fn = [f]() { cout<<"lambda"<<endl; };
     cout<<"Quitting."<<endl;
     return 0;
   }
   

   Here's the output this produces:

   Foo::Foo()
   Foo::Foo(const Foo&)
   Quitting.
   Foo~Foo()
   Foo~Foo()
   

   As you can tell, the copy constructor gets invoked even though the lambda itself never gets invoked.

4. As an extension of the previous point, if you capture objects by value in a lambda and then proceed to pass that lambda around to other functions by value, then the variables in the closure will also get copy constructed.

5. If you reference capture a const local variable, it becomes a const reference in the lambda.

   #include <iostream>
   
   using namespace std;
   
   int main() {
       const int val = 10;
       auto f1 = [&val]() {
           val = 20;  // won't compile
           cout<<"val = "<<val<<endl;
       };
       f1();
   
       return 0;
   }
   

6. In general, when you wish to declare a variable that can hold a reference to a lambda in contexts where auto is not permissible (for e.g. function return types or arguments) use std::function. An example:

   #include <iostream>
   #include <functional>
   
   using namespace std;
   
   function<bool(int)> makeLEPredicate(int max) {
       return [max](int val) -> bool {
           return val <= max;
       };
   }
   
   int main() {
       auto le10 = makeLEPredicate(10);
       cout<<le10(8)<<endl;
   
       return 0;
   }
   

   You could alternatively use function templates to achieve the same thing if the semantics of using templates makes sense to your use case.

That's all for now. This list might get expanded as I explore lambdas further. As you might have noticed, the ability to use lambdas really does make a significant difference to productivity without sacrificing performance.

You'll note that the tiles have been grouped into separate sections and each section contains tiles of different sizes. In this case there are only 2 sizes - a wide tile:

And a square tile:

Here's an example of an app that uses different tile sizes in different groups:

I'd been meaning to write down exactly how we can customize the WinJS ListView to create interfaces such as this one and, well, here it is. The basic technique for implementing variable tiles with the WinJS ListView involves the following things:

1. Determine what your "cell unit" is going to be. This is the width and height of a single "unit" in pixels - the idea is that tile sizes must be a multiple of this. For example, I might decide that my cell unit is going to be 15x20 pixels. Then valid tile sizes would be 15x40, 30x20, 45x300 etc. Once you know what this is, implement the groupInfo property on the GridLayout object on your list view's layout like so.

   ready: function (element, options) {
       // more stuff here
       var layout = new WinJS.UI.GridLayout();
       layout.groupInfo = this.getGroupInfo.bind(this);
       // more stuff here
   },
   
   
   getGroupInfo: function () {
       return {
           enableCellSpanning: true,
           cellWidth: 15,
           cellHeight: 20
       }
   },
   

2. Since different tiles in your control can be of different sizes you'll need to tell the ListView what those sizes are going to be. You do this by implementing a method called itemInfo on your GridLayout object. The ListViewcalls itemInfo for every element it renders from the data source. The important thing to remember is that the size you return from the itemInfo method must be a multiple of the size you returned from groupInfo.

   ready: function (element, options) {
       // more stuff here
       var layout = new WinJS.UI.GridLayout();
       layout.itemInfo = this.getItemInfo.bind(this);
       // more stuff here
   }
   
   
   getItemInfo: function (index) {
       var data = ImageData.imagesList.getAt(index);
       var size = {
           width: 150,
           height: 200,
           newColumn: false
       };
       if (data.group.name === 
              ImageData.imageGroups.kittens.name) {
           size.height = 100;
       }
       else if (data.group.name ===
              ImageData.imageGroups.portraits.name) {
           size.width = 120;
       }
       return size;
   },
   

3. Associate a JS function for your list view's itemTemplate property. The job of this function is to render an item.

   listView.itemTemplate = this.selectItemTemplate.bind(this);
   

   It is passed a WinJS.Promise object as a parameter which when resolved will yield the data item which is to be rendered. We can either manually create DOM elements using document.createElement from this routine or, as is more convenient, use declaratively pre-created WinJS.Binding.Template instances from the HTML mark-up. Here's an example implementation showing how to do this:

   selectItemTemplate: function(itemPromise, recycle) {
       return itemPromise.then(function (item) {
           var data = item.data;
           var template;
           if (data.group.name ===
                   ImageData.imageGroups.kittens.name) {
               template = document.querySelector("#wide-template").
                   winControl;
           }
           else if (data.group.name ===
                   ImageData.imageGroups.portraits.name) {
               template = document.querySelector("#long-template").
                   winControl;
           }
           else {
               template = document.
                   querySelector("#default-template").winControl;
           }
           return template.render(item.data);
       });
   },
   

As you can tell we first wait for the promise to resolve and then take the data and do some custom template selection logic to pick a template from the DOM and then call its render method passing in the data object as binding context. You will need to ensure that the styling you use on your template mark-up matches up with the size you return from itemInfo as otherwise you might end up with blank spaces in your tiles where the styling doesn't get applied (now, you might want to do this deliberately of course, in which case its totally fine). That's pretty much it!

1. Go to the modern desktop and type "Debuggable Package Manager" and launch it.

   

   This opens up a powershell window.

2. Run Get-AppxPackage to list the packages installed and use Where-Object to filter for what you're looking for. Since were interested in the mail app we run this:

   Get-AppxPackage | Where-Object PackageFullName -like "*commu*"
   

3. Note the value of the "PackageFullName" property and enable debugging by running this:

   Enable-AppxDebug microsoft.windowscommunicationsapps_17.0.1114.318_x64__8wekyb3d8bbwe
   

4. Now launch the app. Then launch Visual Studio, hit Ctrl+Alt+P and select the instance of WWAHost.exe which looks like the app you're interested in.

   

5. Debug away!

   

This was a problem as I was in no mood to download each file individually and there were 100s of "download" buttons there. I figured I'd see if I can screen scrape the links from the DOM of this page and then write a little script to download all of them in one go. So I fired up an instance of IE and hit F12 to launch the developer tools and used the "Select element by click" button to quickly navigate to the markup associated with a "download" button.

As you can tell, all the download buttons are basically anchor tags and the href attribute points to the MSU file for that particular language. Also, you'll note that each such anchor tag has a class called "download" applied on it. So I should be able to fetch all the links by simply iterating through all anchor tags which have the "download" class applied on them. I switched to the "Console" tab in the developer tools window and ran the following script:

document.querySelectorAll("a.download")

And sure enough this produced a list of all the anchor tags I was interested in. I needed the URL however and not the DOM elements themselves. So I ran this next:

Array.prototype.forEach.call(
    document.querySelectorAll("a.download"),
    function (a) {
        console.log(a.href);
    });

This produced a list of links such as this (snipped since there are quite a lot of them):

http://download.microsoft.com/download/D/9/A/.../IE10-Windows6.1-LanguagePack-x64-zh-tw.msu 
http://download.microsoft.com/download/D/9/A/.../IE10-Windows6.1-LanguagePack-x64-zu-za.msu 
http://download.microsoft.com/download/D/9/A/.../IE10-Windows6.1-LanguagePack-x86-af-za.msu 
http://download.microsoft.com/download/D/9/A/.../IE10-Windows6.1-LanguagePack-x86-am-et.msu

If you're wondering why I had to iterate through each element in the list of nodes returned by querySelectorAll via Array.prototype.forEach.call then that's because what querySelectorAll returns isn't a JavaScript array object, i.e., it doesn't inherit from Array.prototype. It is instead a NodeList object which looks a lot like an array! It has numeric properties starting from 0 to N-1 where N is the number of elements returned and it has a length property as well which is equal to N. It turns out that all the Array methods are perfectly capable of dealing with such "array like" objects just as well as genuine, certified JavaScript arrays. Here's an example of what I am talking about:

var notArray = {
    0: "This ",
    1: "is ",
    2: "not ",
    3: "really ",
    4: "an ",
    5: "array.",
    length: 6
};

console.log(Array.prototype.reduce.call(
    notArray,
    function (previous, current) {
        return previous + current;
    },
    ""));

This snippet prints the following text to the console:

This is not really an array.

If you take another look at the list of URLs our script printed to the console, you'll notice from the file names that this list includes both x86 files and x64 files. I wanted only x64 files. So, I next changed the script to this:

Array.prototype.forEach.call(
    document.querySelectorAll("a.download[href*=x64]"),
    function (a) {
        console.log(a.href);
    });

The selector syntax above looks for all anchor tags in the DOM which has a class called "download" applied where the href attribute's value contains the string "x64". I had first implemented this via another call to Array.prototype.filter before learning that CSS3 selector syntax already provides for it! Pretty nifty no? That's pretty much it. I wanted to run a download script for fetching all the files so I slightly modified the script to produce wget calls like so:

Array.prototype.forEach.call(
    document.querySelectorAll("a.download[href*=x64]"),
    function (a) {
        console.log("wget " + a.href);
    });

And plonked the output into a batch file and ran it. Mission accomplished!

Now, it turned out that this particular page in question includes the jQuery library as well as can be seen when you pull up the files list from the "Script" tab in the developer console.

I could have done the same thing I did above using a slightly terser syntax using jQuery as well. Here's how:

$("a.download[href*=x64]").each(function () {
    console.log("wget " + this.href);
});

Not having to resort to the Array.prototype weirdness does make the code a lot cleaner doesn't it?

Drag/Drop

One of the things that InstaFuzz supports is the ability to drag and drop image files directly on to the big blackish/blue box. Support for this is enabled by handling the "drop" event on the CANVAS element. When a file is dropped onto an HTML element the browser fires the "drop" event on that element and passes in a dataTransfer object which contains a files property that contains a reference to the list of files that were dropped. Here's how this is handled in the app ("picture" is the ID of the CANVAS element on the page):

var pic = $("#picture");
pic.bind("drop", function (e) {
    suppressEvent(e);
    var files = e.originalEvent.dataTransfer.files;
    // more code here to open the file
});
pic.bind("dragover", suppressEvent).bind("dragenter", suppressEvent);
function suppressEvent(e) {
    e.stopPropagation();
    e.preventDefault();
}

The files property is a collection of File objects that can then subsequently be used with the File API to access the file contents (covered in the next section). We also handle the dragover and dragenter events and basically prevent those events from propagating to the browser thereby preventing the browser from handling the file drop. IE for instance might unload the current page and attempt to open the file directly otherwise.

File API

Once the file has been dropped, the app attempts to open the image and render it in the canvas. It does this by using the File API. The File API is a W3C specification that allows web apps to programmatically access files from the local file system in a secure fashion. In InstaFuzz we use the FileReader object to read the file contents as a data URL string like so using the readAsDataURL method:

var reader = new FileReader();
reader.onloadend = function (e2) {
    drawImageToCanvas(e2.target.result);
};
reader.readAsDataURL(files[0]);

Here, files is the collection of File objects retrieved from the function handling the "drop" event on the CANVAS element. Since we are interested only in a single file we simply pick the first file from the collection and ignore the rest if there are any. The actual file contents are loaded asynchronously and once the load completes, the onloadend event is fired where we get the file contents as a data URL which we then subsequently draw on to the canvas.

Rendering the filters

Now the core functionality here is of course the application of the filters. In order to be able to apply the filter to the image we need a way to access the individual pixels from the image. And before we can access the pixels we need to have actually rendered the image on to our canvas. So let's first take a look at the code that renders the image that the user picked on to the canvas element.

Rendering images on to the canvas

The canvas element supports the rendering of Image objects via the drawImage method. To load up the image file in an Image instance, InstaFuzz uses the following utility routine:

App.Namespace.define("InstaFuzz.Utils", {
    loadImage: function (url, complete) {
        var img = new Image();
        img.src = url;
        img.onload = function () {
            complete(img);
        };
    }
});

This allows the app to load up image objects from a URL using code such as the following:

function drawImageToCanvas(url) {
    InstaFuzz.Utils.loadImage(url, function (img) {
        // save reference to source image
        sourceImage = img;

        mainRenderer.clearCanvas();
        mainRenderer.renderImage(img);

        // load image filter previews
        loadPreviews(img);
    });
}

Here, mainRenderer is an instance created from the FilterRenderer constructor function defined in filter-renderer.js. The app uses FilterRenderer objects to manage canvas elements - both in the preview pane as well as the main canvas element on the right. The renderImage method on the FilterRenderer has been defined like so:

FilterRenderer.prototype.renderImage = function (img) {
    var imageWidth = img.width;
    var imageHeight = img.height;
    var canvasWidth = this.size.width;
    var canvasHeight = this.size.height;
    var width, height;

    if ((imageWidth / imageHeight) >= (canvasWidth / canvasHeight)) {
        width = canvasWidth;
        height = (imageHeight * canvasWidth / imageWidth);
    } else {
        width = (imageWidth * canvasHeight / imageHeight);
        height = canvasHeight;
    }

    var x = (canvasWidth - width) / 2;
    var y = (canvasHeight - height) / 2;
    this.context.drawImage(img, x, y, width, height);
};

That might seem like a lot of code but all it does ultimately is to figure out the best way to render the image in the available screen area considering the aspect ratio of the image. The key piece of code that actually renders the image on the canvas occurs on the last line of the method. The context member refers to the 2D context acquired from the canvas object by calling its getContext method.

Fetching pixels from the canvas

Now that the image has been rendered we will need access to the individual pixels in order to apply all the different filters that are available. This is easily acquired by calling getImageData on the canvas's context object. Here's how InstaFuzz calls this from instafuzz.js.

var imageData = renderer.context.getImageData(
    0, 0,
    renderer.size.width,
    renderer.size.height);

The object returned by getImageData provides access to the individual pixels via its data property which in turn is an array like object that contains a collection of byte values where each value represents the color rendered for a single channel of a single pixel. Each pixel is represented using 4 bytes that specify values for the red, green, blue and alpha channels. It also has a length property that returns the length of the buffer. If you have a 2D co-ordinate you can easily transform that into an index into this array using code such as the following. The color intensity values of each channel ranges from 0 through 255. Here's the utility function from filters.js that accepts as input an image data object along with 2D coordinates for the pixel the caller is interested in and returns an object containing the color values:

function getPixel(imageData, x, y) {
    var data = imageData.data, index = 0;

    // normalize x and y and compute index
    x = (x < 0) ? (imageData.width + x) : x;
    y = (y < 0) ? (imageData.height + y) : y;
    index = (x + y * imageData.width) * 4;

    return {
        r: data[index],
        g: data[index + 1],
        b: data[index + 2]
    };
}

Applying the filters

Now that we have access to the individual pixels, applying the filter is fairly straightforward. Here, for instance is the function that applies a weighted grayscale filter on the image. It simply picks intensities from the red, green and blue channels and sums them up after applying a multiplication factor on each channel and then assigns the result for all 3 channels.

// "Weighted Grayscale" filter
Filters.addFilter({
    name: "Weighted Grayscale",
    apply: function (imageData) {
        var w = imageData.width, h = imageData.height;
        var data = imageData.data;
        var index;
        for (var y = 0; y < h; ++y) {
            for (var x = 0; x < w; ++x) {
                index = (x + y * imageData.width) * 4;
                var luminance = parseInt((data[index + 0] * 0.3) +
                                         (data[index + 1] * 0.59) +
                                         (data[index + 2] * 0.11));
                        data[index + 0] = data[index + 1] =
                    data[index + 2] = luminance;
            }

            Filters.notifyProgress(imageData, x, y, this);
        }

        Filters.notifyProgress(imageData, w, h, this);
    }
});

Once the filter has been applied we can have that reflected on the canvas by calling the putImageData method passing in the modified image data object. While the weighted grayscale filter is fairly simple most of the other filters use an image processing technique known as convolution. The code for all the filters is available in filters.js and the convolution filters were ported from the C code available here.

Web Workers

As you might imagine doing all this number crunching to apply the filters can potentially take a long time to complete. The motion blur filter for instance uses a 9x9 filter matrix for computing the new value for every single pixel and is in fact the most CPU intensive filter among them all. If we were to do all this computation on the UI thread of the browser then the app would essentially freeze every time a filter was being applied. To provide a responsive user experience the app delegates the core image processing tasks to a background script using the support for W3C Web Workers.aspx) in modern browsers.

Web workers allow web applications to have scripts run in a background task that executes in parallel along with the UI thread. Communication between the worker and the UI thread is accomplished by passing messages using the postMessage API. On both ends (i.e. the UI thread and the worker) this manifests as an event notification that you can handle. You can only pass "data" between workers and the UI thread, i.e., you cannot pass anything that has to do with the user interface - you cannot for instance, pass DOM elements to the worker from the UI thread.

In InstaFuzz the worker is implemented in the file filter-worker.js. All it does in the worker is handle the onmessage event and apply a filter and then pass the results back via postMessage. As it turns out, even though we cannot pass DOM elements (which means we cannot just hand a CANVAS element to the worker to have the filter applied) we can in fact pass the image data object as returned by the getImageData method that we discussed earlier. Here's the filter processing code from filter-worker.js:

importScripts("ns.js", "filters.js");

var tag = null;
onmessage = function (e) {
    var opt = e.data;
    var imageData = opt.imageData;
    var filter;

    tag = opt.tag;
    filter = InstaFuzz.Filters.getFilter(opt.filterKey);

    var start = Date.now();
    filter.apply(imageData);
    var end = Date.now();

    postMessage({
        type: "image",
        imageData: imageData,
        filterId: filter.id,
        tag: tag,
        timeTaken: end - start
    });
}

The first line pulls in some script files that the worker depends on by calling importScripts. This is similar to including a JavaScript file in a HTML document using the SCRIPT tag. Then we set up a handler for the onmessage event in response to which we simply apply the filter in question and pass the result back to the UI thread by calling postMessage. Simple enough!

The code that initializes the worker is in instafuzz.js and looks like this:

var worker = new Worker("js/filter-worker.js");

Not much is it? When a message is sent by the worker to the UI thread we handle it by specifying a handler for the onmessage event on the worker object. Here's how this is done in InstaFuzz:

worker.onmessage = function (e) {
    var isPreview = e.data.tag;
    switch (e.data.type) {
        case "image":
            if (isPreview) {
                previewRenderers[e.data.filterId].
                    context.putImageData(
                        e.data.imageData, 0, 0);
            } else {
                mainRenderer.context.putImageData(
                    e.data.imageData, 0, 0);
            }

            break;
        // more code here
    }
};

The code should be fairly self-explanatory. It simply picks the image data object sent by the worker and applies it to the relevant canvas's context object causing the modified image to be rendered on screen. Scheduling a filter for conversion with the worker is equally simple. Here's the routine that performs this function in InstaFuzz:

function scheduleFilter(filterId,
                        renderer,
                        img, isPreview,
                        resetRender) {
    if (resetRender) {
        renderer.clearCanvas();
        renderer.renderImage(img);
    }

    var imageData = renderer.context.getImageData(
        0, 0,
        renderer.size.width,
        renderer.size.height);

    worker.postMessage({
        imageData: imageData,
        width: imageData.width,
        height: imageData.height,
        filterKey: filterId,
        tag: isPreview
    });
}

In conclusion

We saw that fairly intricate user experiences are possible today with HTML5 technologies such as Canvas, Drag/Drop, File API and Web Workers. Support for all of these technologies is quite good in pretty much all modern browsers. One thing that we did not address here is the question of making the app compatible with older browsers. That, truth be told, is a non-trivial but necessary task that I will hopefully be able to talk about in a future article.

When I started out on this app I was only really just interested in seeing if the web platform had really evolved to a point where an app like the hugely popular Instagram app could be built using just HTML, JavaScript and CSS. As it turns out we can in fact do exactly that. This article walks you through the technologies that make this possible and shows how it is entirely feasible today to build interoperable web applications that provide a great user experience no matter what brand of browser the user is running.

If you happen to be one of the two or so people who have not heard about Instagram then you might be pleased to hear that it is a hugely popular photo sharing and social networking service that allows you to take pictures, apply interesting digital filters on them and share them with the world to see. The service got so popular that it was acquired by Facebook for a bag full of cash and stock in April of 2012.

InstaFuzz is the name of the app I put together and while I don't expect to be acquired by Facebook or anybody else for a billion green it does however make the case that an app such as this one can be built using only standards compliant web technologies such as Canvas, File API, Drag/Drop, Web Workers, ES5 and CSS3 and still manage to run well on modern browsers such as Internet Explorer 10, Google Chrome and Firefox.

About the app

If you'd like to take a look at the app, then here's where it is hosted at:

http://blogorama.nerdworks.in/arbit/InstaFuzz/

You can download the source and run locally from here. While this is a Visual Studio 2012 project there really isn't any server code or anything like that. You can use your favorite editor to look at the source and run it from the file system if you are so inclined.

As soon as you load it up, you're presented with a screen that looks like this:

The idea is that you can load up a photograph into the app either by clicking on the big red "Add" button on the bottom left hand corner or drag and drop an image file into the blackish/blue area on the right. Once you do that you get something that looks like this:

You'll note that a list of digital filters are listed on the left of the screen showing a preview of what the image would look like if you were to apply the said filter. Applying a filter is a simple matter of clicking on one of the filter previews on the left. Here's what it looks like after applying the "Weighted Grayscale" filter followed by a "Motion Blur". As you can tell filters are additive - as you keep clicking on filters, they are applied on top of what was applied earlier:

Let's next take a look at how the UI layout has been put together.

UI Layout

The HTML markup is actually so little that I can actually reproduce the contents of the BODY tag in its entirety here (excluding the SCRIPT includes):

<header>
    <div id="title">InstaFuzz</div>
</header>
<section id="container">
    <canvas id="picture" width="650" height="565"></canvas>
    <div id="controls">
        <div id="filters-list"></div>
        <button id="loadImage">Add</button>
        <input type="file" id="fileUpload"
               style="display: none;"
               accept="image/gif, image/jpeg, image/png" />
    </div>
</section>

<!-- Handlebar template for a filter UI button -->
<script id="filter-template" type="text/x-handlebars-template">
    <div class="filter-container" data-filter-id="{{filterId}}">
        <div class="filter-name">{{filterName}}</div>
        <canvas class="filter-preview" width="128" height="128"></canvas>
    </div>
</script>

There's nothing much going on here. Pretty much everything should be standard fare. I will however draw attention to the fact that I am using the Handlebars JavaScript templating system here for rendering the markup for the list of filters on the left of the screen. The template markup is declared in the HTML file (the SCRIPT tag in the snippet shown above) and then used from JavaScript. The template markup is then bound to a JavaScript object that supplies the values for handlebars expressions such as {{filterId}} and {{filterName}}. Here's the relevant piece of JS from the app with a bit of DOM manipulation help from jQuery:

var templHtml = $("#filter-template").html(),
    template = Handlebars.compile(templHtml),
    filtersList = $("#filters-list");
var context = {
    filterName: filter.name,
    filterId: index
};

filtersList.append(template(context));

As you can tell from the HTML markup all the filter preview boxes feature a CANVAS tag as does the big box on the right where the final output is rendered. We'll go into a bit more detail later on in the article as to how canvas technology is used to achieve these effects.

The app also uses CSS3 @font-face fonts to render the text in the header and the "Add" button. The fonts have been taken from the excellent Font Squirrel site and here's what the declaration looks like:

@font-face {
    font-family: 'TizaRegular';
    src: url('fonts/tiza/tiza-webfont.eot');
    src: url('fonts/tiza/tiza-webfont.eot?#iefix')
           format('embedded-opentype'),
         url('fonts/tiza/tiza-webfont.woff') format('woff'),
         url('fonts/tiza/tiza-webfont.ttf') format('truetype'),
         url('fonts/tiza/tiza-webfont.svg#TizaRegular') format('svg');
    font-weight: normal;
    font-style: normal;
}

This directive causes the user agent to embed the font in the page and make it available under the name assigned to the font-family rule which in this case is "TizaRegular". After this we can assign this font to any CSS font-family rule like how we normally do. In InstaFuzz I use the following rule to assign the font to the header element:

font-family: TizaRegular, Cambria, Cochin, Georgia, Times,
   "Times New Roman", serif;

You might also have noticed that there is a subtle shadow being dropped on the page by the container element.

This is made possible using the CSS3 box-shadow rule and here's how it's used in InstaFuzz.

-moz-box-shadow: 1px 0px 4px #000000, -1px -1px 4px #000000;
-webkit-box-shadow: 1px 0px 4px #000000, -1px -1px 4px #000000;
box-shadow: 1px 0px 4px #000000, -1px -1px 4px #000000;

This causes the browser to render a shadow around the relevant element. Each comma separated section in the value specifies the following attributes of the shadow:

1. Horizontal offset

2. Vertical offset

3. Spread distance - positive values have the effect of softening the shadow

4. Shadow color

One can specify multiple shadow values separated by comma as in fact has been done above. Note that I've also specified the shadow using vendor prefix syntax for Firefox and Chrome/Safari using the moz and webkit prefixes. This causes the shadow to continue to work in versions of those browsers where support for this capability was provided using the vendor prefixed version of the rule. Note that the W3C version of the rule - box-shadow - is specified last. This is done deliberately to ensure that in case the browser supports both the forms then only the W3C behavior is actually applied to the page.

One often finds that web developers either fail to include vendor prefixed version of a given CSS3 rule for all the browsers that support that rule and/or fail to include the W3C version as well. Often developers just put the webkit version of the rule ignoring other browsers and the W3C standard version. This causes two problems - [1] poor user experience for users who are using non-webkit browsers and [2] it ends up resulting in webkit becoming a de-facto standard for the web. Ideally we want W3C to be driving the future of the web and not one specific browser implementation. So here are some things to remember when playing with experimental implementations of CSS features:

1. Use vendor prefixed versions of CSS rules by all means but remember to specify the rule for all supported browsers and not just the one that you happen to be testing the page in (if you're using Visual Studio to edit your CSS then you might be interested in the supremely excellent extension for Visual Studio called Web Essentials that makes the job of managing vendor prefixes about as simple as it can possibly get).

2. Remember to specify the W3C version of the rule as well.

3. Remember to order the occurrence of the rules so that the W3C version shows up last. This is to allow clients that support both the vendor prefixed version and the W3C version to use the W3C specified semantics for the rule.

That's all for now. In the next and final post in this series we'll take a look at how the app supports drag/drop of files, the use of File API, how the filters themselves work and how we prevent the UI thread from freezing by delegating the core number crunching work to web workers.