Joel Abrahamsson

Why is the async keyword needed in JavaScript?

Mon, 11 Sep 2023 21:02:57 +0100

Last week a colleague asked me what the purpose of the async keyword was in JavaScript. Not because he didn't know how to use async/await. He was wondering why the async keyword was needed to use await. This set me off on a quest to find the answer. To avoid beating around the bush: its purpose is backwards compatibility.

Any experienced JavaScript developer is probably well versed in the usage of async/await these days. The await keyword unwraps a Promise. Until the Promise is either fulfilled or rejected the execution of the code after it is put on hold and the event loop is released to do other work. But why is the async keyword needed in order for us to use await in a function?

A common misconception seems to be that it "is used to communicate that the function is async" to anyone who uses it. However, that's not true. Consider the following code:

export async function fetchTextLength() {
 const resp = await fetch("https://www.wikipedia.org");
 const text = await resp.text();
 return text.length;
}

The function above is marked async, but as a consumer of it that's hardly relevant. What is relevant is that it returns a Promise. Indeed, if we hover over it in a file that uses it in VS Code that's what we see:

Further, I think we can all agree that to a consumer of the above module it shouldn't matter if we decide to change the module to instead have the following implementation:

export function fetchTextLength() {
    return inner();
}

async function inner() {
    const resp = await fetch("https://www.wikipedia.org");
    const text = await resp.text();return text.length;
}

The important thing is that our function returns a Promise, not whether it's marked as async or not. So, if the purpose of the async keyword isn't to communicate anything to other developers then what's its purpose? One could think that it's being used by the JavaScript interpreter (or JIT compiler) to identify that the function is using await and to do its thing, implicitly having the function return a Promise. While that's true the interpreter could easily do that simply by looking at the code and identifying usage of the await keyword.

The real motivation for the async keyword can be found by digging through the original proposal for async/await in ECMAScript. There in a comment by Brian Terlson in issue 88 we can read the following:

"Long ago we considered whether it was possible to await everywhere but we could not find a good way to do it without breaking existing usage of await as an identifier"

In other words, with the introduction of await in ECMAScript a new reserved keyword needed to be added to the language. However, you can't just go around adding reserved keywords to programming languages if you want to be compatible with existing code. This is especially true for interpreted ones run in browsers. So the async keyword was introduced as a way to ensure backwards compatibility.

To illustrate, the below code is perfectly valid and outputs "test":

function test() {
    const await = "test";
    console.log(await);
}

test();

However, if we add the async keyword, which would not have been valid in older versions of JS, to the function like below we'll get an error when we try to run it:

async function test() {
    const await = "test";
    console.log(await);
}

test();

The above code produces:

const await = "test";
        ^^^^^

SyntaxError: Unexpected reserved word

Exception order when awaiting multiple async tasks in C#

Mon, 09 Sep 2019 22:30:06 +0100

A C# has-been returns to C# and experiments with this new hip thing called async/await and how that relates to execution order and exceptions.

I recently returned to .NET and C# development after a six year hiatus in Node.JS land. A lot has changed since I last wrote C#, although Jon Skeet still writes the best books.

One feature that I never got to use during my previous C# development days were async/await. However, I'm very familiar with the concept having written a lot of JavaScript with promises and async/await. However, one thing that I didn't intuitively knew in C# was in what order Exceptions are thrown, or rather caught, when awaiting multiple Tasks. Therefor I created this little experiment:

using System;
using System.Threading;
using System.Threading.Tasks;

namespace AsyncExceptions
{
  class Program
  {
    async static Task Main(string[] args)
    {
      try
      {
        var one = Do("One", 500);
        var two = Do("Two", 500);

        await one;
        await two;
       } catch(Exception ex)
       {
          Console.WriteLine(ex.Message);
        }
           
      }
      
      public static Task Do(string name, int time)
      {
        return Task.Run(() =>
        {
          Console.WriteLine($"Task {name} starting");
          Thread.Sleep(time);
          Console.WriteLine($"Task {name} pre exception");
          throw new Exception($"Exception {name}");
        });
      }
  }
}

Note: This is a contrived example that intrigues me and which we are exploring in this article. In practice you shouldn’t write code like the one above, with two separate awaits but instead use Task.WhenAll().

The above program will output five lines in total. Two lines will be printed for when each task starts to run. Two additional lines will be outputted when each task has waited for 500 milliseconds and is about to throw. Finally there will be a line telling us which exception was caught. Can you guess what the output will be?

As it turned out in my experiments the first four lines are quite random. Sometimes the first tasks starts and completes first, sometimes it's the other way around. Here are three sample outputs (the final line omitted for now):

Sample 1:

Task One starting
Task Two starting
Task Two pre exception
Task One pre exception

Sample 2:

Task One starting
Task Two starting
Task One pre exception

Sample 3:

Task Two starting
Task One starting
Task Two pre exception
Task One pre exception

What about the final line? That's always reads `Exception One`. Meaning that although the second exception may sometimes be thrown first in the async context back in our main thread we'll always catch the exception from the task that we await first. That holds true even if we make the second task throw much sooner by modifying the Main method to look like this:

var one = Do("One", 500);
var two = Do("Two", 1);

Of course in hindsight of this experiment the result makes sense. While the second async operation may complete and be ready to return to the initiating thread first in that thread we are waiting for the first task to complete before caring about the result of the second one.

JavaScript

So what about the equivalent code i JavaScript? It could look something like this:

"use strict";

async function main() {
    try {
        const one = Do("One", 500);
        const two = Do("Two", 500);

        await one;
        await two; 
    } catch(ex) {
        console.log(ex.message)
    }
}

main();

function Do(name, time) {
    return new Promise((resolve, reject) => {
        console.log(`Task ${name} starting`)
        setTimeout(() => {
            console.log(`Task ${name} pre exception`);
            reject(new Error(`Exception ${name}`))
        }, time);
    })
    
}

Note: Again this is a contrived example. In practice you should use Promise.all() in code like this.

Here the output is more consistent. In all my attempts the output read like this:

Task One starting
Task Two starting
Task One pre exception
Task Two pre exception
Exception One
(node:6564) UnhandledPromiseRejectionWarning: Error: Exception Two

The first task is put queue for execution on the event loop first and therefor is executed and throws first. We catch the first exception and then get an angry error message due to us not catching the second exception. Clearly Promise.all would be a good solution for that.

But what if we change the second task to complete much faster than the first one?

const one = Do("One", 500);
const two = Do("Two", 1);

Then the output is this:

Task One starting
Task Two starting
Task Two pre exception
(node:6577) UnhandledPromiseRejectionWarning: Error: Exception Two
Task One pre exception
Exception One

The second task throws much earlier and that results in a an UnhandledPromiseRejectionWarning. Then the execution continues and the exception thrown by the first, and first awaited, task is caught. While I find this fascinating I think I'll leave the "why!?" to a different blog post.

Flatten array of arrays with JavaScript

Thu, 30 Aug 2018 18:46:11 +0100

const arrays = [[1], ["2"], [3]];

const merged = [].concat(...arrays);

console.log(merged); // [ 1, '2', 3 ]

Pre ES6:

var merged = [].concat.apply([], arrays);

Quickly mapping an array of URLs to responses with JavaScript async/await and Promise.all

Sun, 26 Aug 2018 19:09:00 +0100

While perhaps not the most readable, a compact version (using window.fetch) can look like this:

const urls = [
  "https://jsonplaceholder.typicode.com/comments/1",
  "https://jsonplaceholder.typicode.com/comments/2",
  "https://jsonplaceholder.typicode.com/comments/3"
];

async function fetchAll() {
  const results = await Promise.all(urls.map((url) => fetch(url).then((r) => r.json())));
  console.log(JSON.stringify(results, null, 2));
}

fetchAll();

Quickly creating and mapping an array in JavaScript

Thu, 23 Aug 2018 20:50:25 +0100

When, for instance, creating test data one might do something like this:

const data = [];

for(let i = 0; i < 100; i++) {
 data.push({ num: i }); 
}

This may be written slightly shorter and more wrist friendly using Array.fill:

const data = Array(100).fill(null).map((val, i) => {
 return { num: i };
});

Responsibly Responsive Web Design at Expressen

Thu, 05 Jul 2018 17:19:42 +0100

When building a new version of the news site Expressen.se the team hesitated to use responsive web design. Here's why and what we ended up doing.

Expressen.se is the second largest news site in the Nordic region with millions of page views per day. During the fall of 2017 we set out to rebuild the site from scratch with a focus on building as fast a site as possible. You can read more about how and why we did this in Simon Hjälmefjord's blog post (in Swedish).

Another objective that we had when building the new version of the site was to end up with a single code base for the site. Prior to the rebuild we had different versions of the site for different channel and these were distributed over multiple applications. More specifically we had:

A fixed width version for computers.
A partially responsive version for tablets.
A fluid version for smart phones.

The first two of the above versions were served by one application, Ariel. Ariel naturally had functionality to handle the fact that it was serving two versions, including channel specific templates. The third version above, the mobile site, was served by a different application.

This division of labor was not without benefits. We could optimize for each channel's unique pre-requisites as well as perform experiments on subsets of the total traffic and code base. However, the fact that we had three different versions scattered across two code bases also brought some obvious drawbacks. The two code bases diverged heavily from day one and most developers tended to only know one of them well. It was also difficult to maintain a coherent user experience and design across the different channels.

Another problem was that we quite often implemented new functionality for only one of the channels. That wasn't necessarily always a bad thing as the functionality may only make sense for one type of device. In other cases it would make sense for all channels but we decided to test it on a single channel first. However, sometimes we really meant to, and/or wanted to, have the new functionality in all channels but after having built it for a single channel and having spent some time measuring its impact it we had moved on to other features and it was never implemented for the remaining channels.

So, when we were about to rebuild the site the entire team was very much in agreement that we wanted a single code base for the site. However, while we had no doubts about wanting a single code base for all channels, we were a lot less convinced about how to accomplish that. Should we build an application that could handle three different channels, with separate views etc., or should we build a responsive site?

It's the 21st century; of course we should build a responsive site! Shouldn't we?

These days it seems like a non-question whether to use responsive web design or not. Responsive web design has become the de facto standard and all of our competitors had already built responsive sites. However, we wanted to build the best and fastest site possible and asked ourselves the question: is responsive web design really the best for us?

Responsive web design, implemented using media queries etc., brings a number of significant advantages:

Coherent design across all device types and screen sizes is the default and lapses from that require conscious decisions.
A single code base for all channels is obvious and more or less a requirement.
Compared to designing, developing and maintaining separate versions for different channels building a responsive site is cost efficient.

While the above advantages are significant it's interesting to note that none of them have any direct positive impact on the end users experience. For the end user there are few benefits associated with visiting a responsive site.

On the other hand there are (often) drawbacks in the form of additional HTML markup, CSS and JavaScript that the visitor's browser has to download, parse and interpret. Sometimes there are entire blocks of HTML and CSS which are never even shown to the user. There may also be custom JavaScript code and modules that are only relevant in a specific channel or which are there specifically to manage the responsive nature of the site.

In other words: to build a responsive site means taking the risk of serving a slower site to the end user compared to a site which is optimized for the user's device type. It's quite obvious really; when we build a responsive site we force the user's browser to deal with the problem of adapting the content depending on device and viewport witdh as well as using different layouts in different contexts.

There are of course various creative ways of minimizing the performance impact but the end result tends to be less than perfect. At the same time some of the cost efficiency benefits of building a responsive site tends to get lost when we spend time on optimizing the performance of the site.

Having the cake while eating it too

So, what should we do? We wanted the development related benefits of building a responsive site without the performance penalties for our visitors. Therefore we decided to try to have the cake while eating it too. We decided to try an approach that we came to call "responsibly responsive".

When I as a developer run the site on my local machine or when a UX designer views the site on a test environment what we see is a responsive site. When we design and develop we work with a responsive site.

However, when you as a public visitor load the site you only get the HTML, CSS and JavaScript that is required for your device type. Every trace of the site's responsiveness, except for code required to give you a good experience within the context of your device type, is gone.

This way we achieve exactly what we want; the many development related benefits of building a responsive site at the same time as our visitors get a performance optimized experience tailored for the type of device that they are using.

The solution

In order to build a responsive site where all traces of it's responsive nature are cleaned out when it faces public visitors we had to tackle a number of problems:

Channel detection
Filtering of HTML
Filtering of CSS
Filtering of JavaScript

Channel detection

In order to detect what channel a visitor is in we're using device detection in our CDN (Akamai). When a visitor navigates to www.expressen.se the request is routed to Akamai, which inspects the request's user agent and determines whether the user is using a mobile phone, a tablet or a computer. If Akamai doesn't already have the appropriate version of the content in its cache it proceeds to make a request to our servers. This request contains a header telling our servers what type of device the original request was made from.

If our application finds that header in an incoming request it knows that it should serve a version of the site optimized for the device type specified in the header. If on the other hand a request doesn't contain the header our application knows that the request isn't external and will proceed to serve the responsive version of the site.

Using a CDN for device detection is convenient for us but by no means a requirement. If we hadn't been using a CDN or other form of caching layer outside our application we could have implemented the same functionality in our app by inspecting each request's user agent.

HTML filtering

When building a responsive site one can choose between two different approaches for handling how elements should be displayed depending on view port size in HTML and CSS. With the first approach, which is often preferable, the HTML markup doesn't know that it's responsive and instead lets the CSS handle how elements should be positioned and look depending on screen size etc. In the other approach, which Twitter's Bootstrap is an example of, one uses helper CSS classes in the HTML markup to decide whether an element should be displayed or not (or how it should be displayed) depending on browser size.

If we had built a site that would be responsive when it faced public visitors we would probably have used the first method. However in our case we wanted to make it easy to clean up unnecessary HTML elements and for that the second approach, using helper classes, was better. This means that when I look at the site's HTML on my local computer, without channel filtering, a small sample of it can look like this:

<aside class="site-body__column-3 
 hidden-mobile lp_right">
 ...
</aside>

When I instead browse the public version of the site using a computer or tablet the same HTML block looks like the below snippet. Note the absence of the hidden-mobile CSS class.

<aside class="site-body__column-3 
 lp_right">
 ...
</aside>

If I make the same request using a mobile phone the entire HTML block would instead be missing.

In order to accomplish this we first let our application render the responsive HTML with support for all channels and helper classes included. After that, just before the app is about to respond to the incoming HTTP request, a middleware (we use Node.JS) kicks in. The middleware inspects the incoming request and looks for a header (set by Akamai or manually by a developer) containing information about what channel is requested.

If the middleware finds such a header it proceeds to pass the generated markup through a parser. The parser removes all elements that, according to helper classes, shouldn't be shown for the requested channel. The parser also removes all helper classes.

Generating markup only to then parse and rebuild it may sound like an expensive operation. However, our parser, which is built on top of htmlparser2, is simple and pretty fast and only adds a few milliseconds to the total response time. Every millisecond counts though but in practice most of our requests are served directly from our CDN's cache so only a small fraction of all requests are afflicted by the minor overhead added by the parser.

CSS filtering

We write our CSS (actually Stylus) code just as we would have if we would have been working on a regular responsive site with one small exception; in cases where we need to use media queries to adapt for different device types and view port sizes we always do that using helper functions. A fictive example may look like this:

.myElement {
  display: block;
  +mqMinWidth(960px) {
    color: red;
  }
}

The above code example says that elements with the myElement class always should have display: block in all channels. It also says that if the view port is 960 pixels or wider text inside such elements should be red.

When the CSS is built (from Stylus) we create four different versions of it; one for each channel (computers, tablets, phones) and one responsive version. In the responsive version the result of the above Stylus code is:

.my-element {
  display:block
}
@media (min-width:960px) {
  .my-element { color:red }
}

In the CSS built for mobile phones the result is instead:

.my-element {
  display:block
}

In the CSS for tablets, where the view port may or may not be wider that 960 pixels, the result is the same as in the responsive version:

.my-element {
  display:block
}
@media (min-width:960px) {
  .my-element { color:red }
}

For computers we have a minimum width for the site that is above 960 pixels. Therefore the CSS for computers looks like this:

.myElement {
  display:block;
  color:red
}

To summarize; after building our Stylus code we get four different CSS files. One of these makes the site responsive while the other three are heavily optimized for a specific type of device. The decision of what CSS file should be used is handled by our HTML filtering functionality. In practice the markup that includes CSS files looks like the below example prior to HTML filtering.

<link class="hidden-tablet hidden-desktop hidden-responsive" 
  href="/styles/main.mobile.css">
<link class="hidden-mobile hidden-desktop hidden-responsive" 
  href="/styles/main.tablet.css">
<link class="hidden-mobile hidden-tablet hidden-responsive" 
  href="/styles/main.desktop.css">
<link class="visible-responsive" 
  href="/styles/main.responsive.css">

After HTML filtering, in this case for tablets, the above markup is reduced to:

<link href="/styles/main.tablet.css">

JavaScript filtering

Last but not least we need to build channel optimized JavaScript files. The principle is the same as for CSS. We build four different JavaScript bundles, one for the responsive mode and one for each channel. Then we let the HTML filtering decide which one should be used.

Unlike the CSS it's pretty rare that we do anything channel specific in our JavaScript code. In those rare cases though we can do so by inspecting the value of a number of global variables which tell us what channel the script is executing in.

One example of channel filtering in JavaScript is the call to the function that displays a button for opening Expressen's iOS or Android app. This button should only be visible on devices where it's possible to run the app, i.e. tablets and phones. This is how the code for handling the call to the function looks in the responsive bundle:

if (!CHANNEL_DESKTOP) { 
  openInApp();
}

In the responsive scenario our JavaScript bundle contains functionality that populates among others CHANNEL_DESKTOP based on the user's current view port. In the event that the user resizes the browser window the variable's value is changed and events that other code can listen to are triggered.

For a specific channel the JavaScript code is processed, with the help of Vanilla shake, and if-statements that check which channel the user is in are removed. The code inside such if-statements is either removed or left in place depending on the condition in the if-statement and what channel bundle is being built.

The result

When we first started discussing the idea of creating a responsive site that was performance optimized when facing public visitors many of us in the team were skeptical. Would this really work in practice? However, the potential benefits if it did work were very attractive so we decided to give it a try.

Initially there was some effort required to create the components and functionality that I've briefly described in this post. After that initial investment each individual component and the solution in its entirety has worked well.

It has also brought the benefits that we initially hoped for in terms of ways of working and thinking, although we've sometimes caught ourselves thinking in channel specific ways. That may be due to us coming from having worked with channel specific versions previously though.

In terms of performance the solution has been a hit. Visitors to expressen.se only receive the HTML, CSS and JavaScript that is actually needed for the type of device that they are using. Below is an example of a Lighthouse audit of the responsive version of the site, without channel filtering:

Given that the start page of Expressen that is measured here is very long and contains a lot of content the above result isn't exactly bad. However, let's take a look at the result of the same audit performed against the same page but with channel filtering active. Especially note the difference in KB under the "Unused CSS rules" metric.

Book announcement: ElasticSearch Quick Start

Sun, 18 Jan 2015 20:33:54 +0100

I'm happy to announce a book that I've been meaning to write for quite some time - An introduction to ElasticSearch for developers in tutorial form.

About 18 months ago I was working on a customer project in which ElasticSearch was a major component. While I had quite a lot of prior experience with ElasticSearch something at this time got me thinking about the lack of good, simple, tutorials for helping developers new to ElasticSearch get started. So, one night I sat down and I wrote such a tutorial, a blog post titled "ElasticSearch 101". Since it was first published it has accounted for 50-70% of the total traffic to my site and it has received a lot of positive feedback in the comments.

While the positive reception of ElasticSearch 101 is great I've always felt that there's more I'd like to say on the topic. While there has been a number of extensive books about ElasticSearch published the last couple of years and I've written more blog posts about it myself I've always wanted to write a longer version of ElasticSearch 101 in the form of a book. A book that would be shorter than the typical technical book but more detailed and covering more topics than the original blog post.

For 18 months I made excuses for myself for not writing this book. However, over this Christmas holiday I finally started writing. Wise from my previous book (about EPiServer CMS) I decided that I a) wouldn't strive to make each chapter perfect prior to writing the next and b) not to publish it before all chapters were done (in their first versions). This weekend I finished the (probably) last chapter and hit the publish button on Leanpub.

The book is titled ElasticSearch Quick Start. It's not a hundred percent done yet, but it is pretty much finished in terms of the topics that it will cover. What remains is making it *perfect* based on reader feedback, possibly rewriting some parts and by adding more examples.

If you're interested in ElasticSearch check it out. And, if you decide to buy it please give me feedback! I'd really like the book to be the ultimate way to learn how to be productive with ElasticSearch quickly and in order to do that reader feedback is essential.

Notes from learning Go - the basics

Sat, 08 Nov 2014 19:17:06 +0100

I recently decided to learn Go. As in Go the programming language, also known as golang. These are my notes from doing so. In the form of code.

Hello World

package main

//Here's a comment

import "fmt"

func main() {
    fmt.Println("Hello world")
}

Try it.

Variables and types

package main

import "fmt"

func main() {
    // Variables are declared using "var [name] [type]"
    var myVariable string
    myVariable = "I'm a string"
    myVariable += " and I was declared the long way"
    fmt.Println(myVariable)
    
    var length int = len(myVariable)
    fmt.Println("I'm this long:", length)

    // Using short variable declaration, omitting the var keyword
    mySecondVariable := "I'm also a string but I was declared the short way"
    fmt.Println(mySecondVariable)

    const constant = "I'm a constant with inferred type"
    fmt.Println(constant)

    var (
        var1 = "This is"
        var2 = "multiple"
        var3 = "variables"
    )

    fmt.Println(var1, var2, var3)
}

Try it.

If statements and loops

package main

import "fmt"

func main() {
    i := 1
    for i <= 5 {
        fmt.Println(i)
        i += 1
    }

    for i := 6; i <= 10; i++ {
        if i % 2 == 0 {
            fmt.Println(i, "even")
        } else if i % 3 == 0 {
            fmt.Println(i, "divisible by 3")
        } else {
            fmt.Println(i, "odd and not divisible by 3")
        }
    }
}

Try it.

Switch

package main

import "fmt"

func main() {
    i := 3

    switch i {
        case 1: fmt.Println("One")
        case 2: fmt.Println("Two")
        case 3: fmt.Println("Three")
        default: fmt.Println("Uhmn, not sure")
    }

    switch i%2 {
        case 0: fmt.Println("Even")
        default: fmt.Println("Odd")
    }
}

Try it.

Arrays

package main

import "fmt"

func main() {
    var myArray [5]string
    myArray[2] = "I'm the third item"
    fmt.Println(myArray)
    myArray[0] = "Woho, I'm the first item!"
    fmt.Println(myArray)
    fmt.Println("Array length:", len(myArray))

    var intArray [3]int
    fmt.Println("\"Empty\" int array:", intArray)

    intArray = [3]int{1, 2}
    fmt.Println("Integer array with the two first values set:", intArray)

    var messageParts = [3]string{
        "Hello ",
        "there",
        "!",
    }

    var message string
    for i := 0; i < len(messageParts); i++ {
        message += messageParts[i]
    }
    fmt.Println(message)

    // Usign the range keyword. The underscore tells the compiler that we don't need the value (index in this case)
    var secondMessage string
    for _, value := range messageParts {
        secondMessage += value
    }
    fmt.Println(secondMessage)

    for idx, _ := range messageParts {
        fmt.Println("Index:", idx)
    }
}

Try it.

Slices

package main

import "fmt"

func main() {
    var myEmptySlice []string
    fmt.Println(myEmptySlice)

    sliceAssociatedWithArray := make([]int, 5)
    fmt.Println(sliceAssociatedWithArray)

    array := [5]int{1, 2, 3, 4, 5}
    middleSlice := array[1:4]
    fmt.Println(middleSlice)
    
    array[2] = 42
    fmt.Println(middleSlice)

    middleSlice[0] = 41
    fmt.Println(array)

    messageSlice := []string{"hello", "there"}
    messageSlice = append(messageSlice, "how", "are", "you")
    fmt.Println(messageSlice)

    helloOnlySlice := make([]string, 2)
    copy(helloOnlySlice, messageSlice)
    fmt.Println(helloOnlySlice)
}

Try it.

Maps

package main

import "fmt"

func main() {
    var myMap map[string]string
    myMap = make(map[string]string) 

    myMap["hello"] = "hej"
    fmt.Println(myMap)
    fmt.Println(len(myMap))
    fmt.Println(myMap["hello"])

    delete(myMap, "hello")
    fmt.Println(myMap)
    fmt.Println(len(myMap))
    fmt.Println(myMap["hello"])

    // Maps, no matter if they have been initialized or not, return the 
    // zero value for the type if they key doesn't have a value
    var ints map[string]int
    fmt.Println(ints["nothing"]) // Outputs 0
    // Accessing an element can return two values where the second is
    // the result of the lookup
    value, exists := ints["nothing"]
    fmt.Println(value) // Still 0
    fmt.Println(exists) // false

    // Short way of creating maps
    lengths := map[string]int {
        "hello": 5,
        "there": 5,
        "!": 1,
    }
    fmt.Println(lengths)

    // Iterating over elements
    for key, value := range lengths {
        fmt.Println(key)
        fmt.Println(value)
    }
}

Try it.

Functions

package main

import "fmt"

func main() {
    secondFunction()
    thirdFunction("Hello yourself!")
    fourthFunction("How", "are", "you", "today?")
    fmt.Println(fifthFunction())
    fmt.Println(sixthFunction())
    part1, part2 := seventhFunction()
    fmt.Println(part1, part2)
    fmt.Println(eightFunction()())
}

func secondFunction() {
    fmt.Println("Hello world!")
}

func thirdFunction(message string) {
    fmt.Println(message)
}

func fourthFunction(messages ...string) {
    for index, message := range messages {
        fmt.Print(message)
        if index < len(messages)-1 {
            fmt.Print(" ")
        } else {
            fmt.Println()
        }
    }
}

func fifthFunction() string {
    return "Hello world as return value"
}

func sixthFunction() (message string) {
    message = "Hello world as named return value"
    return message
}

func seventhFunction() (string, string) {
    return "Hello world", "from multiple return values"
}

func eightFunction() func() string {
    message := "Hello world from closure"
    return func() string {
        return message;
    }
}

Try it.

Pointers

package main

import "fmt"

func main() {
    message := "I'm the original, and best"

    modifier1(message)
    fmt.Println(message) // "I'm the original, and best"

    // The & operator finds the address for a variable, meaning that &message returns a *string
    modifier2(&message)
    fmt.Println(message) // "modifier2 was here"

    secondMessage := new(string) // the new operatore can be used to create pointers
    modifier2(secondMessage)
    fmt.Println(*secondMessage) // "modifier2 was here"
}

func modifier1(message string) {
    message = "modifier1 was here"
}

func modifier2(message *string) {
    *message = "modifier2 was here"
}

Try it.

Structs

package main

import "fmt"

func main() {
    rect := Rectangle{width: 10, height: 5}
    fmt.Println("Height:", rect.height, "Width:", rect.width)
    fmt.Println("Area:", area(rect))
}

type Rectangle struct {
    width, height int
}

func area(rect Rectangle) int {
    return rect.width*rect.height
}

Try it.

Methods

package main

import "fmt"

func main() {
    rect := Rectangle{width: 10, height: 5}
    fmt.Println("Area:", rect.area())
}

type Rectangle struct {
    width, height int
}

func (rect Rectangle) area() int {
    return rect.width*rect.height
}

Try it.

Embedding

package main

import "fmt"

func main() {
    bird := Bird{}
    bird.Animal.Eat()
    bird.Eat()
    bird.Fly()
}

type Animal struct {
}

func (animal *Animal) Eat() {
    fmt.Println("I'm eating")
}

type Bird struct {
    Animal
}

func (bird *Bird) Fly() {
    fmt.Println("I'm flying")
}

Try it.

Defer

package main

import "fmt"

func main() {
    defer first()
    second()
}

func first() {
    fmt.Println("First function here")
}

func second() {
    fmt.Println("Second function here")
}

Try it.

Panic and Recover

package main

import "fmt"

func main() {
    defer func() {
        message := recover()
        fmt.Println(message)
    }()

    panic("Something is seriously wrong")
}

Try it.

Dynamic mappings and dates in ElasticSearch

Sat, 07 Jun 2014 10:58:18 +0100

JSON doesn't have a date type. Yet ElasticSearch can automatically map date fields for us. While this "just works" most of the time, it can be a good idea to help ElasticSearch help us by instead using naming conventions for dates. Here's why, and how.

ElasticSearch has a feature called dynamic mapping which is turned on by default. Using this we don't have to explicitly tell ElasticSearch how to index and store specific fields. Instead ElasticSearch figures it out itself by inspecting the content of our JSON properties.

Let's look at an example.

curl -XPOST "http://localhost:9200/myindex/tweet/" -d'
{
    "content": "Hello World!",
    "postDate": "2009-11-15T14:12:12"
}'

Given that there isn't already an indexed named "myindex" the above request will cause a number of things to happen in our ElasticSearch cluster.

An index named "myindex" will be created.
Mappings for a type named tweet will be created for the index. The mappings will contain two properties, content and postDate.
The JSON object in the request body will be indexed.

After having made the above request we can inspect the mappings that will have been automatically created with the below request.

curl -XGET "http://localhost:9200/myindex/_mapping"

The response looks like this:

{
   "myindex": {
      "mappings": {
         "tweet": {
            "properties": {
               "content": {
                  "type": "string"
               },
               "postDate": {
                  "type": "date",
                  "format": "dateOptionalTime"
               }
            }
         }
      }
   }
}

As we can see in the above response, ElasticSearch has mapped the content property as a string and the postDate property as a date. All is well.

However, let's look at what happens if we delete the index and modify our indexing request to instead look like this:

curl -XPOST "http://localhost:9200/myindex/tweet/" -d'
{
    "content": "1985-12-24",
    "postDate": "2009-11-15T14:12:12"
}'

In the above request the content property is still a string, but the only content of the string is a date. Retrieving the mappings now gives us a different result.

{
   "myindex": {
      "mappings": {
         "tweet": {
            "properties": {
               "content": {
                  "type": "date",
                  "format": "dateOptionalTime"
               },
               "postDate": {
                  "type": "date",
                  "format": "dateOptionalTime"
               }
            }
         }
      }
   }
}

ElasticSearch has now inferred that the content property also is a date. If we now try to index our original JSON object we'll get an exception in our faces.

{
   "error": "MapperParsingException[failed to parse [content]]; nested: MapperParsingException[failed to parse date field [Hello World!], tried both date format [dateOptionalTime], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: \"Hello World!\"]; ",
   "status": 400
}

We're trying to insert a string value into a field which is mapped as a date. Naturally ElasticSearch won't allow us to do that.

While this scenario isn't very likely to happen, when it does it can be quite annoying and cause problems that can only be fixed by re-indexing everything into a new index. Luckily there's a number of possible solutions.

Disabling date detection

As a first step we can disable date detection for dynamic mapping. Here's how we would do that explicitly for documents of type tweet when creating the index:

curl -XPUT "http://localhost:9200/myindex" -d'
{
   "mappings": {
      "tweet": {
         "date_detection": false
      }
   }
}'

We then index our "problematic" tweet again:

curl -XPOST "http://localhost:9200/myindex/tweet/" -d'
{
    "content": "1985-12-24",
    "postDate": "2009-11-15T14:12:12"
}'

When we now inspect the mappings that has been dynamically created for us we see a different result compared to before:

{
   "myindex": {
      "mappings": {
         "tweet": {
            "date_detection": false,
            "properties": {
               "content": {
                  "type": "string"
               },
               "postDate": {
                  "type": "string"
               }
            }
         }
      }
   }
}

Now both fields have been mapped as strings, which they indeed are, even though they contain values that can be parsed as dates. However, this isn't good either as we'd like the postDate field to be mapped as a date though so that we can use range filters and the like on it.

Explicitly mapping date fields

We can explicitly map the postDate field as a date by re-creating the index and include a property mapping, like this:

curl -XPUT "http://localhost:9200/myindex" -d'
{
   "mappings": {
      "tweet": {
         "date_detection": false,
         "properties": {
             "postDate": {
                 "type": "date"
             }
         }
      }
   }
}'

If we now index our "problematic" tweet with a date in the content field we'll get the desired mappings; the content field mapped as a string and the postDate field mapped as a date. That's nice. However, this approach can be cumbersome when dealing with many types or types that we don't know about prior to documents of those types are indexed.

Mapping date fields using naming conventions

An alternative approach to disabling date detection and explicitly mapping specific fields as dates is instruct ElasticSearchs dynamic mapping functionality to adhere to naming conventions for dates. Take a look at the below request that (again) creates an index.

curl -XPUT "http://localhost:9200/myindex" -d'
{
   "mappings": {
      "_default_": {
         "date_detection": false,
         "dynamic_templates": [
            {
               "dates": {
                  "match": ".*Date|date",
                  "match_pattern": "regex",
                  "mapping": {
                     "type": "date"
                  }
               }
            }
         ]
      }
   }
}'

Compared to our previous requests used to creating an index with mappings this is quite different. First of all we no longer provide mappings for the tweet type. Instead we provide mappings for a type named _default_. This is a special type whose mappings will be used as the default "template" for all other types.

As before we start by disabling date detection in the mappings. However, after that we no longer provide mappings for properties but instead provide a dynamic template named dates.

Within the dates template we provide a pattern and specify that the pattern should be interpreted as a regular expression. Using this the template will be applied to all fields whose names either end with "Date" or whose names are exactly "date". For such fields the template instructs the dynamic mapping functionality to map them as dates.

Using this approach all string fields, no matter if their values can be parsed as dates or not will be mapped as string unless the field name is something like "postDate", "updateDate" or simply "date". Fields with such names will be mapped as dates instead.

While this is nice, there's one caveat. Indexing a JSON object with a property matching the naming convention for date fields but whose value can't be parsed as a date will cause an exception. Still, adhering to naming conventions for dates may be a small price to pay compared to the headaches of seemingly randomly having string fields mapped as dates simply because the first document to be indexed of a specific type happened to contain a string value that could be parsed as a date.

ElasticSearch - nested mappings and filters

Fri, 06 Jun 2014 15:59:19 +0100

There's one situation where we need to help ElasticSearch to understand the structure of our data in order to be able to query it fully - when dealing with arrays of complex objects.

Arguably one of the best features of ElasticSearch is that it allows us to index and search amongst complex JSON objects. We're not limited to a flat list of fields but can work with object graphs, like we're used to when programming with object oriented languages.

However, there's one situation where we need to help ElasticSearch to understand the structure of our data in order to be able to query it fully - when dealing with arrays of complex objects.

As an example, look at the below indexing request where we index a movie, including a list of the cast in the form of complex objects consisting of actors first and last names:

curl -XPOST "http://localhost:9200/index-1/movie/" -d'
{
   "title": "The Matrix",
   "cast": [
      {
         "firstName": "Keanu",
         "lastName": "Reeves"
      },
      {
         "firstName": "Laurence",
         "lastName": "Fishburne"
      }
   ]
}'

Given many such movies in our index we can find all movies with an actor named "Keanu" using a search request such as:

curl -XPOST "http://localhost:9200/index-1/movie/_search" -d'
{
   "query": {
      "filtered": {
         "query": {
            "match_all": {}
         },
         "filter": {
            "term": {
               "cast.firstName": "keanu"
            }
         }
      }
   }
}'

Running the above query indeed returns The Matrix. The same is true if we try to find movies that have an actor with the first name "Keanu" and last name "Reeves":

curl -XPOST "http://localhost:9200/index-1/movie/_search" -d'
{
   "query": {
      "filtered": {
         "query": {
            "match_all": {}
         },
         "filter": {
            "bool": {
               "must": [
                  {
                     "term": {
                        "cast.firstName": "keanu"
                     }
                  },
                  {
                     "term": {
                        "cast.lastName": "reeves"
                     }
                  }
               ]
            }
         }
      }
   }
}'

Or at least so it seems. However, let's see what happens if we search for movies with an actor with "Keanu" as first name and "Fishburne" as last name.

curl -XPOST "http://localhost:9200/index-1/movie/_search" -d'
{
   "query": {
      "filtered": {
         "query": {
            "match_all": {}
         },
         "filter": {
            "bool": {
               "must": [
                  {
                     "term": {
                        "cast.firstName": "keanu"
                     }
                  },
                  {
                     "term": {
                        "cast.lastName": "fishburne"
                     }
                  }
               ]
            }
         }
      }
   }
}'

Clearly this should, at first glance, not match The Matrix as there's no such actor amongst its cast. However, ElasticSearch will return The Matrix for the above query. After all, the movie does contain an author with "Keanu" as first name and (albeit a different) actor with "Fishburne" as last name. Based on the above query it has no way of knowing that we want the two term filters to match the same unique object in the list of actors. And even if it did, the way the data is indexed it wouldn't be able to handle that requirement.

Nested mapping and filter to the rescue

Luckily ElasticSearch provides a way for us to be able to filter on multiple fields within the same objects in arrays; mapping such fields as nested. To try this out, let's create ourselves a new index with the "actors" field mapped as nested.

curl -XPUT "http://localhost:9200/index-2" -d'
{
   "mappings": {
      "movie": {
         "properties": {
            "cast": {
               "type": "nested"
            }
         }
      }
   }
}'

After indexing the same movie document into the new index we can now find movies based on multiple properties of each actor by using a nested filter. Here's how we would search for movies starring an actor named "Keanu Fishburne":

curl -XPOST "http://localhost:9200/index-2/movie/_search" -d'
{
   "query": {
      "filtered": {
         "query": {
            "match_all": {}
         },
         "filter": {
            "nested": {
               "path": "cast",
               "filter": {
                  "bool": {
                     "must": [
                        {
                           "term": {
                              "firstName": "keanu"
                           }
                        },
                        {
                           "term": {
                              "lastName": "fishburne"
                           }
                        }
                     ]
                  }
               }
            }
         }
      }
   }
}'

As you can see we've wrapped our initial bool filter in a nested filter. The nested filter contains a path property where we specify that the filter applies to the cast property of the searched document. It also contains a filter (or a query) which will be applied to each value within the nested property.

As intended, running the abobe query doesn't return The Matrix while modifying it to instead match "Reeves" as last name will make it match The Matrix. However, there's one caveat.

Including nested values in parent documents

If we go back to our very first query, filtering only on actors first names without using a nested filter, like the request below, we won't get any hits.

curl -XPOST "http://localhost:9200/index-2/movie/_search" -d'
{
   "query": {
      "filtered": {
         "query": {
            "match_all": {}
         },
         "filter": {
            "term": {
               "cast.firstName": "keanu"
            }
         }
      }
   }
}'

This happens because movie documents no longer have cast.firstName fields. Instead each element in the cast array is, internally in ElasticSearch, indexed as a separate document.

Obviously we can still search for movies based only on first names amongst the cast, by using nested filters though. Like this:

curl -XPOST "http://localhost:9200/index-2/movie/_search" -d'
{
   "query": {
      "filtered": {
         "query": {
            "match_all": {}
         },
         "filter": {
            "nested": {
               "path": "cast",
               "filter": {
                  "term": {
                     "firstName": "keanu"
                  }
               }
            }
         }
      }
   }
}'

The above request returns The Matrix. However, sometimes having to use nested filters or queries when all we want to do is filter on a single property is a bit tedious. To be able to utilize the power of nested filters for complex criterias while still being able to filter on values in arrays the same way as if we hadn't mapped such properties as nested we can modify our mappings so that the nested values will also be included in the parent document. This is done using the include_in_parent property, like this:

curl -XPUT "http://localhost:9200/index-3" -d'
{
   "mappings": {
      "movie": {
         "properties": {
            "cast": {
               "type": "nested",
               "include_in_parent": true
            }
         }
      }
   }
}'

In an index such as the one created with the above request we'll both be able to filter on combinations of values within the same complex objects in the actors array using nested filters while still being able to filter on single fields without using nested filters. However, we now need to carefully consider where to use, and where to not use, nested filters in our queries as a query for "Keanu Fishburne" will match The Matrix using a regular bool filter while it won't when wrapping it in a nested filter. In other words, when using include_in_parent we may get unexpected results due to queries matching documents that it shouldn't if we forget to use nested filters.