SoftwarePhilosophy

Literate programming really works for tutorials

2016-09-19

Have you heard about literate programming? It's a technique that lets you write a computer program the same way as if you were writing a piece of literature. And while it may seem exotic and unnecessary, there are some cases when it's quite useful. I believe I have found one of them. In this post, I will describe a solution for writing a tutorial in the spirit of literate programming by using nodejs-based, language-agnostic, literate programming tool.

Background

Literate programming is a technique of writing computer programs in natural (spoken) language. When using this technique, documentation receives most of programmers attention while code is treated as a formality. Code snippets are typically embedded in the essay and get extracted for compilation during project build.

The main idea is to regard a program as a communication to human beings rather than as a set of instructions to a computer.

~ The CWEB System of Structured Documentation by Donald E. Knuth and Silvio Levy

First literate programming tool, developed by Donald Knuth, was called WEB and worked with Pascal only. Short after, Knuth developed CWEB, which is WEB for C language. In response to WEB, Norman Ramsey created noweb—a literate programming tool compatible with any programming language. It started a language-agnosticity trend in the field.

Research

So, I was faced with a task of writing a tutorial on using hyper-text-slider with node and browserify. Since I have this habit of making my life more exciting, and since writing a tutorial is an inherently dull task, I started thinking. From this thinking emerged a question—Is it possible to automatically test a tutorial?

After few minutes of Googling around, I had my answer—literate programming.

HyperText Slider is a web slideshow component that utilizes CSS3 transitions to control background and foreground animations of changes between slides. It has declarative (HTML-only) API which is very easy to use, and JavaScript API for fine-grained control. Using hyper‑text‑slider with node and browserify means using a node-base build system for front-end technology stack (HTML, CSS and JavaScript). I needed a literate programming solution that supports front-end stuff and has a plugin for gulp (my build utility of choice for node-based projects).

As it turns out, number of literate programming solutions for JavaScript in suprisingly big (npm search literate gives 288 results). I really disliked docco which seemed to be the most popular one. It was advertized as quick and dirty yet it was still too complicated for my taste. After trying a few, I ended up choosing writ. It does only one thing—extracts any code from Markdown (with very basic support for macro-definitions).

Solution

Documentation (tutorial), human-written, contains snippets of code needed for a website to be built. It is a single source of information (actual sources). Tutorial is written in markdown and hosted on Github which automatically renders markdown files into nice HTML documents.
Code snippets are extracted from the documentation by invoking writ from a gulp script. Writ produces one output file for each markdown file, so the tutorial must have been divided into multiple pages—3 source files (HTML, Sass and JavaScript), a shell script that sets up the project, and a gulpfile that builds generated sources. There actually are 2 gulpfiles in the project. First that invokes writ (human-written), second that builds the project (generated). Extracted JavaScript and Sass code is linted by invoking linter plugins after writ. All generated sources are checked into the source control so that the project can be cloned and built.
Project is built by invoking gulpfile script which was extracted from the documentation. Building process produces browser-runnable files (HTML, CSS, JavaScript), which are also checked into the source control. Github Pages is active with master branch set as source so that compiled project can be run in the browser.

This way, I'm killing three birds with one stone. The project is:

A tutorial on creating a web slideshow using hyper-text-slider, node and browserify,
An example project which can be cloned and compiled,
A website which shows live results of the compilation.

Configuring Writ

There are two gulpfiles in the project. Generated one has the default name (gulpfile.js). Handwritten one must be named differently (.writ.gulpfile.js).

NOTE

Below paragraphs require basic understanding of gulp. If you're not familiar with it, please read Build Configuration chapter from the tutorial. If you know gulp, it may still be useful to read it as it contains soure code of generated gulpfile.

We need gulp, gulp-util, gulp-writ, gulp-rename2, gulp-eslint, gulp-stylelint, merge-stream and del modules (they must be installed with npm install in order to make this gulpfile work).

'use strict';

var gulp = require('gulp');
var gutil = require('gulp-util');
var writ = require('gulp-writ');
var rename = require('gulp-rename2');
var eslint = require('gulp-eslint');
var stylelint = require('gulp-stylelint');
var merge = require('merge-stream');
var del = require('del');
var child = require('child_process');

The project contains following folders:

doc/ containing the tutorial,
src/ containing generated sources,
dist/ containing build output files (browser-comsumable).

As it will be used as a dependency of generate, clean task must be defined first. Deleting files with del module requires using gulp in asynchronuous mode (task function has a callback which is called by del). We want to delete all files generated in previous run of writ which are contained in src folder, but without deleting the folder itself.

gulp.task('clean', function(callback) {
  return del([ 'src/**/*', '!src/' ], { force: true }, callback);
});

The tutorial consists of following files:

doc/0_introduction.markdown,
doc/1_setup.sh.md,
doc/2_gulpfile.js.md,
doc/3_script.js.md,
doc/4_index.html.md,
doc/5_style.scss.md.

First file (0_introduction.markdown) does not contain any code snippets so it has different extension. This way, it is still properly rendered on Github yet not catched by writ's glob pattern.

Each file is prefixed with a number which indicates reading order of the tutorial. Numbers also keep the files properly ordered when listed on Github. gulp-writ plugin removes .md extension but we need to use rename2 in order to remove number prefixes. setup.sh and gulpfile.js will be placed in project main folder while all generated sources will be placed in /src.

function removeNumberPrefix(pathObj, filePath) {
  return pathObj.dirname(filePath) +'/'+ pathObj.basename(filePath).replace(/^[0-9]+_/, '');
}

gulp.task('generate', [ 'clean' ], function() {
  var sources = gulp.src([ 'doc/*.md', '!doc/*.sh.md', '!doc/*gulpfile.js.md' ])
    .pipe(writ().on('error', gutil.log))
    .pipe(rename(removeNumberPrefix))
    .pipe(gulp.dest('src/'))
  ;
  var setupAndConfig = gulp.src([ 'doc/*.sh.md', 'doc/*gulpfile.js.md' ])
    .pipe(writ().on('error', gutil.log))
    .pipe(rename(removeNumberPrefix))
    .pipe(gulp.dest('.'))
  ;
  return merge(sources, setupAndConfig);
});

Generated sources are ready to be linted.

gulp.task('lint:sass', [ 'generate' ], function() {
  return gulp.src([ 'src/*.sass' ])
    .pipe(stylelint({
      reporters: [ { formatter: 'string', console: true } ],
    }))
  ;
});

gulp.task('lint:javascript', [ 'generate' ], function() {
  return gulp.src([ '*.js', 'src/*.js' ])
    .pipe(eslint())
    .pipe(eslint.format())
    .pipe(eslint.failAfterError())
  ;
});

gulp.task('lint', [ 'lint:sass', 'lint:javascript' ]);

Linted sources are ready to be built. In case of this tutorial, building the project means invoking another gulp process with generated gulpfile.

gulp.task('dist', [ 'lint' ], function() {
  child.execSync('gulp');
});

Running Writ

Invoking generate task extracts sources from the tutorial.

gulp --gulpfile=.writ.gulpfile.js generate

Invoking dist task extracts sources, lints them and builds the project.

gulp --gulpfile=.writ.gulpfile.js dist

Above command is equal to invoking lint task and then running gulp with generated gulpfile.

gulp --gulpfile=.writ.gulpfile.js list
gulp

Invoking dist task after changing the documentation will propagate changes into src/ and dist/ folders. It should be invoked before each commit.

Wrap Up

After setting everything up, all that remains is writing the tutorial. It will be automatically tested in a sense that contained code snippets will be linted and checked if they compile successfully. Resulting program must still be run by a human, but it's much less work than manually testing each step of the tutorial.

Implementing a watch task would add more automation into the process It's there in my tutorial's .writ.gulpfile.js, but as it's not essential, I haven't include it this blogpost.

I must say that writing a tutorial using literate programming was an exciting experience. Writ is a great, simple tool for literate programing in Markdown. I really recommend using it. I'm curious of your opinion on literate programming and writ. If you have any thoughts to share or questions to ask, please do in the comments below.

References

literateprogramming.com is a website that gathers literate-programming-related white paper fragments,
In his original paper on literate programming, Donald Knuth describes first attempt of implementing this concept in a framework called WEB,
Literate Programming—a book by Donald Kunth—contains a collection of his papers on this technique alongside some new material,
Cunningham & Cunningham Wiki contains an interesting discussion on literate programming.
Haskell Wiki contains information on how to do literate programming by using only Haskell's native features,
Github Pages Help website contains information on how to set a source branch for Github Pages in your project,
Tutorial described in this blogpost is hosted on github.

Compile-Time Unit Testing

2016-04-04

These days, unit testing became a major part of software engineer's work. We familiarized ourselves with unit testing frameworks based on xUnit architecture (e.g. JUnit, SUnit) where tests are executed during run time of the test program. As of time of this writing, it's the most popular way of testing in an object-oriented environment. Few years ago, I started using compile-time unit testing in my C++ experiments. As it turned out quite well, I will try to share my experience in here. This article describes reasons for using compile-time unit testing, provides examples of code that can be tested this way, and shows how to debug such tests.

NOTE

Examples in this article are written in C++, but used terminology and the concept of compile-time unit testing applies to many other statically-typed languages. All examples should be compilable with gcc4.9. They can be found on github in mchalapuk/static-tests repo.

Static Binding

There are two ways of binding values to symbols in C++ language—static (AKA early) binding and dynamic (AKA late) binding. To make a long story short—a (not only) example of dynamic binding is method polymorphism, where pointer to method's implementation is resolved during program run time (wich is late). Static binding happens during compilation. Value is computed by the compiler and inserted into the output code instead of a computation procedure.

The simplest example of a value that is bound statically is a literal of any type.

auto i = 0; // zero is statically bound (value is known during the compilation)
auto s = "static binding"; // string literal is statically bound

But what makes static binding really powerful is a computation performed at compile time.

// Mathematical PI constant.
// Literal of type double is bound statically.
const auto PI = 3.141592L;

// Degrees to radians convertion ratio.
// The compiler will compute right-hand side expression and replace it with
// single value, because all values used in the expression are bound statically.
// This means that value of DEG_2_RAD is also bound statically.
const auto DEG_2_RAD = 2 * PI / 360;

Those examples are just scratching the surface of static binding in C++.

Constant Expressions

Since C++11 standard, we have a feature of constant expressions in the language. It extends the posibilities of static binding from simple expressions to whole blocks of code.

// Function converting degrees into radians.
// Declared as constant expression, which means that it can be executed
// during the compilation (the code is statically bound).
constexpr double deg2rad(double deg) {
  return deg * DEG_2_RAD;
}

There are many requirements that a function must meet to be a valid constant expression¹. In essence, blocks of code declared as constant expressions can not have any side effects. As its code is statically bound, deg2rad function can be tested during the compilation.

Writing compile-time unit tests is quite easy since C++11 brougth static_assert into the game. It works the same way as normal assertion but with one difference—it is evaluated during the compilation. This implies that its argument must be a constant expression. If expression doesn't convert to true, compilation fails.

#include <cmath>

// A helper function for comparing floating point numbers.
constexpr bool float_equals(double lhs, double rhs) {
  return std::abs(lhs - rhs) < 0.00001L;
}

namespace deg2rad_tests {

static_assert(float_equals(3.141592L, deg2rad(180)), "deg2rad(180) == 3.141592L");
static_assert(float_equals(0.017453L, deg2rad(1)), "deg2rad(1) == 0.017453L");
static_assert(float_equals(17.104226L, deg2rad(980)), "deg2rad(980) == 17.104226L");

} // namespace deg2rad_tests

Above code contains three compile-time unit tests of deg2rad function. Compiling this code runs the tests. Tests in this form are not very maintainable as they contains many duplications. Function calls contain the same information as assertion messages. This problem can can be resolved by using a macro-definition instead of a helper function.

#include <cmath>

#define STATIC_ASSERT_FLOAT_EQUALS(expected, actual) \
  static_assert(std::abs(expected - actual) < 0.00001L, #expected " == " #actual);

namespace deg2rad_tests {

STATIC_ASSERT_FLOAT_EQUALS(3.141592L, deg2rad(180));
STATIC_ASSERT_FLOAT_EQUALS(0.017453L, deg2rad(1));
STATIC_ASSERT_FLOAT_EQUALS(17.104226L, deg2rad(980));

} // namespace deg3rad_tests

All further examples of tests will use macro-definitions.

Object methods, static methods and construtors can also have constexpr modifier. The next example contains a simple, statically bound class that represents a vector in 2D space. Its constructor, getters and arithmetic operators are unit-tested during the compilation.

// Two-dimentional vector.
// Class is immutable, all operations (including constructor) are constant expressions.
class vec2 {
 public:
  constexpr vec2(int x = 0, int y = 0) : _x(x), _y(y) {
  }

  constexpr int x() const {
    return _x;
  }
  constexpr int y() const {
    return _y;
  }

  constexpr vec2 operator+(vec2 const& rhs) const {
    return vec2(_x + rhs._x, _y + rhs._y);
  }
  constexpr vec2 operator-(vec2 const& rhs) const {
    return vec2(_x - rhs._x, _y - rhs._y);
  }

 private:
  int _x;
  int _y;
}; // class vec2

#define STATIC_ASSERT_EQUALS(expected, actual) \
  static_assert(expected == actual, #expected " != " #actual)

namespace vec2_tests {

STATIC_ASSERT_EQUALS(1, vec2(1, 2).x());
STATIC_ASSERT_EQUALS(2, vec2(1, 2).y());
STATIC_ASSERT_EQUALS(3, (vec2(1, 2) + vec2(2, 3)).x());
STATIC_ASSERT_EQUALS(5, (vec2(1, 2) + vec2(2, 3)).y());
STATIC_ASSERT_EQUALS(-1, (vec2(1, 2) - vec2(2, 3)).x());
STATIC_ASSERT_EQUALS(-1, (vec2(1, 2) - vec2(2, 3)).y());

} // namespace vec2_tests

Number of unit tests in above example is minimal. Production code will probably contain more than one test per operation.

Meta-Functions

There is a popular statement that meta-language of C++ is C++ itself. It's true, but more precicely, meta-programming in C++ is done using C++ templates, which are whole another, purely functional programming language, embedded into C++. Templates provide a way to abstract out a common code from a family of classes (or functions) without use of runtime polymorphism². C++ code is generated from template specializations during the compilation. That's why templates can be compile-time unit-tested.

Meta-function is a template structure that takes a type as parameter and returns another type in a typedef or a value in a static variable (or both). One of the simplest meta-functions is the one that adds pointer to a type.

// Since C++11, there is a one-liner for doing this.
template<class arg_t> using add_pointer_t = arg_t*;

In order to test this meta-function, one of type traits from the standard library (std::is_same) must be used. It takes two types in template parameters and returns true it they are identical.

#include <type_traits>

#define STATIC_ASSERT(expr) static_assert(expr, #expr)

namespace add_pointer_t_tests {

STATIC_ASSERT((std::is_same<add_pointer_t<int>, int*>::value));
STATIC_ASSERT((std::is_same<add_pointer_t<long>, long*>::value));
STATIC_ASSERT((std::is_same<add_pointer_t<float>, float*>::value));
STATIC_ASSERT((std::is_same<add_pointer_t<int*>, int**>::value));

} // namespace add_pointer_t_tests

The next example is less trivial. It contains a meta-function that counts types passed in its template parameters. Function uses:

parameter packs, a C++11 feature that enables creating variadic templates,
partial specialization, which in this example is used to make the template recursive.

// Meta-function that counts types.
// It takes a parameter pack (which can be empty) in its template argument.
// There is no implementation, because everything will be handled in specializations.
template <class ...args_t>
struct count_types;

// Partial specialization that will be chosen by the compiler
// if there is at least one type (head_t) in parameter pack.
template <class head_t, class ...tail_t>
struct count_types<head_t, tail_t...> {
  // value is computed recursively by calling the same meta-function
  static const size_t value = 1 + count_types<tail_t...>::value;
};

// Partial specialization that will be chosen only if parameter pack is empty.
template <>
struct count_types<> {
  static const size_t value = 0;
};

#define STATIC_ASSERT_EQUALS(expected, actual) \
  static_assert(expected == actual, #expected " == " #actual)

namespace count_types_tests {

STATIC_ASSERT_EQUALS(0, (count_types<>::value));
STATIC_ASSERT_EQUALS(1, (count_types<int>::value));
STATIC_ASSERT_EQUALS(3, (count_types<int,float,double>::value));
STATIC_ASSERT_EQUALS(10, (count_types<int,int,int,int,int,int,int,int,int,int>::value));

} // namespace count_types_tests

Debugging

GCC compiler prints very limited information when static assertion fails. To demonstrate this, let's change one of the partial specializations of count_types, so it will contain a bug.

template <class head_t, class ...tail_t>
struct count_types<head_t, tail_t...> {
  // A programmer forgot to pass tail_t... to recursive call
  static const size_t value = 1 + count_types<>::value;
};

Now, when processing following assertion...

STATIC_ASSERT_EQUALS(3, (count_types<int, float, double>::value));

...the compiler will produce following output.

error: static assertion failed: 3 == (count_types<int, float, double>::value)
   static_assert(expected == actual, #expected " == " #actual)

The information of wich assertion has failed is present, but it is not known what value was returned from the tested meta-function. A way to print a static value is to put it in a template parameter that will be visible in the ‘required from’ stack trace (a part of compilation error output). Next code snippet contains an utility for printing static values this way.

// Compilation error must be generated in order to print anything.
// This function generates this error (static assertion always fails).
// Template parameters of calls containing failed static assertions
// are never printed by gcc compiler (val_t will not be printed).
template <class val_t>
constexpr val_t staticAssertFail(val_t value) {
  // Value must be passed to trick the compiler
  // that it's needed to evaluate the assertion.
  static_assert(value && false, "debugging value");
  return value;
}

// Another function that calls the function above.
// Template parameters of this function will be printed
// in 'required from' stack trace.
template <class val_t, val_t value>
constexpr val_t debugStaticValue() {
    return staticAssertFail(value);
}

// Helper macro-definition for usage in debugged code.
// To use it, just wrap a value you wish to debug.
#define DSV(v) \
  debugStaticValue<std::remove_cv<std::remove_reference<decltype(v)>::type>::type, v>()

DSV helper macro is reusable. You may copy and paste it into your project if you like (or download debug.hpp file from mchalapuk/static-tests project).

// Erroneus value wrapped in DSV macro.
STATIC_ASSERT_EQUALS(3, DSV((count_types<int, float, double>::value)));

Above code produces following compiler output.

required from ‘constexpr val_t debugStaticValue() [with val_t = size_t; val_t value = 1ul]’
required from here
error: static assertion failed: debugging value
error: static assertion failed: 3 == DSV((count_types<int, float, double>::value))
note: in expansion of macro ‘STATIC_ASSERT_EQUALS’

Use of helper macro resulted in three extra lines at the beginning of the output. First line contains erroneus value (1ul) and its type (size_t).

Unfortunately, above method can not be used on all values. There are strict requirements that a value must meet to be a valid non-type template parameter.

Rationale

There are many reasons for using static unit testing. First of all, having statically bound code implies statically bound tests. If all code is executed during the compilation, but tests are done in runtime, resulting program will contain only comparisons between meaningless values.

// For example following program.
int main() {
  assert std::numeric_limits<unsigned char>::max == 255;
}

// Will compile to something like this.
int main() {
  assert 255 == 255; // What are the meanings of this values?
}

Using static assertions instead of runtime ones will result in everything (code and tests) being run on the compiler. Resulting test program will be empty, which is much more elegant.

Elegance is not a dispensable luxury but a quality that decides between success and failure.

~ Edsger W. Dijkstra

Static assertions are also the only way of checking that code is statically bound, as intended. Skipping this type of test may result in instructions sliping to runtime by accident which can drastically affect program's performance.

Compile-time unit testing reduces turn around time, as full project doesn't have to be built before running any tests. Static assertions fail during compilation of a single file which can greatly reduce feedback time and increase maintainability of a code base.

Summary

Examples from this article include C++ constructs of three types: statically bound functions, statically bound classes, and meta-functions. These (more or less) cover the full spectrum of code which can be tested during the compilation. Things to take out from this article are that many features of modern C++ utilize static binding, and that statically bound code requires compile-time unit tests.

As of now, compile-time unit-testing in C++ is not ideal. Besides static_assert, there aren't any debugging facilities of compile-time code. Templates provide a simple way of debugging statically bound values, but this technique works only with types that are accepted as template parameters, which of—most notably—floating point numbers are not part. Also, the fact that there are no test reports may be interpreted as a problem. The only way of proving that test is run by the compiler is to make it not pass. On the other hand, this is consistent with Linux's ‘empty output means no error’ convention, so maybe it's not such a downside after all.

Good knowledge of rescent C++ standards and build environments is definitely a requirement for an efficient use of compile-time unit testing. The technique still needs work, but the benefits most definitely outweight the problems. If you are not up-to-date with current C++, this may be a good reason to brush up your skills.

References

A White paper titled “Compile-time Unit Testing” written by Aron Barath and Zoltan Porkol (Lorand University) contains many examples written in C++ and in experimental language created for purposes of testing this concept.
Peter Dimov's article titled “Simple C++11 metaprogramming” contains many examples of operations on type lists implemented using parameter packs.
Eric Niebler wrote a blogpost titled “Tiny Metaprogramming Library” where he argues that we don't need big meta-programming libraries like Boost.MPL, as type lists are now implemented in C++ with just one line of code. He provides many examples.
An old StackOverflow thread titled “Runtime vs Compile time” in great detail describes the difference between runtime and compile-time errors.

Annotations

^{1} Please consult the documentation of constexpr keyword for full list of requirements.

^{2} There is even a template technique called static polymorphism, where implementations are bound to calls during the compilation.

Why do we learn next programming languages?

2016-02-19

Self-education can be as painful as it can be gratifying. The trick is to know when to expect what. Contents of this article are aimed at establishing a correct (IMHO) mindset in the topic of learning programming languages. I will try to convince you that knowing multiple languages well is the way to excell as a software developer. You'll get to know the benefits of polyglot programming, reasons for not liking it, what programming gurus have to say in this matter, and how to choose the next language to learn.

Mindful Engineering

One of the most important skills of any engineer is being able to choose a tool for the job. Programming languages come with unique traits, which often makes them well suited for solving problems of particular type. Game engines are written in C++, as it enables precise control over system resources. Java has one of the best technology stacks for writing server-side business logic. It's good to code data processors in functional languages such as Scala or Haskell. The list goes on... Coding everything in the same language, if it's even possible, means not using the best tools out there.

If you're a hammer, everything looks like a nail.

~ Abraham Maslow's The Psychology of Science

Existence of a better solution than the one just developed—in another language or not—is highly probable. Different languages promote different ways of solving the same problems. Techniques learned in one language can be used in many others, which means, that by learning a new language you are getting more skilled at using all the languages you know. Each programming language you know makes you more capable of making decisions about technology and a better software engineer overall.

Another benefit from an engineer's point of view is doing full-stack development, which is lately getting more in-demand by employers. Many technology stacks are heterogeneous in terms of programming languages. Typically, different languages are used on different layers. Software developers familiar with entire technology stacks are more efficient at finding bugs and much better at performance and optimization work.

Efficient Learning

Most software developers have read something about functional programming and predicate logic during their period of formal education, but this knowledge stays very theoretical until something useful is created using it. As you practically try languages implementing other paradigms than you already know, you learn new idioms and techniques, while getting to know their pros and cons in the process. This is a great way to grow as a programmer.

Once you stop learning, you start dying.

~ Albert Einstein

While learning multiple programming languages, we unconsciously build a meta-language in our heads, which is then used to translate between programming languages. In time, this meta-language becomes very handy in learning yet new languages. A master of a dozen programming languages may start efficiently using a next one just after few-hour scrolling through its specification. This ability is referred to as a generalized language learning skill.

More Contribution

Trying new languages involves digging through forums and mailing lists. By introducing ourselves to the new community, we may learn about new ways of standardization, coding conventions, or even social interactions. Active involvement in multiple programming communities creates a potential for expansion of professional network.

Programmers, that are highly skilled in using many languages, are often the ones writing new libraries, frameworks and tools. If you are able to identify the need for a library that is already available in another language, you may also try to reimplement it in the language you're currently using. Examples of such successful reimplementations include (but are not limited to):

Jasmine(JavaScript) and Mocha(JavaScript) based on RSpec, a BDD framework for Ruby¹,
offensive.js(JavaScript) and prego(Go) based on Preconditions(Guava, Java)²,
std::future class from C++11 standard based on quite old Future from Java,
Play framework for Java and Grails for Groovy, both based on concepts from Ruby on Rails,
Clojure language, which is essentially a copy of Lisp that works on JVM.

One of the highest forms of contribution is language design. After mastering multiple languages from various programming paradigms, you can improve some of them by adding your 2 cents to the work of standard communities. People that see the need, have proper theoretical knowledge and feel up for it, may even want to start a project of a totally new programming language on their own.

Better Career

As you learn more and more languages, the number of job opportunities that matches your profile raises. What's more, being a polyglot programmer gives you an advantage over other candidates for a position you may be interested in. With higher probability of recruitment resulting in hire, you can be more precise in selecting positions for which you would like to apply. This ultimately leads to being happier with your job.

Freelancers can utilize the benefits of knowing multiple languages even more. As freelancing jobs are project-based, there will be more occasions to change daily-used languages. The purpose of learning new languages will be to use them in the next project, instead of just getting better at programming. As in the case of permanent-job programmers, more languages means more job offers, more expertise and better work efficiency, but for freelancers better efficiency is also directly connected with more money.

Following Gurus

Many succesful book authors and designers of popular programming languages have something to say about the importance of continuous learning.

Robert C. ‘Uncle Bob’ Martin writes in his second best-selling book titled The Clean Coder:

The frenetic rate of change in our industry means that software developers must continue to learn copious quantities just to keep up. Woe to the architects who stop coding—they will rapidly find themselves irrelevant. Woe to the programmers who stop learning new languages—they will watch as the industry passes them by. Woe to the developers who fail to learn new disciplines and techniques — their peers will excel as they decline.

In this statement he touches another problem (high-flying architects), but the essence is about learning. This guy has 40+ years of experience in computer programming. He's witness how many languages and technologies got obsolete.

The Pragmatic Programmer—a popular book by Andy Hunt and Dave Thomas—contains a list of very useful programming tips. Tip #8 “Invest Regularly in Your Knowledge Portfolio” states:

Learn at least one new language every year. Different languages solve the same problems in different ways. By learning several different approaches, you can help broaden your thinking and avoid getting stuck in a rut. Additionally, learning many languages is far easier now, thanks to the wealth of freely available software on the Internet.

Authors of this book took a huge part in creating the Agile Manifesto. Some Agile values were based on the tips from The Pragmatic Programmer.

Larry Wall—the author of Perl scripting language—lists following five languages as must-haves in the world of modern programming:

JavaScript, to know the consequence of enabling scripts in a web browser,
Java, because managers like it due to the fact that programmers are easily interchangeable,
Haskell—a language for geniuses by geniuses—even just for the sake of knowing how smart one must be in order to program in it,
C, because most of the compilers, VM's and kernels are implemented in this language,
Python, Ruby or Perl—one of the scripting languages—to have a tool for everyday use.

Having Doubts

The most common reason for not learning another language is the lack of time. These days, not every software developer is a total nerd that do coding and sleep on the same chair. Some of us would like to have a life besides work. Sure, there's nothing wrong with that, yet we have to remember that the further in the process of self-education, the less time it takes to learn new stuff. An expert polyglot programmer should be able to educate himself, do active menthoring and write good-quality code, all during a highly productive 8-hour workday.

One must specialize in order to be good at something. This statement is quite often used as an argument against polyglot programming. And while the statement in itself is very true, there is an open question of how narrow a succesful specialization should be. Nobody is a master at every language he knows. That is not the point. We learn more languages because we want to be better software developers. We want to have broader views, know strengths and weeknesses of the languages we know by comparing them to new ones. Besides that, keeping up-to-date knowledge in more than five languages is virtually impossible.

At some point, after mastering several languages, a programmer may already know most of the features a language can have. What's the point of learning another language by then...? Well, it's not about language features. We learn new stuff to have more and more tools in our toolboxes. Knowledge of many language features is helpful when learning next languages, but this doesn't mean another language isn't worth learning. A language may have similar features to the one you already know, but it may have a better syntax, a different target platform or some detail, which makes it more suitable for a particular use case.

Another reason for not learning another language could be that languages are easy to learn in comparison with frameworks. In other words—learning a framework takes more time than learning a language. This is true for commercially standardized frameworks with very long specifications (e.g. EJB), yet there are many frameworks, which require very little learning and almost no configuration. All thanks to the concept called Convention over Configuration (CoC), popularized by Ruby on Rails (RoR) framework. Today's frameworks learn from each other and adopt the same good solutions. Generalized learning skill is also applicable in here.

Choosing The Next Language

So, you're a programmer with experience of a year or two and you use one programming language. What should you do next? There are so many options out there...

My advice is to take a look at programming paradigms on Wikipedia and choose something that is totally different from what you already know. If you know an imperative language, pick a declarative one. If you know meta-programming with reflection, try a language with template meta-programming. Choose a language with different model of concurrency, different features of the type system, different overall complexity, and even different target platform.

It is probably a good idea to check out which languages are in demand before choosing the next step in your learning process. Tiobe Index and IEEE Spectrum are good sources of information about trends in programming language popularity.

Making a Decision

When still in doubt about starting to learn another programming language, you may ask yourself some of the following questions:

Is programming my passion?
How much of a salary would I like to receive?
What are my priorities is life?
Will I move to management at some point?

Not everyone is ought to be the best, but there is a great, forever unsatisfied need for more experts in the field of software development. Experience of an average programmer is slowly declining as beginners are produced by universities in greater quantities every year. I encourage you to become an expert and make things just slightly better for everyone.

I hope that this article provides information helpful in making a decision about a career. Which path did I take? I think that's obvious. But what do you think? Are you willing to become a polyglot programmer?

References

Detailed information about how full-stack developers are good at performance and optimization work can be found in Carlos Bueno's article on Facebook's technical blog.
Good tutorial on how to learn multiple programming languages can be found on WikiHow.
Robert C. Martin argues that Clojure is the best pick for a functional language on his blog.
Question on Learning Multiple Languages Simultaneously at programmers.stackexchange forum contains information on pros and cons of learning multiple languages at once.
Lauren Orsini describes the story behind Facebook's Hack language in article titled How Many Languages Do Developers Need To Know? posted on ReadWrite tech news website.
Coding Dojo offers some good advice on what to focus on when learning new languages.
Comprehensive discussion and suggestions of languages to learn can be found in thread on programmers.stackexchange titled (Why) Should I learn a new programming language?
Big Think interviewed Bjarne Stroustrup and Larry Wall about five programming languages everyone should know. Both interviews are available on Youtube.
Kira M. Newman lists her reasons for learning many programming languages on tech.co.
Mary Branscombe describes her concept of causality between more programming languages and more money on pluralsight.com blog.

Annotations

^{[ 1 ]} Ultimate precursor of this style of testing is JBehave framework for Java.

^{[ 2 ]} Preconditions class from Java is itself based on Eifel's require keyword, which was part of one of the early Design by Contract utility implementations.

Distributed Transactions

2015-11-24

Recently, I had few email conversations with past job coleague of mine. The most challenging one was on the topic of distributed transactions. It turns out we both had some experience with them, but our opinions were rather different. As this topic is pretty hot these days, I decided to write down our insights in this article. The goal is to list disadvantages of distributed transactions, show how they are overused, provide an alternative approach, and specify conditions under which each technique may be applied.

Why Not?

Software developers are attracted to distributed transactions because they provide all‑or‑nothing solution across multiple database resources. Standardized interfaces of a distributed resource (XA) and standardized transactional APIs (JTA, repoze) greatly reduce the time-cost of applying this technique. It results in less code written, because domain logic is free of transaction-related code. On the other hand, there are many downsides to distributed transactions.

Two Phase Commit¹ lowers maximum throughput of the system. The more nodes are involved in the transaction, the lower throughput is achievable. Cost of needed hardware is increased.
Risk of contention grows with number of users. More contention affects performance and limits scalability to a point where performance becomes unacceptably low. Performance problems can be resolved (in retrospective) by optimizing the data model and re-designing the transactions, but it means that a system built using distributed transactions will never be scale-agnostic².
All-or-nothing availability. If one resource inside a transaction is unavailable, whole operation must fail. Partial availability of the system is not possible. What's more, availability of a single operation is inversely proportional to number of resources at use in the transaction. A system containing 5 resources, each 99.999 % available, will be availabile pow(99.999, 5) ~= 99.995 % of the time.
All resources must implement the same transactional API. If a resource don't natively implement required interface, an adapter, which implements it, must be added to the system. This increases overall system complexity because of additional dependency added.
Distributed transaction is just a technical candy. The concept is based on the assumption that transactions⁴ between subsystems are a cross-cutting concern (like logging or concurrency), while in true they should be treated as parts of program's domain logic. Generic implementation of a distributed transaction is strictly technical and very complicated. Thinking about errors in the second phase of 2PC is hard and requires detailed knowledge of the commit protocol⁵, while transaction implemented as part of entity's interface is always simple.

Transaction⁴ Inside Contract

Let's say we have to model a newsletter subscription system with prizes for first 10 subscribers. Messages must be sent to winners during successful subscription. We have two resources to be used inside a distributed transaction—a database for storing subscriber entities and a mail server for sending emails to subscribers. Each time a subscriber is added to the database, distributed transaction must be set up. After adding subscriber, we have to count all subscribers added so far. If there is less than or equal 10, add an email to send in the transaction. If all is well, commit.

The relationship between subscriber entities and sent emails, which is an essential part of the model, gets marginalized when using distributed transactions (hidden inside a technical solution). All ACID properties are kept, yet not all of them are required across all resources. Next paragraph describe alternative model, wich doesn't use distributed transactions.

After checking the database for a winner, we can store another entity that represents an email. Then write a simple daemon that periodically checks the database for unsent emails, send them and mark them in database as sent. Done. Operations are now asynchronous, transactions are explicitly reflected in the contracts between subsystems. In case of an error occuring between sending and marking, message will be sent second time (in the next batch), which is acceptable. No distributed transactions needed.

Tradeoffs

Starting a green-field project is the best time to consider abandoning distributed transactions technique. Situation is less obvious when reality imposes many constrains. Following questions may be helpful in making a decision in this matter.

Are all ACID properties needed accross distributed resources?
What is the probability of an error in the second phase of 2PC?
Is an inconsistency after an error in 2PC acceptable?
What is the cost of manually removing such an inconsistency?
What is the cost of automatically removing such an inconsistency?
What are the requirements for availability?
What are the requirements for scalability?
How much of operations consist of communication with legacy systems?

Very often the most significant factor will be the time-cost comparison between distributed-transaction-based solution (containing cost of hardware) and solution with transactions inside contracts.

Dedicated solutions are often better than general ones, but since they are dedicated, they may be not compatible with legacy systems. Much may depend on company profile and policies. Start-ups must be innovative and will tend towards dedicated solutions. Corporations seek to standardize things, so they prefer vastly used and generally accepted technology.

References

"Distributed Transactions are Evil" and "Avoiding Distributed Transactions" pages on Cunningham & Cunningham wiki touch many problems connected with the topic.
Gregor Hohpe (ToughWorks) wrote a paper titled "Your Coffee Shop Doesn’t Use Two-Phase Commit", in which he states that real-world workflows are often asynchronuous, concurrent and simple. He suggests that information systems should be modeled the same way.
In white paper titled "Life beyond Distributed Transactions" Pat Helland (Amazon) describes the impact that distributed transactions have on scalability.
Kevin Hoffman, in his article on distributed transactions in microservice environment, suggests that the need of distributed transactions may indicate an violation of SRP.
In his article on 2PC, Dan Pritchett writes about how distributed transactions affect performance and availability. He also provides a simple alternative.
On his personal website, Martin Fowler (ToughWorks) wrote a short essay about transactionlessness in eBay's architecture. References are worth checking out.

Annotations

^{1} More advanced commitment protocols are out of scope of this document. They share many of the disadvantages listed in here.

^{2} With exception of data models designed for no contention. Shared Nothing architectural pattern satisfies non-functional requirement of no contention.

^{4} Term ‘transaction’ (not distributed transaction) is used throughout this article in traditional (non-technical) meaning — an execution of a deal.

^{5} Probability of error during second phase is extremely low (years may pass before one happens). Commit errors are often ignored by software developers (data integrity must be fixed manually or restored from backup after the incident).

Go to SPA or Just REST?

2015-11-08

Single Page Application (SPA) is an old, yet only lately popular way of creating programs for the web. The idea is to fetch most of the resources from the server when user loads a website and then, as the user interacts with the site, fetch and store data via calls to an API endpoint (typically REST). In effect, page is never reloaded, wich makes the expecience more smooth than with traditional webpage. This article highlights main characteristics of SPA and specifies conditions under which the concept is preferably applicable.

With recent advent of client-side frameworks (AngularJS, EmberJS, MeteorJS, ReactJS), SPA gained much more attention in the community. New frameworks provide new abstractions that force good client-side programming techniques. Creating rich web clients is now much easier and faster. Despite the benefits that come with new frameworks, SPA isn't always considered the right way to write software for the web and not all new web projects end up being SPA.

Pros

Application can be usable when offline. An example of popular SPA, that is connection-aware, is GMail. When user goes offline, functionality is very limited, but the app is still usable. This makes the client fault-tolerant. Things can be taken one step further. Since HTML5 brings multiple client-side storage technology to the table, server-side storage may be used only as backup and synchronization medium¹.
Application can respond very fast after each click. According to research done at Google, user feels that application is fast and responsive only if it gives him a feedback after less than 200 ms. If operation lasts longer (potentially), an indication of operation in progress should appear (e.g. spinner or progress bar). These requirements may be hard to achieve with traditional server-side HTML rendering. SPA makes it much easier.
API is a well defined boundary that slows down software entropy. Sole fact that there is an API to be designed forces a very good practice of actually designing the software. Lack of well defined boundaries may (in time) lower maintainability of a system, because of accidental inter-dependencies being added.
Moving presentation logic to the client reduces the need for server resources. Total cost of the system may be reduced significantly, depending on the technology at use and size of the part that handles presentation³. This point gains weight with rescent popularization of many IaaS solutions, where payments are directly dependent on resources at use.
Modern ways of creating sofware increase morale. New things are more exciting and give people a space to grow. This can be a big factor affecting team's productivity.

Cons

Multiple programming languages used in one project. Either people of two skillsets (client and server) or more experienced (full-stack) developers are needed⁴ when building SPA. To alleciate this problem, single-language technology stacks are being developed. We see successful expansions of client-side languages to the server (Node for JavaScript) and vice-versa (GWT for Java, Opal for Ruby, pyJS for Python, GopherJS for Go).
Indexer robots do not run JavaScript. As of 4^th quarter of 2015 Google crawler is able to interpret JavaScript and CSS just like modern web browsers, but if some pages from SPA website are to be posted on Facebook, authors must implement mechanisms that provide the same content in another form, which can be read by Facebook's robot⁵ ⁶.
Extensive use of JavaScript can lead to memory leaks. It is especially apparent in older versions of Internet Explorer, where there are two memory managers (one for DOM elements and one for JavaScript objects). Many of moders JavaScript libraries contain IE leak-protection code, but expert knowledge of the matter may also be required in the team. Page reload frees all memory associated with it, thus problem is circumvented in non-SPA websites.
Coding in JavaScript requires much discipline, as the language contain many features that are considered bad programming practices⁷ ⁸. What's more, JavaScript is untyped⁹, which makes it more prone to mistakes made by the programmer. Multiple linting solutions (jslint, jshint, eslint) are being developed by the community in order to mitigate these problems.

When to use SPA?

Following conditions can be considered as favorable for SPA:

Required high responsiveness,
Required connection awareness of the client,
Heterogenous client applications,
Code base estimated to be large (> 100kLOC),
Experienced development team.

Under following conditions SPA may be not advisable:

Limited knowledge of JavaScript language and browser technology,
Web-app without additional client applications,
Required compatibility with very old browsers.

References

Jose Maria Arranz Santamaria (ItsNat Java Web Framework) wrote a document titled The Single Page Interface Manifesto, where he advises on how to transition from server-side thinking to modern SPA.
Personal website of Douglas Crockford (member of ES standards committee) contains article on causes of memory leaks in IE and how to avoid them.
Report on research of client-side template engines written by Veena Basavaraj can be found on LinkedIn's engineering blog.

Annotations

^{1} The concept of client-side web application that provide functionality on its own (without the assistance of a server) is often called Rich Internet Application.

^{3} I participated in a project in which half of server-side resources were consumed by presentation logic and session storage (JSF was used extensively).

^{4} Multiple programming languages in one project can be considered the right approach. Experienced programmers often seek to use the right tool for the job.

^{5} Prerender.io provides serverside components that prerenders HTML pages in headless browser. Rendered HTML can be sent to robots visiting the website.

^{6} Facebook's robot fetches the content of first URL written in a post. To test if Facebook's robot runs JavaScript paste http://softwarephilosophy.ninja/js-test in the post.

^{7} EcmaScript standard version 5 instroduced Strict Mode, which changes the semantics of JavaScript by removing many unwanted behaviours.

^{8} EcmaScript standard version 6 resolves many semantical problems that were present in previous versions of the language.

^{9} A super-set of JavaScript called TypeScript (created by Microsoft) adds type declarations to the syntax and type inference to the semantics of the language.

SoftwarePhilosophy

Literate programming really works for tutorials

Background

Research

Solution

Configuring Writ

NOTE

Running Writ

Wrap Up

References

Compile-Time Unit Testing

NOTE

Static Binding

Constant Expressions

Meta-Functions

Debugging

Rationale

Summary

References

Annotations

Why do we learn next programming languages?

Mindful Engineering

Efficient Learning

More Contribution

Better Career

Following Gurus

Having Doubts

Choosing The Next Language

Making a Decision

References

Annotations

Distributed Transactions

Why Not?

Transaction4 Inside Contract

Tradeoffs

References

Annotations

Go to SPA or Just REST?

Pros

Cons

When to use SPA?

References

Annotations

Transaction⁴ Inside Contract