CHRISBERKHOUT.COM

Printing plain text

2017-06-15T00:00:00+00:00

Plain text formats are great for working with on screens. They’re easy to generate and manipulate, and often presented with basic styling to highlight their structure.

For example, the output of the tree command shows a directory tree, with box drawing characters indicating the hierarchy, directory names in bold, and different colors for different file types.

Generating a printable equivalent of such things can be tricky. Let’s take a look at the options.

Problematic options

Text editors will sometimes let you print, but generally without much control over how the output is styled.
Screenshots let you capture what exactly what’s on screen, however, as relatively low-resolution raster data the text won’t be very sharp, and will be a real problem if you want to adjust the size. The on-screen style may not be suited to to a medium that isn’t backlit.
LaTeX involves many concepts beyond plain text and basic styling. Use it for native print or multi-format documents, not for a paper equivalent of the plain text screen experience.
Pandoc converts between many markup formats, but is focussed on structured document formats rather than free-form text. Document semantics determine default styling but anything more needs to be applied separately.
unoconv converts between office document formats, including from plain text to ODF or PDF. You can specify a template document, but you’d need to open LibreOffice and apply manual styles to highlight particular parts of your content.
cupsfilter converts plain text to a PDF, but offers no styling options.
a2ps will convert plain text to PostScript, but it’s styling options are only for very basic syntax highlighting of code, and it doesn’t support Unicode.
Enscript can convert plain text to PostScript. It has a lot of options, can do automatic syntax highlighting of many programming languages, and lets you add your own basic styling using special escape sequences. Unfortunately it doesn’t support Unicode.
paps version 0.6.8 is widely packaged and often recommended as a UTF-8 compatible alternative to Enscript and a2ps. It allows basic styling via command-line options and Pango markup. There are a couple of issues with version 0.6.8. It’s support of Pango markup is limited: bold works but colors don’t. Also, rather than putting actual text into the output document, it renders text using its own mechanism, which results in large file sizes and unselectable text.

A lightweight solution

Fortunately, in 2015 the paps author released a new version on Github: paps 0.7.0. It now uses Cairo for its rendering and generates compact text-based files. It offers good basic styling options via a more complete implementation of Pango markup.

If you’re generating your own data, it’s easy to apply basic styling with Pango markup.

If you want to print some data that’s already already styled for the terminal, you can use Ansifilter to convert terminal control codes to Pango markup (or one of several other markup formats). You can remap the 16-color palette to your own set of RGB values.

This command converts the output of tree to a printable PDF with styling intact:

tree -C | ansifilter --pango | paps --markup --format=pdf > tree.pdf

For printing programming languages, there are several ways to apply syntax highlighting with terminal control codes, which you can then feed into Ansifilter. I like the pygmentize command from the Pygments project.

Heavyweights: HTML and CSS

To go beyond basic styling, you need to commit to a heavier-weight styling language, but you don’t have to write raw PostScript. You can use HTML and CSS.

First, generate some HTML yourself, or take HTML output from another tool. If your data can represented in a plain-text markup language like Markdown, reStructuredText or AsciiDoc, a tool like Pandoc can convert it to HTML, to which you can apply your own stylesheet.

CSS gives you a lot of control over presentation, including some features made specifically for print design, although you will need to check which features are implemented by your renderer.

You can manually print from a web browser, but there are also several tools for converting an HTML/CSS document to a PDF non-interactively. WeasyPrint focusses on CSS layout for paged media. PhantomJS and wkhtmltopdf can both produce PDFs using versions of the WebKit rendering engine.

A simple webhook with Netcat and systemd

2017-05-29T00:00:00+00:00

A webhook is a little program that listens for web requests and triggers a given action when it receives one. It lets you use simple web protocols and tools to trigger an action on a certain system, from anywhere.

Unix-like systems make it easy to string together commands to perform just about any action you’d like. Even a webhook itself can be implemented with a one-line shell script and some system config.

In the following example I’ll show the webhook that I run on my GNU/Linux home server so I can restart Kodi (the free and open-source media center software) from a button on my phone.

Handling a request

With GNU Netcat¹ we can write a shell command to wait for a single web request, answer it, and restart Kodi:

echo -e "HTTP/1.1 204 No Content\r\nConnection: close\r\n\r" | \
  nc -l 0.0.0.0 -p 7777 -c; \
sudo systemctl restart kodi

The nc command here waits until it receives a connection on port 7777, on any interface. Once a connection is established, it reads the data we piped into its standard input and sends it over the network to the client. When it’s reached the end of the input data, it closes (-c) the connection and exits.

The data we send here is just enough to gracefully² close a HTTP connection:

HTTP/1.1 204 No Content
Connection: close

Handling successive requests

Many GNU/Linux distributions use systemd to manage start-up and system services.

Here’s a systemd service definition that will run our shell command, and restart it after it finishes handling each request. I put this in /usr/lib/systemd/system/kodi-restart.service:

[Unit]
Description=Kodi Restart webhook
After=network.target

[Service]
User=http
Type=simple
ExecStart=/bin/bash -xc 'echo -e "HTTP/1.1 204 No Content\\r\\nConnection: close\\r\\n\\r" | nc -l 0.0.0.0 -p 7777 -c; sudo systemctl restart kodi'
Restart=always
StartLimitInterval=1min
StartLimitBurst=60

[Install]
WantedBy=multi-user.target

Note that the newline escaping (\\r\\n) now has double slashes: one for Bash, one for systemd.

The command is run as the http user (more on this later). The Type=simple says that we’re just executing a normal command, not one that launces a separate background processes. We want to Restart=always, whenever the command exits. The StartLimit options say that the command can be restarted up to 60 times per minute before the service is marked as failed.

With the service definition in place, we can start the service:

$ sudo systemctl start kodi-restart

And check that it’s running correctly:

$ systemctl status kodi-restart
● kodi-restart.service - Kodi Restart webhook
   Loaded: loaded (/usr/lib/systemd/system/kodi-restart.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-05-30 22:38:09 CEST; 11s ago
 Main PID: 24341 (bash)
   CGroup: /system.slice/kodi-restart.service
           ├─24341 /bin/bash -xc echo -e "HTTP/1.1 204 No Content\r\nConnection: close\r\n\r" | nc -l 0.0.0.0 -p 7777 -c; sudo systemctl restart kodi
           └─24343 nc -l 0.0.0.0 -p 7777 -c

May 30 22:38:09 hostname systemd[1]: Started Kodi Restart webhook.
May 30 22:38:09 hostname bash[24341]: + echo -e 'HTTP/1.1 204 No Content\r\nConnection: close\r\n\r'
May 30 22:38:09 hostname bash[24341]: + nc -l 0.0.0.0 -p 7777 -c

To have it automatically start after rebooting the system, we can enable it:

$ sudo systemctl enable kodi-restart
Created symlink /etc/systemd/system/multi-user.target.wants/kodi-restart.service → /usr/lib/systemd/system/kodi-restart.service.

Granting permissions

Our new service runs as the http user, but that user doesn’t yet have permission to run the restart command.

We can grant the appropriate³ permissions with the following sudoers configuration, in a new file named /etc/sudoers.d/http-kodi-restart:

http ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart kodi

Remote control

This webhook can be triggered from any web browser, but a helper app makes for a nicer remote control experience.

HTTP Shortcuts is a simple, open-source (MIT) Android app that lets you create shortcuts for web requests, and place buttons on your home screen to trigger them.

Here’s the one I set up for restarting Kodi:

Conclusion

A one-line shell script wrapped in a systemd service definition is a good way to expose webhooks for triggering a few simple actions within a trusted network.

More demanding scenarios may call for validation or processing of request data, authentication and authorization, handling of concurrent requests, or TLS encryption. In those cases you’ll want a more sophisticated tool.

The webhook project is an open-source (MIT) tool, written in Go, that might be a good fit.

There are quite a few Netcat implementations. The original implementation was rewritten to add IPv6 support for BSDs. GNU Netcat is a full rewrite. Ncat is an implementation included in the Nmap suite. Each one implements similar functionality, but has its own set of options. Here I’m using GNU Netcat. ↩
This response, when delivered completely, is enough to gracefully respond to the HTTP request. However, sometimes the client will receive an incomplete response:

Kodi Restart failed: sendto failed: EPIPE (Broken pipe)

The problem seems to be a Netcat issue that others have also observed:

in the past I’ve been burnt by race conditions where the connection is closed before the last few bytes to arrive on stdin have actually been transmitted over it

In the GNU Netcat source code, the connection is closed by a shutdown(), followed immedately by a close(). Some people recommend waiting for a read of size zero between the shutdown() and close() calls, so perhaps that would resolve the issue.

Another way to have the connection closed cleanly would be to leave off the -c option, so that Netcat keeps the connection open until it is closed by the client, which it will only do after it has read the full HTTP 204 response. ↩
With this setup, anyone who can make a request to the right port can restart Kodi up to 60 times per minute. I’m happy to expose this functionality within my home network, but in other environments additional security measures would be appropriate. ↩

Maths learning

2017-01-20T00:00:00+00:00

The following is a list of resources for reviewing maths, starting from the beginning and covering undergraduate-level calculus, linear algebra, probability and statistics.

Other lists and advice

Best Math Books – A Comprehensive Reading List (StumblingRobot.com)
This guide’s recommendations are consistent with the advice I found elsewhere. It describes and compares several of the best books on each topic. It’s aimed at people interested in pure maths, but the basics section seems very relevant to my interests, and it intentionally chooses challenging texts.
Chicago undergraduate mathematics bibliography
This well-respected list is the one I saw most commonly referenced. Its comments are briefer, but come from several contributors and cover many topics and books.
Programming books you might want to consider reading (Dan Luu)
This list of programming books includes a math section. Dan Luu also has an interesting post asking How much math do programmers need?
What are some good books available for learning probability and statistics? (Quora)
Several good answers.
Recommended books (UC Berkeley Statistics Graduate Student Association)
A list of books in many categories, without much commentary.

School maths

The World of Math (Khan Academy)
Khan Academy covers all of school maths and a little more. It starts from counting and goes up to calculus, differential equations and linear algebra, with short video lectures and comprehensive online exercises. The explanations are very clear, but the format makes it slower than other resources for refreshing prior knowledge.
Basic Mathematics (Lang)
This book is aimed at high school students or beginning university students and includes everything needed to prepare a student for going on to calculus, linear algebra and other topics. It makes a point of including some vector geometry earlier than is commonly done and “deals with mathematics on both a manipulative (or computational) level and the theoretical level.”
Engineering Mathematics (Stroud & Booth)
This was highly recommended by a colleague, for building practical maths skills from the beginning. It stops at Fourier transforms (covered in Advanced Engineering Mathematics) and focuses on practice rather than theory, lacking formal definitions. It is designed for self study, with checklists and exercises included.

Mathematical thinking and proof writing

How to Study as a Mathematics Major (Alcock)
A useful and easy to read explanation of what university-level mathematics is about and how to approach studying it. It goes a deeper than Devlin in the discussion, and refers users to other resources (including Velleman) for practice of the proof-writing techniques it introduces. The general study advice won’t be of much interest unless you’re really fresh out of high-school.
How to Prove It: A Structured Approach (Velleman)
The most widely recommended book for learning how to work with proofs.
Introduction to Mathematical Thinking (Devlin)
This book starts with an interesting discussion of how modern mathematics differs from its past, and from what you might have learned in school. The second half introduces proofs and assumes you’re following along with all the exercises. My preference is the previous two books, but this is a good alternative for both. This material is also covered by the author in a Coursera course.

Calculus

I found a particularly helpful Mathematics Stack Exchange answer comparing Apostol and Spivak. Apostol covers integration before differentiation, introduces linear algebra before multi-variable calculus, and has been particularly recommended for older students coming back to mathematics and as the best calculus book for building mathematical maturity.

Calculus (Apostol) Vol. 1
One-Variable Calculus, with an Introduction to Linear Algebra.
Calculus (Apostol) Vol. 2
Multi-Variable Calculus and Linear Algebra with Applications to Differential Equations and Probability.

Linear algebra

Linear Algebra (Hoffman & Kunze)
Apparently covers a lot, and has very good reviews on Amazon and elsewhere.
Linear Algebra Done Right (Axler)
A popular alternative, also recommended by a colleague.

Probability

These two (and Wasserman, below) were recommended in Machine Learning: a Probabilistic Perspective:

Statistics

Statistics (Freedman et al.)
Highly recommended in reviews and by a colleague. Focuses on fundamental ideas and their application rather than equations or proofs.
All of Statistics: A Concice Course in Statistical Inference (Wasserman)
From the publisher: “This book covers a much wider range of topics than a typical introductory text on mathematical statistics. It includes modern topics like nonparametric curve estimation, bootstrapping and classification, topics that are usually relegated to follow-up courses. The reader is assumed to know calculus and a little linear algebra. No previous knowledge of probability and statistics is required. The text can be used at the advanced undergraduate and graduate level.”

The course Basic Statistics (Coursera) is nice and clear, but aimed at humanities students.

Surveys

After the topics above, it might be nice to take a broader look at mathmatics. These two books are often recommended for that purpose.

Mathematics: Its Content, Methods and Meaning (Aleksandrov et al.)
I found this survey by Soviet mathematicians via a highly upvoted Mathematics Stack Exchange answer, and noticed it recommended in several other places.
What is Mathematics? An Elementry Approach to Ideas and Methods (Courant)
Einstein called it “a lucid representation of the fundamental concepts and methods of the whole field of mathematics.”

Other resources

There are a lot of online courses on these topics. Here is a list of many of them:

19 MOOCs on Mathematics & Statistics for Data Science & Machine Learning

I was quite inspired by reading about Scott Young’s MIT challenge, in which he went through 4 years worth of MIT computer science materials in 1 year. MIT has a lot of course materials freely available online, especially in computer science and mathematics. Sometimes it’s video lectures, other times it’s written notes, of varying detail. It’s a great resource.

Log tailing tips

2016-12-14T00:00:00+00:00

The following examples assume you can tail live logs as follows:

tail -f service.log

That command will show lines being added to the file service.log. Replace it with whatever command will stream logs in your environment.

Although many unix tools (head, wc, sort, xargs, etc.) are available with the same name and similar functionality on both BSD-based platforms (such as Mac OS X) and GNU-based platforms (such as Linux), they often have small differences. All of the examples here assume GNU tools. If you are not sure which version of a tool you have on your system, or what options are available for it, check its man page.

Show only the first 5k new error lines

tail -f service.log | grep ERROR | head -n5000

Lines not including ‘SUCCESS’ or ‘200 OK’ or backtraces

tail -f service.log | grep -v SUCCESS | grep -v '200 OK' | grep -Pv '\tat '

Grep’s -v will print lines that don’t match the pattern. -P is a Perl-style regular expression (used here to indicate a tab character).

Show the 3 lines before and 10 lines after each exception

tail -f service.log | grep -B3 -A10 Exception

Grep’s -B says how many lines before the matching line to print, -A is for lines after. Blocks of matching input will be separated by lines with --.

Watch the rate of a certain exception

tail -f service.log | grep IndividualRequestTimeoutException | \
  pv -lra > /dev/null

pv is Pipe Viewer. It is commonly used to show a progress bar or counter when writing or transferring data using pipes in the shell. In this case we specify -l (line mode - counting newline characters instead of bytes), -r (display the current rate of data transfer) and -a (display the average rate of data transferred so far). pv echos its STDIN to STDOUT (often to another program), but in this case we just want to see the rates and not keep the data, so we redirect the STDOUT to /dev/null to discard it.

Show 10k log lines and save them in a file

tail -f service.log | head -n10000 | tee captured.log

tee lets you write input into a file for later use, but also to see the data as it passes through. In this case the file will be overwritten if it already exists.

Show 10k log lines and append them to a file

tail -f service.log | head -n10000 | tee -a captured.log

You can append rather than overwrite by adding -a.

List instances in captured logs

Assuming each log line starts with an instance ID:

cat captured.log | cut -f1 | uniq

cat simply prints the contents of a file. cut lets you pick out a certain field from each line of input data. By default, fields are delimited by tabs, and we use that to select the instance ID (the first field) from each line. You can specify a different field delimiter with -d. For example, a space: -d' '. uniq prints only unique lines of input (removing repeat lines).

Count instances in captured logs

cat captured.log | cut -f1 | uniq | wc -l

wc is the word counter. -l tells it to just count lines.

Group captured logs by instance, in one file

cat captured.log | sort -sk 1,1 | tee grouped.log

sort will sort lines. -k tells it to sort based on certain (whitespace separated) fields. In this case we want to sort using field 1 to 1 (so, only 1 - the instance ID) (note that if the ending field is unspecified it defaults to the end of the line). Usually when two lines have equal sort fields, the whole lines will also be compared to determine the final order, but -s specifies ‘stable sort’, which preserves the incoming order for lines with equal sort keys.

Group captured logs by instance, in separate files

cat captured.log | cut -f1 | uniq | \
  xargs -n1 -I{} bash -c 'grep ^{} captured.log > instance-{}.log'

xargs transforms lines in its STDIN to arguments to another command. By default, multiple input lines will be converted to multiple arguments to a single command. In this case we specify -n1 to use just one input line at a time. -I{} says that in the command, {} should be replaced with the input string. We use grep ^{} captured.log > instance-{}.log to find the lines in captured.log that begin with the given instance ID and write them to a file with the instance ID in the name. Since this this command involves a shell redirect (>), it needs to be run in a shell rather than as a direct command, so we wrap it in bash -c '...'.

Count log lines in each instance file, sort ascending

wc -l instance-* | LC_ALL=C sort

In most locales, by default, sort will consider whitespace insignificant. This means that a 2 will be ordered after 123, because the first significant character in the former (2) comes after the first significant character in the latter (1). We can specify the C locale in an environment variable to force sort to consider whitespace significant and sort 2 before 123. Note that here we are asking wc to count multiple files rather than the STDIN data (as we did above).

Log visualization with gnuplot

2016-07-04T00:00:00+00:00

Often when investigating an issue there is information in logs that isn’t captured well by Prometheus metrics. This includes qualitative information such as stack traces, but the frequency of certain messages may also be of interest and that is what this we will focus on here.

Selecting interesting log messages

A recent issue showed memcached exceptions in the Prometheus metric serviceapi_memcached_exceptions_total (including the exceptions IndividualRequestTimeoutException and KetamaFailureAccrualFactory$$anon$1), but the logs also included messages such as:

2016-04-13_18:09:17.87161 18:09:17.871 [finagle/netty3-9] ERROR SERVICE_API_THRIFT - traceId [2ac36e44bf4c0a96.2ac36e44bf4c0a96<:2ac36e44bf4c0a96] [0 ms] updateByActor -> FAILURE
2016-04-13_18:09:17.87162 java.lang.Exception: Failed adding record to memcached: 'Target(218468078)'
2016-04-13_18:09:17.87162 at com.example.service.record.RecordRepository.com$example$service$record$RecordRepository$$$anonfun$15(RecordRepository.scala:102)
2016-04-13_18:09:17.87163 at com.example.service.record.RecordRepository$lambda$$addToRemoteCache$1.apply(RecordRepository.scala:100)
2016-04-13_18:09:17.87164 at com.example.service.record.RecordRepository$lambda$$addToRemoteCache$1.apply(RecordRepository.scala:100)

which could more easily be tied to a specific code path.

I could see these messages appearing throughout the incident, but I wanted to get a sense of the frequency and at what stages of the incident they appeared.

After retrieving some logs from the relevant instance, I identified a few interesting subsets of log messages:

grep 'MEMCACHED set failed with .*IndividualRequestTimeoutException'
grep 'WARN.*Endpoint is marked dead by failureAccrual'
grep 'Failed adding record to memcached'

Extracting times & generating counts per second

The strptime utility from dateutils can parse and convert timestamps. By default it discards non-matching portions, so piping in log lines beginning with timestamps of the form:

2016-04-13_18:09:17...

and specifying the input and output formats to be %F_%T and %s%n generated output lines with a Unix timestamp only, in seconds since the epoch.

The next step in processing was to count how many times each second-resolution timestamp occurred, using uniq‘s -c (count) option.

The full data preparation process was:

cat logs.txt | grep 'ERROR' | strptime -i '%F_%T' -f '%s%n' | uniq -c > error.dat
cat logs.txt | grep 'MEMCACHED set failed with .*IndividualRequestTimeoutException' | strptime -i '%F_%T' -f '%s%n' | uniq -c > memcached-set-failed.dat
cat logs.txt | grep 'WARN.*Endpoint is marked dead by failureAccrual' | strptime -i '%F_%T' -f '%s%n' | uniq -c > dead-by-failureaccrual.dat
cat logs.txt | grep 'Failed adding record to memcached' | strptime -i '%F_%T' -f '%s%n' | uniq -c > fail-adding.dat

Each of the four generated files was in this format (count, timestamp):

685 1460570400
483 1460570401
953 1460570402
587 1460570403
694 1460570404
...

Plotting with gnuplot

gnuplot is an open source plotting tool, first released 1986. It is a command line program that is useful both for interactive data analysis and scripted presentation of final graphics in many different output formats.

I like to build up a gnuplot script by running a text editor in one window and sending commands from there into an interactive gnuplot terminal in another window. If your editor isn’t set up for that (e.g. with tslime.vim), you can always use cut and paste or tell gnuplot to load your script from a file (load 'yourfile.gnuplot').

I began with these gnuplot commands:

set xdata time
set timefmt "%s"
set format x "%H:%M"
set yrange [0:*]
plot 'error.dat' using 2:1 title 'error' with impulses lw 0.1, \
     'memcached-set-failed.dat' using 2:1 title 'memcached-set-failed' with impulses lw 1, \
     'dead-by-failureaccrual.dat' using 2:1 title 'dead-by-failureaccrual' with impulses lw 1, \
     'fail-adding.dat' using 2:1 title 'fail-adding' with impulses lw 2

Which generated this plot:

Next I restricted the range of the y-axis and refreshed the plot:

set yrange [0:500]
refresh

And again:

set yrange [0:100]
refresh

Here it is clear that IndividualRequestTimeoutExceptions were only present during an initial 30s period, after which Finagle’s Endpoint is marked dead by failureAccrual warnings began, with a much greater frequency than our own Failed adding record to memcached messages. This indicated that the finagle-memcached client’s internal retries were responsible for a large proportion of the exceptions, and the finally failed incoming requests were actually much fewer. These error remained stable from 18:10 to 18:20 (despite the serviceapi_finagle_memcached_client_dead_nodes metric in Prometheus showing recovery of all dead nodes by 18:10).

In this chart, the x-axis range is set automatically by the range of the input data, and gnuplot’s defaults are used for tick spacing, color selection and legend formatting, although it is possible to control all this and more if required.

Let’s take a closer look at the plot command:

plot 'error.dat' using 2:1 title 'error' with impulses lw 0.1

part	meaning
`plot`	The gnuplot command
`'error.dat'`	Our data file, with space separated values (by default)
`using 2:1`	Use columns 2 (time) and 1 (count) for x and y values. Column numbers always start from 1
`title 'error'`	The title for this series, to be displayed in the legend
`with impulses`	Use the `impulses` plotting style for this data
`lw 0.1`	`lw` is the abbreviated form of `linewidth`. Many gnuplot options have abbreviations

When multiple data sets are to be plotted, they should be separated by commas. Note that the different series of data are drawn in the order they are specified in the plot command, so it is important to find an order that does not obscure significant features of the data. I also chose different line widths in order to improve the visibility of certain features.

Plotting style choice

In this example we used the impulses plotting style, which draws a line from y=0 up to the value of each data point. This is very simple and quite readable, and it’s what I recommend for plotting frequencies from log data.

The lines style would plot data from the same files, have the benefit of not drawing over the top of smaller values, and perhaps even look a little nicer, but it is complicated by interpolation. If there is no data point for a given x coordinate, it will be interpolated from the values of the points to either side of it, which means that times for which there was no count observed will not display as zero. Additional pre-processing could be done to insert zero values for any missing x coordinate in the data range, but for ad hoc analysis my preference is to just use impulses.

Another alternative would be to draw a histogram using the boxes plotting style, but this requires additional logic for assigning data points to different buckets.

Lower resolution counts

In this example we generated counts per second for the different log messages. To have wider counts, for example, per minute, we can use dateutil’s dateround instead of strptime and round the timestamps as we parse them:

cat logs.txt | grep ERROR | dateround -i '%F_%T' -f '%s%n' /15s | uniq -c > errors-per-quarter-minute.dat
cat logs.txt | grep ERROR | dateround -i '%F_%T' -f '%s%n' /1m | uniq -c > errors-per-minute.dat

Dealing with localStorage errors

2013-08-30T00:00:00+00:00

Web storage is pretty straightforward, but several otherwise helpful tutorials I went through brushed over error handling:

Dive Into HTML5: Local Storage

“QUOTA_EXCEEDED_ERR” is the exception that will get thrown if you exceed your storage quota of 5 megabytes.

PaperKilledRock.com: HTML5 localStorage

Due to browser specific quotas you would want to check for exceptions so you would change line 1 above to the following.

try {
  localStorage.setItem("name", "Hello World!"); //saves to the database, "key", "value"
} catch (e) {
  if (e == QUOTA_EXCEEDED_ERR) {
    alert('Quota exceeded!'); //data wasn't successfully saved due to quota exceed so throw an error
  }
}

Is QUOTA_EXCEEDED_ERR some kind of global constant?

Chrome Developer tools says: ReferenceError: QUOTA_EXCEEDED_ERR is not defined.

Google suggests it might actually be DOMException.QUOTA_EXCEEDED_ERR, and Chrome agrees - DOMException.QUOTA_EXCEEDED_ERR is 22.

Is there a certain attribute of the exception that we should be matching to error code 22?

Let’s take a look. To generate the QUOTA_EXCEEDED_ERR exception we’ll attempt to store a 12MB string, which should exceed the quota. Open your developer tools console and enter:

var exception;
var repeat = function(str, x) { return new Array(x+1).join(str); };
var too_big = repeat("x", 12*1024*1024/2); // each JS character is 2 bytes
localStorage.clear();
try {
  localStorage.setItem("test", too_big);
} catch (e) {
  exception = e;
}

Then inspect exception.

Trying that in a few browsers, we observe:

Chrome

Firefox

Safari

There is no single attribute that we can depend on across browsers to detect that the error is in fact a QUOTA_EXCEEDED_ERR.

We could check exception instanceof DOMException, but to test such code we would need to be able to produce a mock DOMException instance. A SitePoint reference notes:

In practise, what we find is that most browsers implement DOM exceptions as part of their native mechanism…

It’s not possible to produce them cleanly from normal JavaScript.

Conclusion

Although localStorage quota errors are known by names such as QuotaExceededError and QUOTA_EXCEEDED_ERR, the only way to write testable, non-browser specific code is to accept any exception thrown during a storage attempt as a quota error. To differentiate between quota and support/security errors, be sure to detect the availability of localStorage separately.

Mapping suburbs with jVectorMap

2012-10-16T00:00:00+00:00

At REA we have a lot of data about the Australian property market. On our last hack day we explored new ways to put it to use. I prepared some map data and examples with jVectorMap to help visualise suburb data.

jVectorMap comes with border data for the countries of the world and the states of the USA and several other countries. We needed to map suburbs in Sydney and Melbourne, so I took some data from the Australian Bureau of Statistics, selected a relevant subset using QGIS, and ran it through jVectorMap’s conversion script to produce files in its JSON format. I ended up making a few changes to the conversion script to get the results I wanted. The process is explained in more detail on GitHub.

It is good for heat maps, but jVectorMap also lets you add interactivity by providing hooks to respond to hover, enter, exit and click events.

It can zoom in and out, but since it works with a single set of vector data, it may not be the best choice for applications where your zoom range is big enough to call for data at several levels of detail. For example, data that’s detailed enough to show individual Australian suburbs looks great at the city level, but doesn’t render so well at the national level.

Several teams put it to use on hack day, with great results.

A colleague demonstrating his team’s hack day project

Clean documentation

2012-08-29T00:00:00+00:00

I spoke about this at Eurucamp 2012 in Berlin.

My goal here is not to give you a step by step guide to writing a readme. Instead, I want give you a few ideas to think about the next time you consider writing documentation or actually end up doing it.

What do I mean when I say Clean Docs? There’s an explanation of clean code in Uncle Bob’s Clean Code book, but even if you haven’t read that, I’m sure you can tell the difference between code that’s clean and code that’s dirty. We have rules in our heads about that and alarm bells go off when we start writing code that’s dirty and going to cost us later on. I want to encourage you to develop that same kind of awareness and discipline for when you’re writing documentation.

One of the Agile principles states: ‘We have come to value working software over comprehensive documentation’. When you read that it’s easy to get a feeling of relief and think ‘okay, great! I don’t have to worry about documentation. I’m just going to make software that works.’ But of course that’s not quite what it’s saying. It’s actually saying that comprehensive documentation has value, but that working software has more. I agree. However, I believe this is referring largly to project documentation - things like requirements documents and design plans - and that’s not what I’m talking about here. Also, working software is good but I suggest we aim even higher and create usable software. Documentation can help you achieve that.

There are many kinds of documentation. One way to break them down is by proximity to code. We can start with the actual code itself, which can serve as a form of documentation if it is clear enough, and move through technical documentation, project websites and general documentation, all the way to published books. What I have to say about Clean Docs applies to the forms of documentation that are closer to the code. The readme is right in the middle of that range but the rest is relevan as well.

It has been said that ‘good code is its own best documentation’. That’s very true in a sense, but unfortunately it’s become the biggest excuse for developers to completely ignore documentation. Without digging too deep - just by rereading that statement - we can see that code being its own best documentation is dependent on the code being good. Beyond that immediate observation I would add that the relevance of code as documentation depends on what the user is trying to get out of it. Even the best code will have a hard time explaining the purpose and scope of your project in the first few seconds that a new user looks at it, but that’s easily acheivable in a couple of sentences of English.

The users (including the maintainers) of your documentation will vary in their proximity to the code and their proximity to the present time. You should consider users across this whole plane when writing documentation. Writing more will help users unfamiliar with the code today, but require more work in maintenance later. Writing less will save on writing time, but may make it harder to maintain a clear scope, attract new users, and remember what the project is all about two years from now. Think about who your documentation is serving, and who is it costing, and when.

Without too much rewording, the principles of clean code listed above also all apply to documentation.

Take particular notice of the last point. A lot of people facing the prospect of writing documentation fear that it’s going to take a long time and it’s going to be difficult to maintain. The more you write, the more you really have to fear. So, minimise it.

You can minimise what you write by referencing things rather than reproducing them. For example, if your API documentation contains an explanation of some concept that readers of your readme will need in order to understand discussion there, just refer them to it, rather than reproducing the explanation in a second location.

What is brittle documentation? It’s the same a brittle code, it’s something that’s going to break as soon as you touch it.

For example, I had some code in which we had an API key. Let’s put aside the fact that it probably would have been better as configuration than as code. The immediate problem I had was that there had been confusion about whether this was a company key or a team key. I clarified that for myself and then I wanted to make it clear to future readers of the code.

I thought that I could use a more explicit variable name or alternatively I could put in a short comment that would document what the key was, but I realised that both of these solutions would be quite brittle. Someone would one day come along and change the API key. They probably wouldn’t update the comment and they almost certainly wouldn’t update a variable name.

A better solution was to come up with a form of documentation that would automatically expire if careless changes were made. In the last line above I’m saying that they key is cde0fc8a5c2b1ecc, and the key starting cde0 is the company key. That explains the situation to readers today and if the key is carelessly changed at some point, future readers will not be misled by the comment.

That code comment example is a small one, but the same principle can be used throughout your documentation. Try to write things that will tolerant of careless changes; that will make sense for as long as possible and that will be obviously irrelevant if things change in the future.

The first way to test your documentation is to test it against your code. Most of the time that just means cutting and pasting your code examples into IRB or another REPL and checking that you’ve got your syntax correct and that the examples run against the current version of your software. Sometimes it will be appropriate to automate that. If you’re writing a book, for example, you might write a script to pull lines of code out of executable files and insert them into your readable documentation. But manual checks are good enough most of the time.

You can also test against readers. A teammate will be able to read your writing and give you feedback, but the best person to check with is someone who doesn’t already know the information you’re trying to convey. Take a fresh person and see if they can understand you project by reading the documentation.

When we think about good documentation several examples come to mind. We’re generally pretty lucky in the Ruby world. The Sinatra readme is a fine example. Several people have told me they really love it and I know why. It’s well structured, it’s easy to read, it’s got appropriate code examples, you can easily search within the page, it links to external resources… it’s great! A lot of people use Sinatra, so it’s obvious that all of the effort that has been put into building up its comprehensive documentation is worthwhile.

But what about those projects that don’t seem worth documentation? Maybe it’s something that you hacked up on a weekend. You’re the only person who will ever use it and only a handful of other people might ever look at it. Even those projects deserve a readme.

There are some obvious things to put in: what it is, who you are and how to get started. To those I would add that you should be linking to whatever other things there are. If your project is at the point where there are additional resources that are relevant then link to them. In the example above I’ve linked to a blog post that explains what that demo app is.

Also consider adding a status section. That can be just one word, or up to a few sentences. It might just be something like ‘broken’ or ‘unmaintained’ or ‘in development’ but it could be a bit longer and say ‘this is a spike I did to learn X. It is retained only for reference.’ Having a status section helps new people looking at the project, but it also offers the benefit of taking unpolished code that’s not fit for public consumption and making it respectable, with just a few words. Also, it will make it easier for you to come back and pick up the project again if you decide to do something with it in the future.

The idea of readme driven development is that you write your readme before your code. That gives you an early chance to think about your software from the user’s perspective, and potentially, to discuss it. Of course, you continue to develop your readme and your other documentation as the implementation code grows.

Whether you write your readme before or after your code, you should definitely take the opportunity that documentation production and maintenance offers to re-examine your project; to refresh your view of how things fit together and what might come next.

You documentation should have a single starting point. Usually it will be a readme, but sometimes it will something else, like a project website. Users should be able to start there and find everything that’s relevant to the project. Don’t make Google you starting point, because although it makes it easy to find individual resources, it’s not good at communicating how they fit together and what the status is, which is exactly what your documentation should be doing.

In summary: consider the needs of various readers and writers, both now and in the future. Have a single starting point for your documentation, link to everything from there, maintain that cohesive structure and use its maintenance as a chance to keep a fresh understanding of how things fit together. Cover the basics, usually in a readme and always as briefly as possible, and keep it clean, like you would your code.

Isolated subclass specs in Ruby

2012-07-01T00:00:00+00:00

Unit tests should isolate their subject, but tests of subclass functionality usually test the class as a whole. One reason for this is that in Ruby there is no way to dynamically change the superclass, but more importantly, it’s because you don’t want to modify the thing you’re testing. Sometimes however, isolating a subclass from its parent makes sense.

Schemabot, our tool for applying database changes in a replication friendly way, is based on the Sequel gem and its migration DSL. We added a preview function by subclassing the MySQL adapter and having it ignore any SQL that would change the database.

A simplified version of our implementation (allowing just SELECT and DESCRIBE queries) looks like this:

module Sequel
  module MySQLPreview
    class Database < Sequel::MySQL::Database

      def execute(sql, opts={})
        if sql.match(/^SELECT/) || sql.match(/^DESCRIBE/)
          puts "RUNNING: #{sql}"
          super(sql, opts)
        else
          puts "PREVIEW: #{sql}"
        end
      end

    end
  end
end

Satisfied with the assumption that all queries go through #execute and the fact that Sequel itself is already well tested, we wanted to add a set of specs that would simply validate that our #execute method passed queries on to the superclass or ignored them as appropriate.

Set an expectation on #execute or on #super?

Our #execute calls the #execute in the superclass, and that action is what we want to set an expection on. RSpec lets us set message expectations on a class (ClassName.should_receive), on an instance (ClassName.new.should_receive), or all instances (ClassName.any_instance.should_receive), but the #execute we care about is an instance method on the superclass. It’s not a class method, and it’s not a method on any regular instance.

Instead of checking calls to the superclass’s #execute, it would be nice to set expectation on the call to super. Although Ruby language features are often implemented as methods (require, method_missing, etc.) super is actually a keyword. It is possible to create a method named super, but mocking such a method would be no help, as it would remain completely separate from the super keyword, and require a different, more explicit syntax to invoke.

RSpec’s David Chelimsky has commented on this issue:

Since we don’t get into the class hierarchy in rspec’s mock/stub framework, there is no facility for managing this.

So, it’s up to us to intercept the call if we want to verify that the subclass is behaving correctly.

Sneak a module into the inheritance chain

Somehow we need to get some in between the subclass and its superclass. Including a module in a class will not modify its superclass (which can’t be changed), but the module will be inserted into the class’s list of ancestors, and be able to intercept super before the actual superclass is reached.

To demonstrate this, let’s start with a simple parent class and subclass, both implementing #execute methods:

class ParentClass
  def execute
    puts "ParentClass#execute"
  end
end

class SubClass < ParentClass
  def execute
    puts "SubClass#execute"
    super
  end
end

The output of SubClass.new.execute will be:

SubClass#execute
ParentClass#execute

Including a module changes the ancestors list:

module SneakyModule
  def execute
    puts "SneakyModule#execute"
    super
  end
end

SubClass.ancestors # => [SubClass, ParentClass, Object, Kernel, BasicObject]
SubClass.instance_eval { include SneakyModule }
SubClass.ancestors # => [SubClass, SneakyModule, ParentClass, Object, Kernel, BasicObject]

And SubClass.new.execute shows the sequence of execution to be:

SubClass#execute
SneakyModule#execute
ParentClass#execute

This technique allows us to programatically verify the behaviour of the subclass’s #execute method by having the sneaky module in the middle record and report on method calls it intercepts. Here’s an example implementation:

module FakeMySQLAdapter
  class << self
    attr_accessor :last_execute
  end
  def execute(*args)
    FakeMySQLAdapter.last_execute = args
  end
end

With this done, we can write an example to verify that super is being invoked appropriately:

before(:all) { subject.class.instance_eval { include FakeMySQLAdapter } }
before(:each) { FakeMySQLAdapter.last_execute = nil }

it "should pass on SELECT statements" do
  subject.execute("SELECT * FROM foo;")
  FakeMySQLAdapter.last_execute.should == ["SELECT * FROM foo;", {}]
end

You can see a full implemenation of this on GitHub.

This is a solution. It exercises the logic of our subclass’s #execute and verifies its query handling. It’s not a great solution. One problem is that this code modifies the subclass being tested (by including the module), and another is that the subclass stays modified. It’s not possible to uninclude the module. If we had other tests that relied on the subclass, running the subclass’s specs first might break others.

There is a cleaner way, as we’ll see next.

Extract a module with the method override

Rather than overriding #execute in the subclass directly, we can override it by including a module that implements #execute. This structure allows us to include the logic of interest in a dummy class to test it in isolation.

First, we refactor our preview adapter to the following:

module Sequel
  module PreviewMySQL

    module MethodOverrides
      def execute(sql, opts={})
        if sql.match(/^SELECT/) || sql.match(/^DESCRIBE/)
          puts "RUNNING: #{sql}"
          super(sql, opts)
        else
          puts "PREVIEW: #{sql}"
        end
      end
    end

    class Database < Sequel::MySQL::Database
      include MethodOverrides
    end

  end
end

Before we test the module, let’s write an assertion that Database is actually using the module:

describe Sequel::PreviewMySQL::Database do
  it "should include the override methods" do
    subject.class.ancestors.should include(Sequel::PreviewMySQL::MethodOverrides)
  end
end

After that, we can test the module. We start by creating a dummy superclass for the module to be included in. We’ll use method_missing to record all method calls passed up to the superclass by its subclasses.

By using Class.new to create anonymous subclasses that include the module being tested, we can ensure that each example in our spec runs with a fresh copy of the subject.

describe Sequel::PreviewMySQL::MethodOverrides do

  class FakeSuperClass
    attr_reader :last_super_call
    def method_missing(method, *args)
      @last_super_call = [method, args]
    end
  end

  subject do
    Class.new(FakeSuperClass).instance_eval do
      include Sequel::PreviewMySQL::MethodOverrides
    end.new
  end

  #...
end

This solution is easier to read than the one in the previous post, and it avoids the problems of modifying the subject. Although we are now testing a module rather than the actual subclass we’ll be using it in, our assertion that the subclass should include the module of override methods completes the loop and will automatically warn us if we make changes to our hookup code.

With this in place, our individual examples are clean and simple, as follows:

it "should allow SELECTs" do
  sql = "SELECT * FROM sometable"
  subject.execute(sql)
  subject.last_super_call.should == [:execute, [sql, {}]]
end

You can see a full implemenation of this on GitHub.

This is a clean solution, and one that I’d have been satisfied with if I’d come up with it first. However, the extra time on this got me thinking… it really shouldn’t be that hard to stub out irrelevant and troublemaking parts of the full class.

Test the full class and stub problematic parts

Testing the full class really shouldn’t be hard and RSpec should point out any code that gets in the way. I gave it another go.

First, I reverted to the original implementation of the subclass:

module Sequel
  module MySQLPreview
    class Database < Sequel::MySQL::Database

      def execute(sql, opts={})
        if sql.match(/^SELECT/) || sql.match(/^DESCRIBE/)
          puts "RUNNING: #{sql}"
          super(sql, opts)
        else
          puts "PREVIEW: #{sql}"
        end
      end

    end
  end
end

And started with a simple spec:

describe Sequel::MySQLPreview::Database do
  it "should allow SELECTs" do
    sql = "SELECT * FROM sometable"
    subject.should_receive(:puts).with(/^RUNNING:/)
    subject.execute(sql)
  end
end

The assertion here is on #puts rather than the superclass’s #execute method, but we’ll add more comprehensive assertions later.

That example raised an error with a rather brief backtrace:

1) Sequel::MySQLPreview::Database should allow SELECTs
   Failure/Error: super(sql, opts)
   Sequel::DatabaseConnectionError:
     Mysql::Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
   # <internal:prelude>:10:in `synchronize'
   # ./spec/whole_spec.rb:11:in `execute'
   # ./spec/whole_spec.rb:30:in `block (2 levels) in <top (required)>'

RSpec tries to show you just the parts of the full backtrace that are relevant to the code you’re testing, but you can tell it to be more verbose by running it with the --backtrace option. That got me this:

1) Sequel::MySQLPreview::Database should allow SELECTs
   Failure/Error: super(sql, opts)
   Sequel::DatabaseConnectionError:
     Mysql::Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/adapters/mysql.rb:110:in `real_connect'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/adapters/mysql.rb:110:in `connect'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/database/misc.rb:48:in `block in initialize'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool.rb:100:in `call'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool.rb:100:in `make_new'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:144:in `make_new'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:130:in `available'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:120:in `block in acquire'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:162:in `block in sync'
   # <internal:prelude>:10:in `synchronize'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:162:in `sync'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:119:in `acquire'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/connection_pool/threaded.rb:91:in `hold'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/database/connecting.rb:229:in `synchronize'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/sequel-3.35.0/lib/sequel/adapters/shared/mysql_prepared_statements.rb:23:in `execute'
   # ./spec/whole_spec.rb:11:in `execute'
   # ./spec/whole_spec.rb:30:in `block (2 levels) in <top (required)>'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:87:in `instance_eval'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:87:in `block in run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:195:in `with_around_each_hooks'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example.rb:84:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:353:in `block in run_examples'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:349:in `map'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:349:in `run_examples'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/example_group.rb:335:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:28:in `block (2 levels) in run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:28:in `map'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:28:in `block in run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/reporter.rb:34:in `report'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/command_line.rb:25:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/runner.rb:69:in `run'
   # /Users/chrisberkhout/.rvm/gems/ruby-1.9.2-p290-patched/gems/rspec-core-2.10.1/lib/rspec/core/runner.rb:10:in `block in autorun'

I could see that a #synchronize method was responsible for acquiring a connection to the database. This was the method I needed to stub to to isolate my test from an actual database instance.

Following through the execution path showed that the superclass’s #execute would eventually call #_execute to finish its work. Setting an expectation on this method would allow me to test whether or not the SQL query was in fact being passed up to the superclass for execution.

Incorporating these two points brought me to the final form of my specs:

describe Sequel::MySQLPreview::Database do

  let(:connection) { stub("connection") }
  before { subject.stub(:synchronize).and_yield(connection) }

  it "should allow SELECTs" do
    sql = "SELECT * FROM sometable"
    subject.should_receive(:puts).with(/^RUNNING:/)
    subject.should_receive(:_execute).with(connection, sql, anything)
    subject.execute(sql)
  end

end

The two assertions here cover both the direct actions of the subclass, as well as its correct interaction with the superclass.

So, it turns out that (with --backtrace) this was quite straightforward. You can see the full code on GitHub

In conclusion

If I’d gotten this full class spec working first, I probably would have stopped there. However, even though this tests the class as a whole - the normal way - it is not necessarily the best approach.

The downside here is that it requires stubbing the class under test (even though it’s a part of the class we’re not specifically interested in testing). The other thing is that it couples itself more tightly to the internal implementation of the superclass. It requires the #execute, #synchronize and #_execute methods to behave in certain ways. It also places on the reader the burden of understanding the significance of the #_execute expectation.

The assumptions made by the isolated module spec are simpler: that all SQL goes via the #execute method and that it can be stopped there without any problems. It’s a smaller, cleaner set of assumptions, and there’s no stubbing of the subject. I’d say that extracting and testing a module in isolation is the best approach here.

Local variables - the new global variables?

2012-05-31T00:00:00+00:00

The topic of tonight’s Melbourne Ruby meetup was “things I wish I’d known earlier”.
This was my contribution.

Last year I read the Refactoring book and it gave me some surprising new insights into local variables.

If you look through the contents of the book you’ll quickly find the refactoring “Introduce explaining variable”.

Imagine you have a method that does some processing that is not particularly obvious. Here’s an example that selects a certain number of evenly spaced elements from an array:

You can make it more readable by assigning an intermediate value to a well-named local variable, like so:

There are times when that will be a great help, and you should definitely perform such refactorings.

Tonight, however, I want to focus on the more general case, which is that local variables stink.

Local variables are bad because they are often introduced as a premature optimisation.

Take this Library class for example, which can prepare a report on overdue books for the library detective. It uses the total number of overdue books twice, but it manages to only calculate it once, by storing the value in a local variable. The same is done with the total count of all books.

These are premature optimisations that don’t aid readability or have any perceptible influence on performance, yet they do add additional lines of code.

They should be removed by repeating the calculation call wherever its result is required.

The example of counting elements in an array is perhaps trivial. The mistake is easier to make with other kinds of calculations. Data on disk or across a network may require further consideration, but when it is in memory you should assume that the results of sorting, arithmetic, string manipulation and most other processing can be instantly recalculated as required.

Our #detective_report method still uses the explaining variable percent_overdue. The problem with this is that the logic that went into calculating that value is not ready for reuse. When we add a method to tell us whether late fees need to be reviewed (which they obviously should be if more than 50% of our books are overdue), we end up with duplication.

If we replace that explaining local variable with a method, we significantly improve readability as well as reducing duplication.

Local variables are bad because they hide Law of Demeter violations and other complexity.

The #can_borrow? method says a borrower can borrow a book if they don’t already have a book out and their membership record shows no fees are due.

It’s a fairly short and clear method. We just make a couple of simple method calls, and we don’t have the long method chains that Law of Demeter violations are often recognised by.

However, if we remove our local variables, the equivalent code reveals itself as ridiculously inappropriate. It is reaching deep into the record object - coupling itself to fee structures and introducing duplication of fee handling logic.

ActiveSupport’s #try method would shorten this, but it’s still a clear candidate for refactoring.

We can fully resolve these issues by introducing a new method, good?, which shows whether a borrower’s record finds them in good standing.

Local variables are bad because they go together with big methods.

Big methods usually do several things, and local methods help them store the intermediate values that appear in their multi-part processing.

As well as being involved in the work of methods that are already big, local variables make methods even bigger because they usually add at least one line of code each as they come into existence.

So, are they the new global variables?

Yes!

Local variables are the new global variables because as you write better code, you will use them less, and because when they appear in existing code, it is usually a sign that you should refactor.

The next time you discover or create a local variable, consider extracting a method instead. You’ll find yourself with cleaner code and a clearer mind in no time flat.

Find in project - TextMate vs. Vim

2012-03-31T00:00:00+00:00

I spoke on this topic at the March meetup of the Melbourne Ruby group.

TextMate’s ‘Find in Project’ is an extremely useful feature. It presents search hits in a single window and lets you inspect and replace selected occurrences. It understands plain strings and regular expressions. It makes code exploration, renaming and other refactoring a breeze. Responsiveness is fine in small to medium-sized projects that exclude vendor code and log files, and TextMate 2 enhancements address the search performance issues seen in larger projects. There’s not much more to say about TextMate; it just works.

But how do you do it in Vim? When I switched over I was surprised to find no easy answer. Several Vim users told me they don’t need that feature very often and just do it manually. But renaming is an important part of refactoring. It should be quick and easy. So I kept looking, and eventually, solutions began to emerge.

In this post I’ll be using ack (see betterthangrep.com) as a replacement for grep and find. Most of what I’ll say remains relevant regardless of which you choose. If you go with ack, do consider custom configuration like mine, which includes text files and excludes log files by default.

Go to file

The first tip I got was that with external search results read into a buffer, files can be jumped into one by one to make changes across a project.

Say you want to replace ‘persisted’ with ‘recorded’ in the code of tire (an ElasticSearch client). You can start by running this Vim command:

:new | r ! ack persisted

Where :new creates a new buffer, | runs an additional Vim command, r reads into the buffer, ! executes a shell command and ack persisted actually performs the search. The buffer of ack results looks like this:

lib/tire/model/persistence/storage.rb:66:          def persisted?; !!id;           end
test/integration/active_record_searchable_test.rb:56:        assert         results.first.persisted?, "Record should be persisted"
test/models/active_model_article.rb:28:  def persisted?; true; end
test/models/active_model_article_with_callbacks.rb:30:  def persisted?
test/models/supermodel_article.rb:16:  alias :persisted? :exists?
test/unit/model_persistence_test.rb:262:            assert article.persisted?, "#{article.inspect} should be `persisted?`"

With your cursor over a filename, in normal mode, you can press gf to open the file (then :b# to return to the previous buffer). Even better than that, gF will open the relevant file and jump right to the specified line number.

Perform your first substitution with :s/persisted/replaced/g, then repeat it later with :s. Don’t forget to save the modified buffers as you go with :w (write) or all together at the end with :wa (write all).

The ack.vim plugin

Pulling in the output of a shell command is a great general-purpose trick, but when it comes to ack you can save yourself some typing by installing the ack.vim plugin.

This gives you the :Ack command, which opens ack search results in the the quickfix window. Our search command becomes:

:Ack persisted

From the quickfix window, ack.vim lets you open a match with o or enter, or preview a match (keeping the cursor in the quickfix window) with go. Pressing v will open the match in a vertical split.

The quickfix window is not specific to ack.vim. It’s a general purpose feature of Vim itself. For more information check out :help quickfix. You’ll find that you can (re)open the quickfix window with :copen, but beware that some of the ack.vim shortcuts won’t work anymore.

The args list

Derek Wyatt’s excellent Vim tutorial screencasts do cover editing many files. His example demonstrated loading relevant files into Vim’s args (arguments) list from the shell then recording a macro to process the first file, save it and move on (:n for next, :p for previous). The recorded macro was then repeated to cycle through each file in the list of filename arguments, making the required changes along the way.

This is clearly a very promising technique. Unfortunately, loading the args list by invoking Vim on the command line isn’t always ideal - particularly if you’re not running Vim in client/server mode. You want to be able to set up the args list for a batch edit without leaving Vim.

And it turns out that you can! Run :args (or :ar) to see the current args list. Set the args list by giving filenames to the same command:

:args fileone filetwo filethree

Even better (but less obvious from :help args) is that you can load the args list using wildcards or the output of a shell command:

:args **/*.rb
:args `ack -l persisted`

Check out :help argadd and :help argdelete as well.

Processing all args with :argdo

Not only can you cycle through the args list one by one, you can also run a Vim command on all those files in one go. I discovered the :argdo command via James O’Beirne. You can read more about it in :help argdo.

With argdo, our seach and replace in project (confirming each substitution after examining it in context) becomes a two step process. First, load the relevant files into the args list. Then, in all those files (:argdo), on every line of each file (%), substitute ‘persisted’ with ‘recorded’ (s/persisted/recorded/), everywhere in each line (g), asking for confirmation each time (c), and save each buffer if changes were made (update):

:args `ack -l persisted`
:argdo %s/persisted/recorded/gc | update

Processing the quickfix list

Vim provides ways to iterate over args (:argdo), buffers (:bufdo), windows (:windo) and tabs (:tabdo), but surprisingly it has no equivalent for processing the contents of the quickfix list. Fortunately, this is easy to fix.

You can install a solution in the form of the QFDo plugin, or head over to Stack Overflow to read the code from its original source. With QFDo in place, our search and replace can be performed as follows:

:Ack persisted
:QFDo %s/persisted/recorded/gc | update

Another Stack Overflow contributor has offered an alternative approach: Qargs, which will load the quickfix list into the args list, allowing you to proceed with :argdo:

:Ack persisted
:Qargs | argdo %s/persisted/recorded/gc | update

I’m not yet sure which I prefer, but I’m leaning towards Qargs with :cclose added, to close the quickfix window as it is loaded into args (returning the cursor to the main window, ready for the next command).

Closer to TextMate with greplace.vim

Those who want something closer to the experience of TextMate’s ‘Find in Project’ might like to try the greplace.vim plugin.

With greplace.vim you can search across your project with :Gsearch, edit hits together, in the new buffer it creates, and then write those changes back into their respective files with :Greplace. It’s also possible to restrict your search to files in the args list (:Gargsearch), and to load hits into the editing buffer from the quickfix list (:Gqfopen).

The greplace.vim plugin’s interpretation of filename masks (which :Gsearch requires) remains something of a mystery to me, but if you find you really want to see search hits in one place when editing them, rather than jumping from file to file to confirm changes, greplace.vim is a strong candidate for integration into your Vim search and replace workflow.

Conclusion

TextMate makes find in project easy. It is something you’ll miss for a while if you’re making the switch to Vim. But Vim does have equivalent functionality, and with a bit of practice and streamlining it’s probably more effective overall.

Vim matches TextMate’s power, and raises it. For example, TextMate can do find and replace of strings or regular expressions across selected files, but in Vim it is trivial to perform batch processing of selected files with a complex macro (e.g. :argdo normal @a).

You can use your shell to select files to pass to TextMate, but from Vim you can drive powerful two-way interaction with the shell from your editor, combining and exploiting UNIX’s many tools to a greater degree, without needing to write extensions for each use case.

Processing PDFs for Kindle readability

2011-12-31T00:00:00+00:00

I find it a pleasure to read appropriately formatted eBooks on my new Kindle Touch. However, PDFs are a fixed layout format and the Kindle presents them one full page at a time. Zooming and panning are possible, but not practical for normal reading.

A typically formatted PDF will be presented as the following illegible mess:

I’ll show you how to quickly and easily split and rotate PDF pages for results like this:

Mileage will vary with the source document dimensions, but our example of US-letter sized pages with a small, serif font enclosed in modest margins does go from illegible to relatively comfortable reading. Many documents will yield greater improvements.

Divide and crop with Briss

Briss (the BRIght Snippet Sire) is free, open source software for quickly turning each page of a PDF into several pages defined by rectangles you draw. It runs on Mac, Linux and Windows. Download it now.

If you’re not on Windows, you’ll want to ignore the briss.exe file and run the file named briss-0.0.13.jar. Note that Mac OS X, since Lion, no longer pre-installs the Java Runtime Environment (JRE), which Briss requires. So, if you don’t already have it, you might need to install the JRE first.

Fire up Briss and load your PDF. Briss will group pages of equivalent geometry (e.g. left pages and right pages) and overlay all of each group’s content on a single page.

You’re then free to modify the size of the initial cop box and add additional boxes by clicking and dragging on the page. I suggest splitting your pages in half horizontally and removing as much border space as possible. I’d usually remove page numbers as well. The Kindle shows its own page numbers (although doubled, assuming you’ve split your pages in half) and PDFs with embedded bookmarks will remain navigable via the Kindle’s ‘Table of Contents’ menu. It’s also a good idea to overlap the two areas a bit, so that any line cut off by the new page border will be repeated in full on the following page.

The cropping areas defining your new sub-pages should look something like this:

Select Action > Crop PDF from the menu and save your new PDF. Open it up in a PDF reader and you’ll see that you now have twice as many pages, each with half the content.

Rotate

The Kindle Touch lacks a page rotation function, but even if you have a Kindle that allows it, you’re probably better off doing your page rotation in advance.

Rotate your pages 90° counterclockwise. This ensures that the button or screen area you press to move to the next page will be at the bottom of your rotated, landscape-oriented pages.

On Mac you can rotate pages using the default PDF reader, Preview.app, by turning on the Thumbnail view, selecting all the pages, and choosing Rotate Left from the Tools menu. Many other programs will do this too. If you get stuck finding one for your platform, try Pdftk - the PDF toolkit. It is a free command line tool that runs on Windows, Linux, Mac OS X, FreeBSD and Solaris. Once installed, you’d run pdftk unrotated.pdf cat 1-endW output unrotated.pdf to rotate all the pages of your document 90° counterclockwise.

You’re all done! Send the PDF to your Kindle using your Send-to-Kindle email address or a USB cable and enjoy your book without a magnifying glass.

Internet access from China

2011-11-09T00:00:00+00:00

Parts of the Internet are blocked in China. How much impact does that have on everyday Internet life?

I usually access the Internet via a VPN (an encrypted tunnel) that passes my traffic straight out to the rest of the world, bypassing the blocks. I decided to look back over a month of my Internet history without a VPN and see what was missing.

In the course of the month I used around 100 unique blocked sites. That’s around 3 new blocked sites per day, but do bear in mind that there are a number of blocked sites that I use many times per month.

How are sites blocked?

The technical details of the Great Firewall’s various blocking methods are beyond the scope of this article. However, I would like to mention that the various techniques employed lead to a range of experiences for users who attempt to access blocked sites.

The most extreme case is a site that doesn’t load at all. Either the host name can’t be resolved to an address or the connection is reset before anything is loaded. A step down from there are the sites that take a long time before failing to fully load, or that have certain elements that don’t load. Flickr suffers from the latter. It does mostly work, but is regularly missing photos. Then there are sites that do load entirely, but so slowly that they are unusable. The Basecamp project management application falls into this category. Finally, there are some sites that fully function (and fail) intermittently. Google’s Gmail will often work for several days at a time, then fail completely for the next couple of days.

Public holidays and other days of national significance tend to see stronger blocking. Perhaps this is intended to discourage political activity during the holidays, or to test more restrictive systems at times of reduced commercial impact, or a combination of those, or something else entirely.

What was blocked?

To demonstrate the incompatibility of my web browsing habits with the Great Firewall, I’ve broken down a selection of blocked sites I visited over the month into representative categories. They are:

Google and Twitter

Google Search USA (Hong Kong usually works)
Gmail
Google Groups (discussion platform for many public software projects)
Google Docs (only encrypted connections are blocked)
Twitter
Twitpic (for sharing media on Twitter)
yfrog (more Twitter media sharing)

Things starting with ‘face’

Facebook
Face detection jQuery plugin (free code for finding faces in photos)
Facelette (a decommissioned experimental app related to Apple FaceTime)

Books and films

Goodreads (a place to discover and discuss books)
The Internet Movie Database

Video sharing

Major blogging platforms and the many blogs they host

Other technology blogs

Tools and resources for software development

Travis CI integration testing platform
Instant CSS Documentation Search
Fabrication object generator
Emscripten: An LLVM-to-JavaScript compiler
dotfiles.org
Coderetreats w/ Corey Haines
Code Academy
AreWePlayingYet? (browser compatibility test)
Boston Ruby Group

Software

Dropbox file backup and synchronization (an encrypted connection works)
Linux Mint (a distribution of the Linux operating system)
Basecamp online project management tool
Suavetech
Yazsoft

Miscellaneous

OpenStreetMap (like Wikipedia but for maps)
Wikimapia (open metadata on copyrighted maps and satellite images)
Deutsche Welle world news
The Pragmatic Bookshelf (publisher of programming books)
Khan Academy (an incredible online learning resource)
Flickr (photo sharing)
Trnsfr (for sending web pages to your phone)
StrongVPN

As you can see, it goes well beyond Google, Twitter and Facebook. In particular, the major blogging platforms make up a large chunk of the English-language Internet. The blocking covers the platform sites themselves, but also the many millions of individual blogs hosted by these services.

Was this indended?

One can imagine political, economic or cultural motivations for much of China’s Internet filtering, but the examples from my Internet usage suggest there is also a lot of unintentional over-blocking.

ElasticSearch as a primary data store

2011-10-18T00:00:00+00:00

Note: This post explored the idea that for some applications, it may be possible to model data more naturally and avoid some overheads by using ElasticSearch without a relational database backing it. I warned that this may not be possible or advisable for a range of reasons, including functional requirements, performance characteristics and operational concerns.

These are interesting times to be working with data; particularly because of the new possibilities that NoSQL (non-relational) databases offer for data modelling and manipulation. In addition to the new document databases, key-value stores and graph databases, there is an emerging option that has been hiding in plain sight: using your search engine as the database.

A conventional application begins with data stored in a relational database. If search and aggregation needs exceed what it can provide, an additional system is added: the search engine. The former remains the authoritative data store and the latter is a tacked-on hack that stays downstream but earns its keep by finding results quickly.

As ElasticSearch has developed to address the complications of previous search engines and adapt to modern NoSQL and hosting environments, it has come to resemble a data persistence platform in its own right. It can store and retrieve arbitrary JSON data, so it is useful for richly structured data as well as free-form text. It can be updated in near real-time, so there need not be any appreciable delay between the creation of new data and its availability for retrieval.

Extra tools incur extra overheads. If ElasticSearch already has all your data in a form that is persistent, updatable, and accessible by ID lookup or search queries, why go to the effort of replicating a subset of that functionality in a separate “primary” data store?

Some applications - such as those involving full scans that process whole records and produce large results - are not suited to the retrieval and aggregation options provided by search engines. Others require sophisticated transaction support during writes. Many more are perfectly suited, but there are still a couple of reasons for you to tread carefully.

ElasticSearch provides redundancy that can protect against hardware failure, and recent versions appear free of data corruption issues, but durability and facilities for taking recoverable backups have not yet had as much attention as in other data stores. From an application programming point of view, you should also keep in mind that object-document mappers (ODMs) targeting ElasticSearch are not yet as mature and full-featured as those for document stores such as MongoDB.

Despite these considerations, ElasticSearch is certinaly ready for experimental use as a data store, and depending on the requirements of your application, it may even be the only one you need.

A demonstration in Rails

To show you how simple it is, I’ve whipped up a basic note taking application that uses ElasticSearch as its primary and sole data store. Much of the credit for this should go to Karel Minařík, whose ElasticSearch API, Tire, made it a cinch. You can find the demo application on GitHub and run it locally by following the instructions in the README file.

Read on here for tips on building your own.

Disabling ActiveRecord

After creating a fresh Rails application, the first step is to disable ActiveRecord. In config/application.rb find the line with require 'rails/all' and replace it with:

require "rails"
[
  # 'active_record',
  'action_controller',
  'action_mailer',
  'active_resource',
  'rails/test_unit',
].each do |framework|
  begin
    require "#{framework}/railtie"
  rescue LoadError
  end
end

Then you can delete config/database.yml, remove gem 'sqlite3' from your Gemfile and rerun bundle install.

ElasticSearch-backed models and near real-time updates

With the database stuff out of the way you are free to create models that persist to ElasticSearch. Check out the Tire README, paying particular attention to the last part, which discusses its persistence features. You’ll be up and running in no time.

There is one small trick you should know. After adding a record to an ES-backed model, you may find that you need to reload your index page before the new record appears. This is because ElasticSearch performs updates in “near real-time”.

Near real-time means that it may take up to one second before the creation or update of records is reflected in search results. This is different from GETing a record based on its ID, which in recent version of ES will work immediately. The one second period can be configured to be shorter, but to ensure that a recently indexed record is available you should make an explicit refresh call.

For any application that is not write-heavy this should pose no performance problems and it is easily implemented. In my Note model I do it in these 3 lines:

refresh = lambda { Yire::Index.new(ES_INDEX_NAME).refresh }
after_save &refresh
after_destroy &refresh

Do you need your database?

You might like to read through the related discussions on the mailing list. I hope you’ll give this question some thought, and have fun with ElasticSearch either way!

Blogging with Serve

2011-10-05T00:00:00+00:00

This blog was built by generating a static site with Serve. It was deployed with rsync to an Ubuntu VPS at Linode, itself built using my Babushka deps.

Selecting a static site generator

I decided to use a static site generator because I wanted something Ruby-based, very simple, and easy to extend. I considered several before settling on Serve.

Jekyll is the static site generator that powers GitHub Pages. It uses Liquid templates, which do the job of ERB but can be safely rendered without trusting their author. This is perfect - and necessary - for GitHub Pages, but as I have full control of the rendering and hosting environments, it didn’t seem like the right fit. In hindsight, lighter templates are a good thing, and I wouldn’t let Liquid put me off using Jekyll for trusted sites in the future. I’ve also heard good things about Octopress, which builds on Jekyll to get you started blogging quicker.

Nanoc looked perfect. It has clean and extensive documentation. It uses a Rules file to specify mappings from content files, via filters and layouts, to final output. It has blogging helpers baked in. Things were going okay, but after a while the Rules file got frustrating. I felt like I was having to jump through too many hoops just to get back to simple static pages.

Serve Serve has a nice introductory screencast and straightforward documentation. It works like a slimmed down version of Rails views. You don’t have routes, controllers or models, just templating languages, layouts and helpers. You can run it as a rack app, or export your site to static files as I do.

Blogging with Serve

Serve wasn’t built with blogging in mind, so I needed to handle that myself. My requirements were modest: I wanted to have a table of contents page and an Atom feed generated automatically, and to write new posts in single files without repeating myself anywhere.

Embedding metadata in ERB templates

The first challenge was gathering post metadata. Nanoc and Jekyll let you include YAML at the top of your templates. Serve doesn’t, but I found a solution using just ERB and helpers.

I have each post append a hash of metdata to the end of an array in an instance variable, which is created if it doesn’t already exist. It also includes the template file name and the captured post HTML in the hash. I wrapped this process up in a helper, which also prepares data from the last post for presentation in the blog post layout, using #content_for.

An individual post template look like this:

<%
  capture_post_data(__FILE__, {
    title: "An example post",
    published_on: "1970-01-01",
    updated_on: "1970-01-02"
  }) do
%>

  <p>A blog post!</p>

<% end %>

The helper is:

def capture_post_data(file, metadata, &content)
  @post_data_raw ||= []
  @post_data_raw << {
    file: file,
    containing_dir: file.split('/')[-2],
    updated_on: nil,
    topless_html: capture(&content)
  }.merge(metadata)
  @post_data_raw.last.each { |k,v| content_for :"last_post_#{k}", v }
end

Rendering all the post templates, in any order, will build up the instance variable of all the metadata, which is used by other helpers to create the table of contents and generate the Atom feed. Another helper, #dont_capture_post_data, does the same without leaving the metadata hash in the array, allowing me to easily toggle posts in and out of publication in the index and feed.

Multi-level layouts

Serve lets you wrap your templates in layouts, written in files named _layout.html.erb. It will use the one in the current directory, or the first it finds as it ascends the directory tree. This makes it easy to have branches of your site use the same layout, and to let certain lower directories override a more general layout from higher up. You can also render partial templates to factor out common fragments.

Duplication emerged in two places in my templates. The first was in my individual blog post directories, each of which had a _layout.html.erb file pointing to the blog post layout (overriding the blog index layout from the parent directory). I resolved this by putting logic into the blog root’s layout file, to decide whether to render the post layout, the blog index layout or, for the feed file also in that directory, no layout.

The next issue was that I wanted the content of blog posts to be rendered by a blog post layout, but to have it share a wrapper with the blog index page, and to have those two use the same top level wrapper as the site’s other pages. It would be nice to let every template or layout specify what it should be wrapped in. This didn’t work out of the box, but I was able to cook up something similar using the #capture helper.

My mid-level layouts embed themselves in higher level ones by capturing their own content and explicitly rendering the desired outer layout:

<% inner = capture do %>
  <div>Mid-level markup</div>
  <%= yield %>
<% end %>
<% content_for :inner, inner %>
<%= render template: "/layouts/application" %>

The outer layout, of course, runs yield :inner to wrap itself around content captured at lower levels.

On reflection

Serve is really just about templates and view helpers and I’ve shoehorned in more structured blog content and metadata, and methods to manipulate them. These things would be more at home in a model class (database backed or otherwise). It works well enough for now, and it is still quite simple and very DRY, but there are surely better ways to do this.

CHRISBERKHOUT.COM

Printing plain text

Problematic options

A lightweight solution

Heavyweights: HTML and CSS

A simple webhook with Netcat and systemd

Handling a request

Handling successive requests

Granting permissions

Remote control

Conclusion

Maths learning

Other lists and advice

School maths

Mathematical thinking and proof writing

Calculus

Linear algebra

Probability

Statistics

Surveys

Other resources

Log tailing tips

Show only the first 5k new error lines

Lines not including ‘SUCCESS’ or ‘200 OK’ or backtraces

Show the 3 lines before and 10 lines after each exception

Watch the rate of a certain exception

Show 10k log lines and save them in a file

Show 10k log lines and append them to a file

List instances in captured logs

Count instances in captured logs

Group captured logs by instance, in one file

Group captured logs by instance, in separate files

Count log lines in each instance file, sort ascending

Log visualization with gnuplot

Selecting interesting log messages

Extracting times & generating counts per second

Plotting with gnuplot

Plotting style choice

Lower resolution counts

Further reading

Dealing with localStorage errors

Dive Into HTML5: Local Storage

PaperKilledRock.com: HTML5 localStorage

Chrome

Firefox

Safari

Conclusion

Mapping suburbs with jVectorMap

Clean documentation

Isolated subclass specs in Ruby

Set an expectation on #execute or on #super?

Sneak a module into the inheritance chain

Extract a module with the method override

Test the full class and stub problematic parts

In conclusion

Local variables - the new global variables?

Find in project - TextMate vs. Vim

Go to file

The ack.vim plugin

The args list

Processing all args with :argdo

Processing the quickfix list

Closer to TextMate with greplace.vim

Conclusion

Processing PDFs for Kindle readability

Divide and crop with Briss

Rotate

Internet access from China

How are sites blocked?

What was blocked?

Google and Twitter

Things starting with ‘face’

Books and films

Video sharing

Major blogging platforms and the many blogs they host

Other technology blogs

Tools and resources for software development

Software

Miscellaneous

Was this indended?