Krzysztof Kowalczyk blog

How I ported pigz from Unix to Windows

2013-03-20T11:25:36-07:00

I just finished porting pigz from Unix to Windows (you can download pre-compiled binaries). This article describes how I did it.

Pigz was clearly written with Unix in mind, with no thought given to cross-platform portability.

Thankfully, it's a relatively simple, command-line program that sticks to using standard C library.

Porting pthreads

Pigz uses pthreads for threading. Porting pthreads code to Windows would be a nightmare. Lucky me: someone already did all the hard work and implemented pthreads APIs on top of Windows API, in only 20.000 lines of code. It seems to Just Work.

Porting dirent

Another Unix-only API that pigz uses is dirent.h, for reading the content of directories. I was lucky again: someone created a Windows port of dirent APIs.

Misc fixes

There are few functions that pigz uses that are present in Visual Studio's C library, but under a different name (e.g. stat is _stat, fstat is _fstat etc.). This is easily fixed with a #define.

Visual C doesn't define ssize_t and PATH_MAX. A typedef here, a #define there solves those problems.

I consolidated all such fixes in a single wincompat.h file.

Pigz #includes some .h files that are only available under Unix. I used #ifndef _WIN32 around those.

Build system

Pigz uses standard Unix build tools: gcc and make.

While there are ports of GNU make to Windows and gcc-based compilers that can generate windows binaries (mingw), most Windows developers prefer using Visual Studio IDE.

Creating Visual Studio project files from scratch is time consuming and annoying, especially when you want to support multiple versions of Visual Studio.

Premake makes it easier. It's a meta-build system: you write a text file that defines the project, which files to compile, compilation flags etc. and premake generates Visual Studio files from that description.

Premake supports Visual Studio 2008 and 2010 (and 2012 supports 2010 project files via conversion).

Conclusion

As far as porting goes, this one was easy thanks to existing efforts that created compatibility shims for the hard parts.

Premake is an interesting tool that allows to save time creating and maintaining Visual Studio projects.

Pigz windows port

2013-03-19T18:13:02-07:00

Pigz is a parallel gzip - it speeds up compression/decompression by using multiple threads.

Until today it was only available for Unix. But not anymore!

I've ported it to Windows. Get it while it's hot.

Thoughts on Go after writing 3 websites

2012-12-31T15:30:45-08:00

Go for writing web servers

Summary: in my experience Go is a good language for building websites/web servers.

It’s easy to get excited about new technology like Go. The question is: how does it stand up to scrutiny after daily use?

I’ve written 3 web applications in Go, they’ve been running in production for over a month so I feel justified in publishing my opinion.

In the past I wrote web applications in Perl, PHP, Python (web.py, Tornado, App Engine) so those are the technologies I campare Go to.

Of the 3 websites, AppTranslator, is web service for crowd-sourcing translation for software and was written completely from scratch.

Fofou is a simple forum and is a port of an earlier version I did for App Engine. Finally, this web site is a blog engine (also a port of an earlier App Engine version).

One reason to migrate from App Engine to my own server was to save money. At my levels of traffic (~3 requests per second) I was paying ~$80/month, mostly for the frontend instance hours.

Another reason was to do more complex processing (App Engine is great as long as you don’t have to do something that App Engine doesn’t support).

Finally, I wanted to see how Go will handle a real life project. The best way to test a new technology is on a project with a predictable (and relatively small) scope.

All websites run on the same Kimsufi 24 dedicated server (which is pretty beefy for its $60/month price). I’m using latest Ubuntu for the OS.

Things you need for building a web application

What functionality is typically needed for writing a web application and how does Go support it?

Http server

A very capable http server is part of standard library (net/http).

It parses incoming http requests, provides an easy way to send http responses back.

Url routing

High-level web frameworks provide url routing i.e. a way to say "call this function to handle this url". Go has a simple built-in router. I also use mux which is built on top of the built-in router.

Templates

A lot of what web server does is returning html. Constructing this html is greatly simplified by using templates. Go has a powerful html/template library for that. To me it seems roughly equivalent of Django or Tornado templates.

There are other templating libraries for Go, but I found the above built-in package satisfactory, so I didn’t even try to use them.

Cookies

Basic support for cookies is part of built-in http library. To generate cookies that cannot be spoofed or hi-jacked, I used securecookie library

Databases

There are Go libraries for all of the popular databases (MySQL, PostgreSQL, MongoDB, Redis etc.) but I haven’t used them.

I used what I call NoDB approach i.e. I wrote a very simple storage system that uses text files. Data is kept in memory and persisted in an append-only file.

This wouldn’t be the right approach for services that require more sophisticated functionality but was good enough for my needs and didn’t take long to implement.

Oauth

There are three oauth libraries that I know of. I used this one. I didn’t have any particular reason to choose this one over the others. I only needed it for implementing Twitter-based authentication, this library worked so I used it.

Generating atom (rss) feeds

One cannot respect a blog engine that doesn’t provide full-text rss feed. I couldn’t find an existing package so I build a simple (and small) library for generating atom feeds

JSON and XML support

Go has a built-in support for JSON and XML.
There are APIs for raw parsing/serialization. If you know the shape of the data, you can serialize (marshal, in Go’s parlance) data structures to JSON or XML and de-serialize (unmarshal) from JSON or XML into a struct.

S3 access, support for zip files

This one is not universally needed. My web apps have built-in backup functionality which stores data, sometimes in the form of a .zip file, in s3.

Go has support for creating and decompressing zip and tar archives in the standard library. For s3 support I use goamz.

Unit tests

This is not specific to writing web server - all your important code should have unit tests.

Go has a built-in API and tool support for writing and running tests.

Deployment

Deploying a new version of your code to the server is a pain in the ass regardless of the language used.

I wrote relatively short deployment script using Fabric (which is a python library and a tool for running deployment scripts). It copies the source files to the server, compiles them on the server, runs unit tests, shuts-down existing instance of the service and launches the new version. It stops if there’s anything wrong along the way.

It’s really important to be able to quickly deploy new versions of the software so those days I would write the deployment script as the first thing in the project. Doing all those deployment steps manually would be very annoying.

Server config and misc thoughts

The overall setup of the server is pretty standard: each service is running as a separate process on a local port. Nginx is running on port 80 and proxies the traffic to a given service based on Host header.

Show me the code

The source for all three projects is publicly available on Github, using liberal BSD license: App Translator, Fofou, blog

Feel free to learn from the code or use it in your own projects.

Parting thoughts

I think writing non-trivial web services is a sweet spot for Go.

Most of the needed functionality is part of standard library. For almost everything else there are 3rd party libraries.

Writing in Go is almost as fast and fluent as writing in Python but the code is order of magnitude faster and uses less memory.

SumatraPDF 2.2 released

2012-12-25T20:50:12-08:00

We are happy to announce release 2.2 of SumatraPDF, a multi-format PDF, XPS, ebook (ePub, MOBI), DjVu reader for Windows.

In version 2.2 we’ve added support for FictionBook ebook format.

We’ve added support for PDF documents encrypted with Acrobat X.

In a rare case where printing looks different than display, you can now use “Print as image” compatibility option in print dialog.

For better viewing of manga comic books we now have a new command-line option: -manga-mode [1|true|0|false].

Finally, we’ve made many robustness fixes and improvements. Too small to describe them in detail but in aggregate they make a difference.

Design and implementation of translation system for desktop software

2012-12-16T18:50:28-08:00

A web-based, crowd-sourced translation system.

I just finished building AppTranslator, which is a third iteration of a system that allows people to contribute translations for SumatraPDF.

This post describes the big picture of how it’s designed and implemented. Everything I discuss is open-source, so you can dig into code for more details.

The goals of translation system were to make it easy for me to add new strings for translations and easy for other people to contribute translations.

I call it a "system" because it consists of several parts that are designed to act as a sensible whole.

How translations are stored in C++ code

I try to design for simplicity of implementation to minimize the amount of code to write. Often it means that I reject popular solutions if I can design a simpler one.

SumatraPDF is a C++ desktop software for Windows. There already is a popular system for maintaining translation for such programs: use resource dlls, one dll per supported language. The translations are kept in .rc files as strings and referred to by numeric ids from code via LoadString() API.

That would be very clumsy both for me (when adding new translations) and for potential translators (they have to edit files in a cryptic format). It also requires distributing multiple dlls (I prefer to ship my apps as single executables).

Managing translation strings in C++ code

The first part of the system was therefore a new design for managing translation strings in the app.

All translations are marked with _TR() macro in the code.

A python script extracts those strings from the source. It then reads translations provided by contributors and generates a C++ file with strings. _TR() macro evaluates to calling a simple supporting code in Translations.cpp that looks up a translation of a string given currently selected language.

How translations are contributed by users

SumatraPDF is an open-source project so I rely on contributions from users to keep it translated.

The system for collecting contributions had 3 iterations.

In first iteration, I had a single text file with all translations.

It looked like this:

String to translate
de:German translation
fr:French translation

Another string to translate
de:another German translation

People would download the latest version of the file from svn, add missing translations and e-mail it to me. I would check-in that to svn and re-run the script that rebuilds C++ file with strings.

At some point the file became very big so I split it into multiple files, one per language.

It was working ok but the process of submitting translation was time consuming for translators and the process of updating the code was time consuming for me.

For that reason I built a web-based service which makes it much easier to contribute a translation.

I also added the necessary API endpoints in the server to allow writing scripts for automating uploading strings to translate and downloading latest translations.

The design of web-based UI

A web-based UI for editing translation is not a novel idea. However, my brief research shows that few do it well.

I try to be pragmatic about things. If a decent 3rd party system with enough flexibility to meet my needs already did exist, I would rather use it than develop my own (writing code takes time).

I’m tempted to say that I designed for simplicity but, while being true, it’s also rather vague. More accurately: I designed for simplicity of the translator workflow.

You can judge the UI yourself but let me point out few specific points:

it takes 2 clicks from main page to a point where you can submit translation. The first click is to select the project (as AppTranslator is designed to support multiple projects) and the second click to select the language
on the page for a given language, the untranslated strings are at the top. The rest is sorted
I don’t require creating an account for AppTranslator but authenticate with Twitter

Those might seem like obvious points but I found that other systems I’ve surveyed were baroque enough to inspire Kafka.

Ubuntu’s translation system gets a special recognition for amount of bureaucracy, complicated workflows (joining a translation team to submit a translation?) and bad copy (they feel it’s important to inform you that in order to contribute a translation you need an internet connection, among many other useless bits of information).

Mozilla is marginally better

Technical specs

AppTranslator is written in Go and open-source (BSD license).

I run it on Ubuntu server. It’s possible, but not easy to run your own instance.

SumatraPDF is also open-source. The code is in C++ with a bunch of python helper scripts to automate interaction with AppTranslator server.

Speeding up Go (and C++) with custom allocators

2012-11-26T12:17:53-08:00

Speeding up Go with custom allocators

Summary: using a custom allocator I was able to speed up an allocation heavy program (binary-trees benchmark) ~4x.

Allocation is expensive. It holds true for all languages. At the time of this writing, Go (version 1.0.3) doesn’t have a garbage collector that is as sophisticated as, say, garbage collector in JVM. There’s work being done to improve Go’s GC but today allocations in Go are not as cheap as they could be.

This can be seen in binary-trees benchmark which has almost no computation but millions of allocations of small objects. As a result, Java implementation is about 7x faster.

I was able to speed up Go code by about 4x by using a custom allocator.

The benchmark builds a large binary tree composed of nodes:

type struct Node {
  int item
  left, right *Node
}

To allocate a new node we use &Node{item, left, right}.

Improving allocation-heavy code

First, a correction. When I said that allocation is expensive, I over-simplified.

In garbage-collected languages allocation is actually very cheap. In a good allocator it’s just a single arithmetic operation to bump a pointer, which is orders of magnitude cheaper than even the best malloc() implementation. Malloc() has to maintain data structures to keep track of allocated memory so that free() can return it back to the OS.

More complicated reality is that it’s garbage collection phase that is expensive.

Garbage collection (gc) is triggered every once in a while. It recursively scans all the allocated objects, starting from known roots and chasing pointers. It figures out which objects are not referenced by any other object and frees them (this is the "garbage" in garbage collection that has just been collected).

There are 2 insights we get from knowing how garbage collection works internally:

the more objects there are, the more expensive garbage collection is
the more pointers we need to chase, the more expensive gc is

Knowing what the problem is, we know what the solution should be:

allocate less objects (e.g. by allocating them in bulk or re-using previously allocated objects)
don’t use pointers

As it happens, the majority of the 4x speedup I got in this particular benchmark came from not using pointers

Speeding binary-trees shootout benchmark

We said that we should avoid pointers, so that garbage collector doesn’t have to chase them. The new definition of Node struct is:

type NodeId int

type struct Node {
  int item
  left, right NodeId
}

We changed left and right fields from *Node to an alias type NodeId, which is just a unique integer representing a node.

What NodeId means is up to us to define and we define it thusly: it’s an index into a []Node array. That array is the backing store (i.e. allocator) for our nodes.

When we need to allocate another node, we expand the []Node array and return the index to that node. We can trivially map NodeId to *Node by doing &array[nodeId].

Our implementation is a bit more sophisticated. In Go it’s easy to extend the array with append() but it involves memory copy. We avoid that by pre-allocating nodes in buckets and using an array of arrays for storage. The code is still relatively simple:

const nodes_per_bucket = 1024 * 1024

var (
	all_nodes    [][]Node = make([][]Node, 0)
	nodes_left   int      = 0
	curr_node_id int      = 0
)

func NodeFromId(id NodeId) *Node {
	n := int(id) - 1
	bucket := n / nodes_per_bucket
	el := n % nodes_per_bucket
	return &all_nodes[bucket][el]
}

func allocNode(item int, left, right NodeId) NodeId {
	if 0 == nodes_left {
		new_nodes := make([]Node, nodes_per_bucket, nodes_per_bucket)
		all_nodes = append(all_nodes, new_nodes)
		nodes_left = nodes_per_bucket
	}
	nodes_left -= 1
	node := NodeFromId(NodeId(curr_node_id + 1))
	node.item = item
	node.left = left
	node.right = right

	nodes_left -= 1
	curr_node_id += 1
	return NodeId(curr_node_id)
}

Remaining changes to the code involve adding NodeFromId() call in a few places.

You can compare original to my faster version.

Another minor advantage if using integers instead of pointers in Node struct is that on 64-bit machines we use only 4 bytes for an int vs. 8 bytes for a pointer.

Drawbacks of custom allocators

The biggest drawback is that we lost ability to free objects. Memory we’ve allocated will never be returned to the OS until the program exits.

It’s not a problem in this case, since the tree only grows and the program ends when it’s done.

In different code this could be a much bigger issue. There are ways to free memory even with custom allocators but they require more complexity and evolve towards implementing a custom garbage collector at which point it might be better to go back to simple code and leave the work to built-in garbage collector.

How come Java is so much faster?

It’s a valid question: both Java and Go have garbage collectors, why is Java’s so much better on this benchmark?

I can only speculate.

Java, unlike Go, uses generational garbage collector, which has 2 arenas: one for young (newly allocated) objects (called nursery) and one for old objects.

It has been observed that most objects die young. Generational garbage collector allocates objects in nursery. Most collections only collect objects in nursery. Objects that survived collections in nursery are moved to the second arena for old objects, which is collected at a much lower rate.

Go collector is a simpler mark-and-sweep collector which has to scan all allocated objects during every collection.

Generational garbage collectors have overhead because they have to copy objects in memory and update references between objects when that happens. On the other hand they can also compact memory, improving caching and they scan a smaller number of objects during each collection.

In this particular benchmark there are many Node objects and they never die, so they are promoted to rarely collected arena for old objects and each collection is cheaper because it only looks at a small number of recently allocated Node objects.

There’s hope for Go, though. The implementors are aware that garbage collector is not as good as it could be and there is an ongoing work on implementing a better one.

A win in C++ as well

Optimizing by reducing the amount of allocations or making allocations faster is applicable to non-gc languages as well, like C and C++, because malloc()\free() are relatively slow functions.

Back in the day when I was working on Poppler, I achieved a significant ~19 speedup by improving a string class to avoid an additional allocation in 90 of the cases. I now use this trick in my C++ code e.g. in SumatraPDF code

I also managed to improve Poppler by another ~25% by using a simple, custom allocator

Since then I use this trick whenever I can.

Hiding duplicate content from your site via robots.txt

2012-10-22T01:43:33-07:00

Many blogs, including this one, generate duplicate content. For example, the archive pages duplicate the content of individual posts, they just show them in a different way (a couple of posts per page, as opposed to a single post per page).

That unfortunately clogs search engines. Being a perfectionist that I am, I want that a search for e.g. "15minutes" (my simple timer application) leads people to individual blog posts about it and not to aggregate pages with random other content.

Thankfully there’s a way to tell search engines to not index parts of your site. It’s quite simple and in five minutes I cooked up the following robots.txt for my site:

User-agent: *
Disallow: /page/
Disallow: /tag/
Disallow: /notes/

In my particular case, archive pages all start with /page/ or /notes/ and /tag/ is another namespace with duplicate content (shows a list of articles with a given tag).

For this technique to work the names duplicate pages have to follow a pattern, but that’s easy enough to ensure, especially if you write your own blog software, like I do.

How I sped up Go by 20% (or is Go really slower than Java?)

2012-09-16T01:55:48-07:00

The story of a faulty benchmark

I confess: I didn’t really speed up Go by 20%. However, if you were to believe simplistic arguments, I did.

It all started when a comment on Hacker News claimed that Java is faster than Go and pointed to those benchmarks as proof.

Indeed, every Go benchmark there is 2x to 10x slower than Java 7 program.

That seemed wrong to me - Go compiles to native code. Java, ultimately, does too, but compilation happens at runtime, which is an overhead that Go doesn’t have, so in principle Go should be faster.

I took a look at Go implementation. I’m not an expert in implementing reverse-complement algorithm but I immediately noticed that the code does a lot of unnecessary memory allocations and copying.

I looked at Java implementation and what do you know: it’s implemented more efficiently, without unnecessary memory allocations and copying. Plus it parallelizes the computation.

I wrote my own Go implementation, more similar to Java’s.

The result? On my Mac Pro I got about 20% speedup.

The argument of HN commenter was: look at those benchmarks, Go is 2x slower than Java.

It’s a simplistic but unfortunately powerful argument: most people won’t take the time to look at the code to determine that part of the issue is that Java implementation is simply better than Go one.

By that simplistic argument I’ve improved speed of Go by 20% by writing a slightly better implementation of the benchmark.

The story doesn’t end here. The above results were for single core x86 machine. If you look at results for single-core 64-bit machine, Go actually wins on this and 2 other benchmarks.

The explanation is simple: 64-bit Go compiler is a little bit smarter than 32-bit Go compiler.

Is Go really slower than Java?

There is no simplistic answer to this, but there are rules of thumb.

On equivalent programs, Go generates programs similar in speed or faster. 64-bit version will be slightly better than 32-bit version.

Java garbage collector is much better than Go’s which explains why allocation-heavy benchmarks (e.g. binary-trees) perform so much better (there’s almost no computation there, just allocation of a lot of tree nodes).

Those benchmarks aren’t currently representative of Go performance - from cursory look it seems that C++ and Java implementations are extremely optimized and Go implementations aren’t (I’ve submitted my improved version, hopefully it’ll get incorporated).

Go is already competitive with Java and will only get better. Let’s not forget that Java had over 20 years of investments in code generation and garbage collection. Go is only 5 years old. There already were compiler and garbage collection improvements since the latest released version.

One area where Go wins undeniably is memory usage - the programs use at least order of magnitude less memory. Java is paying the cost of sophisticated virtual machine.

Websites with free ePub and mobi ebooks

2012-05-04T15:58:27-07:00

Since Sumatra now supports 2 most popular eBook formats (ePub and mobi), I’ve started collecting a list of websites that offer free ePub and mobi eBooks.

SumatraPDF 2.1 released

2012-05-03T13:30:42-07:00

SumatraPDF is a multi-format PDF, XPS, ebook (ePub, MOBI), DjVu etc. reader for Windows and we are pleased to announce version 2.1.

The biggest change in this version is a support for ePub ebook format. ePub is one of the 2 most popular ebook standards (the other one, mobi, is already supported by Sumatra). Unfortunately we can only read ebooks that are not DRM protected.

We’ve added File/Rename menu item to rename currently viewed file (contributed by Vasily Fomin).

We’ve added support for multi-page TIFF files, TGA images, more comic book (CBZ) metadata.

We’ve added support for JPEG XR images (available on Windows Vista or later, for Windows XP the Windows Imaging Component has to be installed)

The installer is now signed, for even better installation experience.

Buying a certificate for signing windows applications

2012-04-15T22:28:01-07:00

Recently I’ve bought a code signing certificate so that I can sign my Windows application SumatraPDF (I’m hoping it’ll reduce number of false positive from various anti-virus programs that claim that SumatraPDF is suspect).

There were some things that I wish I knew before starting the process, so I’m documenting the process here for posterity.

There are many places to buy a code signing certificate. I bought it from K Software as they have low prices and there’s enough info on the internet to assure me that it’s the right kind of certificate for signing Windows apps. The certificate is actually issued by Comodo, K Software is only a reseller (but they have lower prices than Comodo - go figure).

I bought a certificate valid for 3 years ($245). The validity only affects whether you can use the certificate to sign. After you sign the app, its signature is valid forever.

After certificate expires, you can renew it. The shortest (and cheapest) validity period is 1 year. I opted for 3 years because it minimizes the hassle of renewing every year.

Important note before you start: you also need to have some internet domain registered in your name (or in your organization’s name) and to minimize the troubles make sure there is a valid e-mail address with that domain that you can receive (I use Google Apps for that domain and use it to forward e-mails for that domain to my personal gmail account).

The domain is necessary to complete verification process of your identity that Comodo does. It’s a strange requirement for a certificate for signing applications but I’m guessing it’s because certificate have roots in SSL/internet and in that case domain name is required.

Signing an app with a certificate basically serves as a stamp that says "this application has been signed by company/individual X". For the system to work, X must be a legitimate company/person and not, say, a hacker.

For that reason the organization that issues certificates (in this case Comodo) needs to verify the identity of the person buying the certificate so that e.g. I can’t order a certificate that says my name is "Microsoft" and start signing my apps as coming from Microsoft.

The verification process starts after your purchase the certificate. The details depend on whether you’re a company or an individual. I ordered as an individual and the verification process was:

They asked for a copy of valid id. I e-mailed them a photo of my driver’s license (taken with iPhone)
Then they asked for a copy of a phone bill. I e-mailed them the PDF bill I downloaded from AT&T’s website
Then they called me on my phone to verify the phone number on the bill is my phone number

All in all, the back and forth took the whole day.

After the certificate is issued you need to download it to a file. To do that you need to visit a web page that Comodo created for you in a supported browser (FireFox or IE, Chrome is not supported, I used FireFox) on the same computer that was used to order the certificate.

That creates a certificate and adds it to browser’s certificate store. Finally, you export the certificate to a file. The steps are detailed at http://blog.ksoftware.net/.

As to actual signing, I use K Software’s ksign tool (the command-line version ksigncmd that I call from my build script).

SumatraPDF 2.0 released

2012-04-02T14:54:09-07:00

SumatraPDF is a multi-format PDF, XPS, ebook (MOBI), DjVu etc. reader for Windows and we are pleased to announce version 2.0.

The biggest change in this version is that we can now read ebooks in mobi format. Mobi is a format developed for MobiPocket Reader and made popular by Amazon’s Kindle. Unfortunately, eBooks purchased from Amazon are protected by DRM so SumatraPDF (or any other third-party reader) cannot read them. The good news is that there are other sources of ebooks formatted in mobi format.

The UI for reading ebooks is different than the UI used for other documents. It’s not hard to notice that the inspiration from the UI came from Kindle’s PC application.

There were other changes as well:

We can now open CHM documents from network drives.

After selecting an area with the mouse, the area can be copied to the clipboard as an image with a right-click context menu.

Sumatra has always been a small and fast program and because we applied extreme size optimization techniques, it’s smaller than ever. If we didn’t apply our extreme size-reduction techniques, the installer would be bigger by 9% (which is around 400 kB) (as a bonus, the size-reducing code we developed is available to other programmers under liberal BSD license).

And as always there are many smaller improvements: even better PDF support etc.

A list of chm readers/viewers for Windows

2011-12-09T16:47:56-08:00

I’ve compiled a list of CHM readers for windows. It’s a surprisingly long list.

SumatraPDF 1.9 released

2011-11-24T00:44:29-08:00

We are pleased to announce 1.9 release of SumatraPDF, a small, fast, free PDF, CHM, DjVu, XPS, CBZ and CBR reader for Windows.

The most significant addition in this release is support for CHM documents. Sumatra is slowly getting support for more and more document formats.

Robert Prouse contributed support for touch gestures (available on Windows 7 or later, if you have the right hardware i.e. touch-enabled screen).

Audio and video files linked from PDF documents are now opened in an external media player.

We’ve also improved support for PDF transparency groups.

Introducing Volante - a database for C# (.NET)

2011-09-22T17:12:27-07:00

Volante is a small, fast, object-oriented, embeddable database designed for seamless integration with C# (and other .NET languages). Today I’ve made a first public release.

Almost every program needs to persist some data. There are many ways to do that: serialize data as XML or JSON, use SQLite etc.

I’ve been recently writing desktop .NET applications in C# and existing (free) options didn’t meet my needs well.

The closest best solution is SQLite, but it doesn’t integrate with C# well: you have to convert your object to/from SQL tables.

If I was the world’s toughest programmer I would write an object-oriented database engine from scratch and it would be designed from the beginning for seamless integration with .NET framework.

Thankfully, I didn’t have to do that. What I needed already existed in the form of Perst project.

There was one wrinkle: while early versions of Perst were under BSD license, with version 2.50 the code was acquired by a company McObject and is now distributed under GPL and those who can’t use GPL can purchase commercial license from McObject.

Not so great for one person who doesn’t yet make money from his software.

I decided to adopt the Perst code base. I picked the latest 2.49 version that was still licensed under BSD (copyright cannot be change retroactively) and I’ve spent the last couple of months writing comprehensive documentation, writing tests, fixing bugs discovered by tests, modernizing the code base.

Today I’ve reached a point where I’m comfortable releasing this code publicly as version 0.9.

I’ve retained the BSD license of early Perst versions so the code is free to use in both open-source and commercial projects.

Volante database serves the same niche as SQLite: an embedded database engine for your desktop C# applications. Like with SQLite, the database is in a single file.

There are significant differences from SQLite.

.NET is an object-oriented framework. Volante is an object-oriented database to offer the best integration with .NET. Volante uses B-Trees to implement indexes, which allows quickly finding objects with desired properties.

Volante is extremely small: Volante.dll is only 180 KB.

I distribute Volante.dll for Microsoft’s .NET (works in .NET 2.0 and later) but it can also be compiled from sources and used under Mono.

I was dogfooding Volante from day one in my three .NET applications and it’s been performing great.

I hope you’ll find it useful too.

SumatraPDF 1.8 released

2011-09-18T19:41:14-07:00

We are pleased to announce 1.8 release of SumatraPDF, a small, fast, free PDF, DjVu, XPS, CBZ and CBR reader for Windows.

This is a smallish release.

We improved support for PDF form text fields.

We did speed up handling of some types of djvu files.

We’ve made a bunch of minor improvements and bug fixes.

SumatraPDF 1.7 released

2011-07-17T18:28:26-07:00

SumatraPDF developers are pleased to announce 1.7 release of SumatraPDF, a PDF, DjVu, XPS, CBZ and CBR reader for Windows.

In this release we’ve added user-defined favorites (i.e. bookmarks). You can create one or more favorites for a given file, navigate to a favorite and delete them.

Favorites are accessed either via a menu items in Favorites top-level menu or displayed as a tree in the sidebar.

We’ve improved support for right-to-left languages, like Arabic.

Logical page numbers are displayed and used, if document provides them (such as i, ii, iii, etc.).

We allow to restrict SumatraPDF’s features with more granularity; see this document for more information.

Command-line argument -named-dest now also matches strings in table of contents.

We’ve improved support for EPS files (requires Ghostscript)

Installer is now more robust. Previously an installation could fail if a web browser using Sumatra’s web browser dll was running. Now installer detects this and will ask to close the browser before proceeding.

Until next release.

How to make software crash less

2011-06-30T15:32:59-07:00

You don’t want your software to crash, do you? This post describes my experiences in making desktop Mac and Windows software crash less.

Know thy crashes

The most important step to fixing your crashes is to know about them.

Given how complex and varied our desktop operating systems are (3 major version of Windows in active use, thousands of minor and major ways that each installation of Windows or Mac can be different) our ability to comprehensively test software is not good.

Sure, if you’re Microsoft or Adobe you can reinvest some of the revenue to hire an army or testers, setup compatibility labs etc. but for a small company or a single developer this is not realistic.

Bugs most often lurk in untested code and even a very good testing effort is unlikely to encounter all the many things that can go wrong in real life.

Get the crash reports automatically

Very few people bother to tell software vendors about crashes. They just shrug and restart. The only realistic way to be informed about crashes is to automatically gather crash reports without user involvement.

This is a proven idea. Microsoft was one of the pioneers of using this technique for Windows OS and they publicly praise it for letting them fix the most frequent crashes and increase stability of Windows. Many clued in organization do it as well: Apple, Mozilla, Google.

Getting the crashes - the mechanics

Regardless of the platform, the solution involves two parts:

a server, which accepts crash reports from the software
code inside the software itself. It gets activated when a crash happens and sends crash report to the server

The server

The server part is simple. You can use any technology to write it, any protocol you want.

Personally I use App Engine, re-use HTTP POST protocol and run on standard HTTP port (80) for maximum compatibility with client-side firewall software.

It’s literally few lines of code to parse incoming POST requests, store them in the database and provide a basic web interface for easy browsing. As an additional bonus, App Engine is free if your traffic is small enough,

If you write in PHP and run on a small VPS, it’ll work just as well.

Client side on Windows in C# apps

In C# I setup a global exception handler (in WPF it means setting up handlers for App.DispatcherUnhandledException and AppDomain.CurrentDomain.UnhandledException).

I then use HttpWebRequest class to send exception message and callstack as HTTP POST (using multipart/form-data).

The code is less than hundred lines and took mere hours to write.

Client side on Windows in C++ apps

In C++ everything is substantially harder. The big picture is similar:

install global handler for unhandled exceptions (with SetUnhandledExceptionFilter())
in that handler generate the crash report and HTTP submit it to the server

The details are substantially more complicated. The code is more than a thousand lines and took me days to perfect.

The biggest issue was that, unlike in C#, you don’t get readable callstack in native code. You need symbols (.pdb files) for that. My solution is convoluted, but works:

during build process, I archive .pdb files on the server (I use S3, but any web server will do)
in the crash handler I download the symbols locally, on demand
I create crash report containing human-readable callstacks of all threads and some other useful information
I HTTP POST it to the server

You can re-use my work. The code is part of my SumatraPDF open-source project. Most of it is in CrashHandler.cpp and, unlike most of Sumatra, is under liberal BSD license.

Client side on Mac OS X

The good thing about Mac is that it already creates human-readable crash reports for you. They are stored in ~/Library/Logs/CrashReporter/ directory.

There’s no need to handle crashes yourself. I just check at startup if there’s a new crash report for my app (the files are named after your application) from a previous run. If there is, I submit it to the server (and delete so that it’s not sent multiple times).

The alternatives

While the general idea is always the same, there are different ways of implementing it.

On Windows a simpler solution is to capture so-called minidumps (using MiniDumpWriteDumpProc() Windows API) instead of going to the trouble of generating human-readable crash reports client side.

I did that too. The problem with that approach is that you have to inspect each crash dump manually in the debugger (e.g. WinDBG). I wrote a python script that automated the process (you can script it by launching cdb debugger with the right parameters and making it run !analyze -v)).

Unfortunately, cdb is buggy and was hanging on some dump files. It’s probably possible to work around with a timeout in the python script, but at that point I stopped caring.

Windows provides native support for minidumps. Google took minidump design and provided cross-platform implementation for Windows, Mac and Linux, as part of breakpad project.

Breakpad is the crash reporting system used by Google for Chrome and Mozilla for Firefox. It contains both client and server parts for native (C/C++ or Objective C) code.

I used it once for a Mac app. For Objective C I prefer the approach described above as it’s simpler to implement, but I’m sure that’s a solid and well tested approach.

On Windows, crash reports from your app are already sent to Microsoft as part of Windows Error Reporting. Apparently, it’s possible to for third party developers to get access to those reports but I never did that, so don’t know what’s involved in the process.

The SumatraPDF experience

So how well does it work in practice?

We’ve implemented the system described here in Sumatra 1.5. Sumatra is a rather complicated piece of C++ code and quite popular (several thousand of downloads per day).

Before 1.5 we had a system where we would save the minidump to a disk and after a crash we would ask the user to report it in our bug tracker and attach minidump to the bug report.

It became obvious that almost no one did that. We’ve only gotten few crash reports from users in few months. Our automated system was sending us tens of crash reports per day.

Once we knew about the problems, we could attempt to fix them. Some problems we could fix just by looking at crash report. Some required writing stress tests to make them easier to reproduce locally. Some of them we can’t fix (e.g. because they are caused by buggy printer drivers or other software that injects buggy dlls into our process).

We do know that we fixed some of the bugs. We can see that a new release generates less crashes and by looking at crash reports we can tell that some crashes that happened frequently in previous releases do not happen anymore.

Building automated crash reporting system was the best investment we could have made for improving reliability of SumatraPDF.

Experience porting 4k lines of C code to go

2011-06-01T23:40:38-07:00

I’ve been recently interested in Go, a new programming language.

The best way to learn a language is to use it in a small but real project.

I needed a program that generates HTML from textile format. The way of least resistance would be to implement it in Python, as I’ve coded a similar thing in the past. Instead I decided to use this as an opportunity to get more familiar with Go.

The biggest part of the problem was the textile to HTML conversion. There is no existing Go code for that so I decided to port upskirt C library, as it does the job in the most performant way (it has a hand-written, disciplined parser as opposed to most other solutions that just throw cryptic regular expression at the problem).

The bottom line is: porting C code, at least in this case, was fast, boring, mechanical process (and that is a good thing).

Go’s syntax is heavily inspired by C. The differences that I’ve encountered most frequently:

syntax for declaring variables is different (and better)
while keyword is missing, replaced by a more versatile for syntax
function declaration syntax is different

Fortunately, the transformations were simple and mostly mechanical.

It took me just few days to manually translate around 4000 lines of C code into ~3200 lines of Go code.

Saving ~800 lines of code (20%) is good, but the interesting part is: where do the savings come from?

The core parsing/html generating logic didn’t shrink much. The big savings came from the fact that Go has a built-in growable array type and upskirt C code had to spend 924 lines re-implementing that in C.

An unexpected advantage of Go was its safety. The C code implements parsing by partying on char * pointers. Such code is notorious for causing lots of subtle, hard to test for bugs. Go doesn’t allow this kind of pointer arithmetic and instead provides slices, which are a view into an underlaying array.

Slices provide out-of-bounds checks. Just by recoding in Go I found out-of-bounds access bug in the original C code.

Thanks to similarity of Go and C syntax, porting algorithmic code from C is simple.

All things considered, Go is quickly becoming my new preferred language (taking the crown away from Python). It combines the good attributes of Python (lightweight syntax, garbage collection) with good attributes of C (fast execution thanks to compilation to native code and programmer’s control over memory layout) and adds some unique capabilities of its own (concurrency via gorutines and channels).

BTW: if you want markdown implementation for Go, use blackfriday. It’s also direct port of upskirt and I abandoned my port in favor of contributing to blackfriday, since it was slightly ahead and there’s no need for two nearly identical projects.

SumatraPDF 1.6 released

2011-05-30T21:42:25-07:00

SumatraPDF developers are quite pleased to announce 1.6 release of SumatraPDF, a PDF, XPS, DjVu, CBZ and CBR reader for Windows.

In this release we’ve added support for DjVu file format.

When no document is open, we display a list of frequently read document as thumbnails. This functionality is inspired by new tab page in Chrome.

We’ve added support for displaying Postscript documents. This requires recent Ghostscript version to be already installed - we don’t bundle it ourselves.

We’ve added support for displaying a folder containing images: drag the folder to SumatraPDF window

We now support clickable links and a Table of Content for XPS documents.

We’ve added printing progress and allow canceling printing process.

We’ve added Print toolbar button.

Experimental: we’ve added previewing of PDF documents in Windows Vista and 7. This creates thumbnails and displays documents in Explorer’s Preview pane. Needs to be explicitly selected during install process. We’ve had reports that it doesn’t work on 64-bit Windows which is why we call it experimental.

This is how "frequently read" list looks like:

Easy vs. probable or how to make money with software

2011-05-14T17:15:17-07:00

It it hard to make $1000 a month?

There’s a question on Quora: Is it hard to build, market and maintain a web app that makes at least $1000 a month?

Most of the answers are happily saying "no, it’s not hard at all", usually justifying it with wishful thinking ("it’s not hard to get 50 people pay you $20 a month").

Those answers are misleading.

I don’t blame the respondents. They answered the question the way it was asked.

What you ask vs. what you want to know

The problem is that the person who asked the question really wanted to know how probable it is, not how hard. There’s a big difference.

A personal anecdote

Many years ago, in the early days of Palm PDAs I wrote an English dictionary for Palm OS. For more than 2 years I made several thousands dollars per months from it.

Was it hard? Not at all. It took me about 2 weeks to write the first version and there was nothing especially difficult about it - there are thousands of programmers that could have done it.

My website was simplistic, my marketing skills limited to uploading my app to an app store-like website (PalmGear) which handled selling the app.

Nothing I did was difficult. However, my success wasn’t very probable.

I wrote one of the first (if not the first) dictionaries. Things were different in 2001. Today everyone and their mother jumps on building iPhone or Android apps bandwagons but back then I had no competition from known dictionary publishers (publishers are not risk taking, future embracing organizations).

For a while I had the market to myself and that helped to establish my program as one of the most popular programs for Palm OS. Eventually competition arrived and it did have negative effect on my sales, but they had a really hard time catching up with me.

Lesson number 1: timing matters.

Getting timing right is not hard or easy, it’s simply not likely that you’ll get the timing right. If the right timing is obvious, it’ll be obvious for many people and, paradoxically, means it’s too late. If it’s not obvious, there’s a big probability that whatever you think the right timing is, you’re wrong.

The right timing is only obvious in retrospect.

In my case, I didn’t write my dictionary because I thought it’s the right time to do it if I wanted to make money. I was just excited about new technology (Palm Pilots), wanted to write some program for it and a dictionary seemed like a good choice.

Do it once vs. do it many times

Another way to look at it is repeatability.

The easy success of my app wasn’t lost on me. If I could replicate that success several times, I would get quite rich with little effort.

So I’ve made more apps for Palm OS. They were more complex than my dictionary. They took longer to write, but still weren’t hard to write.

I assume that my marketing, sales and other business skills were at least as good as before. And yet, the other apps were all failures.

My first success was easy but it wasn’t easily repeatable.

Many people who answered the Quora question did build successful websites making $1000 or more a month. Does that mean that their experience is good enough to declare that achieving what they did is easy?

No. Just like me, they all have only one or two successful products. If it’s easy, then why don’t they re-invest their capital in making more products? After all, if you have 100 products each making $1000 per month, you’ll make more than a million a year.

The answers are a good example of selection bias. Even if something has 1% probability of happening, if there are 100 people who tried, 1 of them will succeed. Given the popularity of Quora, it’s not hard to find lots of people who tried something with low probability of success and did it successfully. When they see someone asking if doing it was hard, they jump in saying that it wasn’t.

They are truthful and trying to be helpful and encouraging but we don’t get the complete picture because we don’t hear from much bigger number of people who tried and failed.

The successes might have been easy, but they were not likely.

Just ship it

Making software (be it a web site, a mobile app, a desktop app) that makes money is something with low probability of success. It’s not as bad as you think: making money in any self-directed business (as opposed to being employed) has low probability of success.

More importantly: you have the biggest influence on that probability.

The person who asked this question probably won’t be successful. The answer is unknowable and knowing it would not make a difference.

You’ll succeed or you’ll fail but you’ll not ever know it until you’ve tried. Spending any time calculating your chances actually decreases the probability because it takes away time and energy from things that matter.

The single most important thing when building money making software product is this: ship it.

My success might have been accidental, but it wouldn’t have happen if I didn’t write the code, the website, the documentation and made my program available for people to buy.

If the product fails, either improve it or make another one. The more times you try, the more likely you’re to succeed.

When you start getting sales, double down. The unassailable logic of software business is that it’s easier to make a successful product even more successful than to create another one that is just as successful.

Look at any software business. Even the biggest successes, like Microsoft or Google, have only few extremely profitable products and despite having the brightest developers, mountains of money, cross-promotional opportunities, they struggle to create more money making products.

90% of success is showing up - a proof

2011-05-07T18:05:04-07:00

Woody Allen said that 90% of success is showing up. This article is a proof of that.

In 2007 a Standford professor gave his student a taks: build a Facebook application and get people to use it.

The result? Some applications became so popular that students made good money out of them. Some even turned into companies.

The important part is that none of those student particularly wanted to build that app. It was simply something they were told to do.

The challenge in life is that no one tells us to do things that we’ll own. Sure, at work bosses tell us what to do but the company owns the result of the work and all potential windfalls.

What the article shows that just by doing something and shipping it to the world, we might create an unexpected success.

SumatraPDF 1.5 released

2011-04-23T19:37:04-07:00

A new version of SumatraPDF, a free, small, fast, open-source PDF reader for Windows, is ready.

What’s new in version 1.5?

We’ve added support for XPS documents.

We’ve also added support for CBZ and CBR files (popular formats for comic book files).

We’ve added File/Save Shortcut menu item to create a shortcut to a specific place in a document.

We’ve added right-click context menu for copying text, link addresses and comments. In browser plugin context menu also has items for saving and printing.

We’ve added folder browsing. Ctrl+Shift+Right opens next PDF in the current folder, Ctrl-Shift+Left opens previous PDF in the current folder. Current folder is the one where currently opened PDF documents is located.

We’ve also fixed handling of large PDF files in browser plugin in FireFox.

SumatraPDF is a creation of SumatraPDF developers.

Let the downloads begin.

SumatraPDF 1.4 released

2011-03-12T16:18:46-08:00

Good news everyone: a new version of SumatraPDF, a free, small, fast, open-source PDF viewer for Windows, is ready.

What’s new in this version?

We’ve added browser plugin for Firefox/Chrome/Opera (Internet Explorer is not supported).

We’ve added IFilter that enables full-text search of PDF files in Windows Desktop Search (i.e. search from Windows Vista/7’s Start Menu).

Browser plugin and IFilter are not installed by default so you need to use options in the installer and check the appropriate checkboxes. You can uninstall them by re-running the installer and de-selecting the options.

In 1.3 we’ve improved text selection to use left mouse button. The downside of that was that left mouse button couldn’t be used for scrolling when mouse cursor is over text. To compensate for that, in 1.4 you can scroll with right mouse button.

In 1.3 we’ve introduced a new installer. Some of you missed the ability of choosing a custom installation directory that was not implemented in 1.3. We’ve re-introduced support for non-standard installation directory in 1.4.

SumatraPDF focuses on reading PDFs and doesn’t implement some more advanced functionality like filling out forms or making annotations. If you need this functionality, we’ve made it easy to re-open current document via File menu in Adobe Reader (if it’s installed). In 1.4 we also support Foxit and PDF-XChange.

To make SumatraPDF files smaller, we used to compress the executable with mpress. Unfortunately that caused some anti-virus programs to falsely report Sumatra as a virus. We no longer compress the executables that ship with the installer version, so don’t be surprised that the files are now bigger. The portable, .zip version still ships as a single, compressed executable (so don’t be surprised if it’s flagged by some anti-virus software).

We’ve removed -title cmd-line option.

We’ve added support for AES-256 encrypted PDF files, fixed an integer overflow reported by Jeroen van der Gun and and made other fixes and improvements to PDF handling.

If you wonder why we don’t have browser plugin for Internet Explorer, the explanation is simple: no-one has written the necessary code.

SumatraPDF has been brought to you by SumatraPDF developers.

XML is really, really slow

2011-03-10T18:25:56-08:00

It’s about 8 years late but W3C finally produced a standard for efficient XML representation. They call it somewhat opaquely Efficient XML Interchange (EXI). I guess it sounds better than what we used to call such things back in the day: binary XML (which is a better description what it’s about).

The money quote from the announcement:

They’ve achieved over 100-fold performance improvements...

One way to interpret this is: wow, those guys are really smart.

There is much simpler explanation: XML is really, really slow.

Speed comes from architecture

I’m somewhat performance oriented in my programming work. One of the reasons I disliked the popularity of XML was that I saw how often it blinded people to engineering realities. Choosing XML was often a reason for dramatic performance issues that then had to be heroically recovered.

Those performance problems were, however, mostly self inflicted. XML is only one of the possible ways to store or exchange data but it certainly is one of the slowest.

You cannot get 100x speed up over technology that wasn’t incredibly inefficient to begin with.

Sometimes you don’t have the choice (when you have to inter-operate with systems that only offer XML) but I’ve seen many cases where people did have the choice and made the wrong one.

XML isn’t such a hot buzzword anymore so I don’t see that problem as often but I still do see it.

For example, more than one text editor decided to store the syntax highlighter definitions as XML while it can be trivially stored in simple text format that can be parsed much faster, using less memory and, as a bonus, can actually be edited by human beings.

A bigger point is: speed comes from architecture. It’s not that choosing the right architecture will make you dramatically better but choosing the wrong architecture will make you dramatically worse.

Choosing XML over more efficient ways of storing data is just one, but particularly frequent, example of that rule.

Binary XML (EXI) is too little too late

As to the standard itself: don’t pay attention to it. It’s too little, way too late.

About 8 years ago I did my small part in expanding XML capabilities of SQL Server. At the time XML was hot so Oracle, Microsoft, IBM raced to add native XML handling to their databases. It was going to be the next big thing. Possibly even bigger than big.

Trust me, the ridiculous inefficiency of XML wasn’t lost on developers working on XML technologies 8 years ago. Coming up with a more efficient binary XML format is easy. Microsoft had its version (if not several of them) and other companies had theirs.

The real problem was: politics prevented people agreeing on a standard so no standard emerged.

There was an eruption of creating standards based on XML (SOAP, XML Schema, XQuery). If you don’t know what those terms mean it’s because they all failed (despite the fact that everyone was convinced they’re going to be the next big thing).

In hindsight it was a terrible mistake to work on those big standards of speculative value but not solve a real problem people already had: an efficient, binary, standard format for storing XML.

If W3C came up with EXI 8 years ago, maybe people wouldn’t feel the need to invent Protocol Buffers or Thrift and it would have won.

But solving this problem today is almost comically late. It has no chance of adoption.

If you’re planning to use a custom, binary way of storing XML, using EXI is probably better but there’s no way EXI will become so universally supported as XML and universal support (in software libraries, books etc.) is really the main thing XML has going for it (speed or human-readable syntax, on the other hand, are not XML’s strengths).