Debuggable.com - Blog

Testing node.js modules with Travis CI

felix@debuggable.com — Fri, 18 Nov 2011 09:17:12 +0000

You have written a node.js module lately? It has a test suite? Awesome! Time to get yourself a nerd badge of honor:

But hang on nerdy warrior, this precious award has to be earned. So go ahead and check out the sweetness that is Travis CI. Travis is an open source, free to use, continuous integration server. Initially it was just building ruby stuff, but these days it supports a ton of other languages, including node.js.

And luckily, getting travis to run your tests on every GitHub push is really easy as well:

Step 1: Go to Travis and login/connect with your GitHub account.

Step 2: Hover over your name on the top right, and select "Profile" from the dropdown.

Step 3: You should see all your GitHub projects. Flip the "Off" switch to "On" for a node.js project you want to use with travis.

Step 4: Add a .travis.yml file to your project with the following:

language: node_js
node_js:
  - 0.4
  - 0.6

Step 5: Make sure your package.json has something like this:

"scripts": {
"test": "make test"
},

Step 6: Git push, and watch travis building your project on the home screen!

Step 7: Assuming your tests are passing, it is time to get your badge of honor. Adding it to your GitHub Readme.md is as simple as:

[![Build Status](https://secure.travis-ci.org/<GITHUB_USER>/<REPO_NAME>.png)](http://travis-ci.org/<GITHUB_USER>/<REPO_NAME>)

If you want to see an example of what this looks like, and you also happen to be in the market for some no-bullshit testing tools, check out my new libs:

utest: The minimal unit testing library.
urun: The minimal test runner.

That's it. And in case you are not excited enough yet, go and check out the Travis Docs to discover additional goodies like how to work with databases, etc.

--fg

Private npm modules

felix@debuggable.com — Thu, 08 Sep 2011 14:18:26 +0000

Thanks to Isaac, npm is getting more and more awesome by the hour. One of the coolest recent additions (you need at least v1.0.26) is the ability to specify private git repositories urls as a dependency in your package.json files.

At transloadit, we are currently using the feature to move some of our infrastructure code into separate packages, allowing for those to be tested and developed in isolation making our core application easier to maintain and work on.

The syntax for referencing a git repository (and commit) is as follows:

{
"name": "my-app",
"dependencies": {
"private-repo": "git+ssh://git@github.com:my-account/node-private-repo.git#v0.0.1",
}
}

This will include a private npm module called "private-repo" from GitHub. The url also contains an optional refspec (#v0.0.1) that tells npm which branch, commit, or in this case tag you want to have checked out.

Now of course this is not the only way to do private npm repositories, but it is much simpler than running your own registry, so I would recommend it to most people.

Before you head of to play with this, here is a final tip that may safe you some headaches. In all your private npm modules, add "private": true to your package.json. This will make sure npm will never let you accidentally publish your secret sauce to the official npm registry.

Happy hacking, --fg

PS: When deploying, don't forget that you need to authorize the servers ssh key for the GitHub repository you are depending on.

How to fork & patch npm modules

felix@debuggable.com — Tue, 26 Jul 2011 13:15:03 +0000

With now more than 3000 modules, there are huge gaps in the quality of things you find in the npm registry. But more often than not, it's easy to find a module that is really close to what you need, except if it wasn't for that one bug or missing feature.

Now depending on who is maintaining this module, you may get the problem resolved by simply opening a GitHub issue and waiting for a few days. However, open source doesn't really work without community, nor do you always want to be at the mercy of someone else. So a much better approach is to actually roll up your sleeves, and fix the problem yourself.

Here is the proper way to do this while using npm to manage your forked version of the module:

Fork the project on GitHub
Clone the fork to your machine
Fix the bug or add the feature you want
Push your commits up to your fork on GitHub
Open your fork on GitHub, and click on the latest commit you made
On the page of that commit, click on the "Downloads" button
Right click on the "Download .tar.gz" button inside the popup, and copy the link ("Copy Link Address" in Chrome)
Open up your package.json file, and replace the version number of the module with the url you just copied
Send a pull request upstream (Optional, but this way you will avoid having to maintain that patch of yours against newer versions of the module you forked)

Example: My new airbrake module uses a forked version of xmlbuilder. I submited my fix as a pull request, but it has not been merged yet. In order to pull in my changes via npm anyway, I simply pointed my package.json to the download url of my fork on GitHub like so:

"dependencies": {
"request": "1.9.8",
"xmlbuilder": "https://github.com/felixge/xmlbuilder-js/tarball/4303eb2650a4b819a980b1dc9d2965862a1e9faf",
"stack-trace": "0.0.5",
"traverse": "0.4.4",
"hashish": "0.0.4"
},

Alright, let me know if this is helping your node.js adventures, or if you have an alternative workflow you are using. Otherwise, happy hacking!

--fg

PS: You should upgrade to the latest npm version first, some older versions had problems with handling url dependencies properly.

Node.js Workshop in Cologne, June 10th

felix@debuggable.com — Fri, 20 May 2011 13:26:03 +0000

We apologize for the short notice, but if you are looking to put node.js in production, this full day node.js workshop we are organizing is where it's at!

The workshop is happening on Friday June 10, one day before nodecamp.eu. Space is limited to 15 people and expected to sell out quickly.

As a reader of our blog, you can get a 15% discount on the regular ticket by using the code 'debuggable'.

Should you attend?

This workshop will teach you everything you need in order to write and deploy powerful node.js applications. We'll try to cover a lot of ground, so if you are interested in any of the following, you should definitly attend:

Setting up node.js on your local machine
Understanding the module system
Using npm for installing and upgrading modules
Publishing your own npm modules
Everything you need to know about http.Server
Structuring your code using OOP in JavaScript
Dealing with all the callbacks in a sane fashion
Using the same code in node.js and the browser
Building realtime apps with Socket.IO
Using the express framework
An overview over testing tools available for node.js
Deploying node.js to Ec2 / Joyent

The first half of the day will be guided by slides (which will be made available afterwards), with the second half being a hands-on session where we will build a small node app from scratch.

About the instructor

This workshop will be led by Felix Geisendörfer (that's me). He is one of the earliest and most active contributors to node.js, author of over 20 npm modules and also running one of the biggest node.js applications in production over at transloadit.com.

In addition, Tim Koschützki who is also a co-founder at transloadit, will be available all day to help with individual questions and trouble shooting.

Questions & More Workshop

If you have additional questions or can't make it to this workshop, please head over to the workshop page which has information on other upcoming workshops and questions.

--fg

Why are you not using a SSD yet?

felix@debuggable.com — Wed, 23 Feb 2011 22:15:01 +0000

If you are a developer, and you have not switched to a SSD yet, what is your excuse?

Let me explain. I've switched to an SSD a little over a week ago, and it's a different world. You know that feeling of having just bought & setup a new machine and everything still runs very fast? Well, a SSD will make every single day feel just like that, except much faster.

But I already knew that, so why has it taken me, and apparently you who is reading this, so long?

Well, my main problem was that I have a few big things on my hard disk, namely music, photos and virtual machine images. This means that I need a hard disk of ~300 GB to work comfortably. However, the SSD I was interested in only comes in 40, 60, 120, 240 and 480 GB. The 480 GB costs ~$1.580 right now.

A 240 GB SSD costs ~$520 which seems much less outrageous, but unfortunately that's still too small if it was my only disk.

So for a while, I thought I'd have to wait another 1 - 2 years before enjoying the SSD experience. That was until I came across this article which explained that you could replace your MacBook Pro's optical drive with an SSD. This means I could add an SSD to my machine without giving up the luxury of cheap mass storage.

With this in mind, I decided to get a 120 GB SSD, which is plenty of space for my core system and applications. I followed a few youtube videos for swapping out the disks, and I also placed my previous hdd in the optical bay slot since I've heard reports of hibernation problems if you put your primary disk there.

Making the new SSD my primary hard disk was easy as well. My initial attempt using time machine failed, so I simply booted up my system from the old primary hdd, and used carbon copy cloner to copy all data (excluding my music, images and vms) to my new SSD. After that I made the SSD my primary boot disk using the "Startup Disk Preference Pane" and rebooted. The whole operation took about 1-2 hours.

So how has this changed my life? First of all, boot time is incredible. Compared to Tim's mac (which is now scheduled for an upgrade ASAP as well), my machine goes from 0 to starting Photoshop in 48 seconds. Tim's machine takes 2 minutes and 50 seconds. Note: It takes about the same time for both machines to boot the kernel, but my machine is instantly ready at that point now.

Starting programs is either instant or 2-3 times faster than before. Recursive grep (using ack) is insanely fast / instant, even on big project. And git - it's a different world. If you've ever waited for minutes while running 'git gc' on a big project, an SSD turns this to seconds. Everything feels just incredibly fast.

With this in mind, what's your excuse for not treating yourself to a SSD now?

--fg

PS: If you think you would miss your optical drive: You can get an external USB one for ~$40 on Amazon. If you really need the internal one back, I guess it would take you about ~10-15 minutes to put it back in once you know the procedure.

PPS: If you're worried about the difficulty of replacing the disk: It's very easy, all you need to know is how to operate a screw driver. However, make sure you've got the right tools. The OWC disk I'm recommending comes with a set of tools if you order it with the data doubler for the optical bay.

PPPS: My friend Joel pointed out the lack of TRIM support in OSX as a reason for not getting an SSD yet. That's a valid argument, but the OWC discs do not suffer from the lack of TRIM.

Talks, talks, talks

felix@debuggable.com — Fri, 18 Feb 2011 15:58:33 +0000

I've been in Atlanta for the past two weeks, and thanks to the kind help of a few folks, I was able to present at 2 meetups, as well as Startup Riot 2011.

First up was a new talk at the Atlanta Ruby Meetup:

Nodejs - Should Ruby Developers Care?

The talk was an attempt of taking out all the cool-aid and hype and focusing on node's true strength as well as weaknesses. I think the whole thing was very well-received, but I certainly could have done a little better on the delivery.

Download: nodejs-should-ruby-developers-care.pdf (733 KB)

Next up was the 4th edition of my general introduction talk to node.js:

Nodejs - A quick tour (v4)

This version of the talk was updated for the freshly released v0.4, and I've also tweaked some other slides to the point where I'm very happy with it. It seems to do a great job getting people excited about node, as well as highlighting sensible use cases.

Download: nodejs-a-quick-tour-v4.pdf (610 KB)

And last but not least, I had the chance to do a 3 minute pitch for Transloadit at Startup Riot:

Transloadit at Startupriot

Download: transloadit-startupriot.pdf (640 KB)

Doing the actual presentation was quite scary. I have never given a talk this short, you basically don't get any time to warm up and get into things. You got to go out, and give your best right away. Having an audience of ~500 people didn't help either.

However, I think I pulled it of fairly well. My main message was: "Save the time, save the money, save the shrink - use transloadit", and I highlighted some of the cooler aspects of our service such as the realtime encoding. Lots of people came by to our table afterwards to find out more about the service, including a few VCs and angels (we're not looking for investment right now, but seeing their interest feels good regardless : ).

There are a few more talks coming up in the next couple of months, but I also hope to find some more time for actual blogging again. I certainly want to start writing a few articles about testing JavaScript.

--fg

Slides: Node.js in production

felix@debuggable.com — Fri, 21 Jan 2011 16:51:48 +0000

Last night I was giving a talk about node.js in production at the Berlinjs user group.

In the talk I shared some of our experiences at Transloadit, but also tried to cover the topic from a general angle.

I concluded that running node apps in production comes with certain challenges, most notably the lack of useful stack traces when things go wrong. But that's something Ryan is working on, and a little unit testing can go a long way to avoid most bugs : ).

Download: nodejs-in-production.pdf

Slideshare: Node.js in production

View more presentations from Felix Geisendörfer.

Let me know if you have any questions!

--fg

Node.js Artikel im t3n Magazin

felix@debuggable.com — Mon, 29 Nov 2010 09:31:48 +0000

In der aktuellen Ausgabe des t3n Magazins gibt es einen von mir verfassten Artikel zum Thema node.js:

Schubrakete für JavaScript - Wie Node.js JavaScript auf dem Server revolutioniert

Der Artikel richtet sich vor allem an komplette Neulinge, eignet sich von daher aber super um mit dem noch ein oder anderen skeptischen Kollegen ins Gespräch zu kommen.

Für alle die an einer kostenlosen Ausgabe des Heftes interessiert sind: Einfach den link re-tweeten und mir dann eine E-Mail schicken. Die ersten 3 tweets bekommen ein Exemplar per Post.

--fg

Test driven development at Transloadit

felix@debuggable.com — Thu, 28 Oct 2010 12:23:53 +0000

You have probably heard a few talks or read a few articles where test driven development is depicted as a magic unicorn that watches over your software and makes everybody happy. Well, about 18.000 lines of "magic unicorn" code later, I'd like to put things into perspective.

The truth is, test driven development is a huge pain in the ass. Writing those damn tests all the time takes a huge amount of discipline, and it doesn't really get a whole lot easier with time.

But do you know what sucks more? The liability that comes without those tests.

Let me clarify. I'm not trying to tell you that you should practice test driven development. What I will try to do however, is to give you a realistic idea about the work it takes and the kind of benefits you get from it.

At this point I basically think of test driven development as an insurance policy. It guarantees that your software will have a certain pre-defined quality mostly measured by the amount of bugs and the risks of changing the software. It can also reduce the costs of your main developer getting run over by a bus, or the same thing happening to the API of the platform you are building on.

However, there are also millions of people on this planet that live without any insurances whatsoever, so clearly this is not the only way of getting from point A to B. It all depends on your liability and how you can / want to deal with it.

At Transloadit 100% of our code is developed test driven. The main reason for that is that we are building on node.js which still is a very risky technology at this point and probably our biggest liability and advantage at the same time. Another big risk are third party tools such as ffmpeg and image magick that are insanely dangerous to upgrade since they love to change APIs and behavior all the time. And last but not least, some of the stuff we are doing is rather complex.

Now let me explain how test driven development lets us deal with those risks. First of all, new features always start out as a system test. Each system test boots up the entire REST service, sends an actual HTTP request to it, and evaluates the response as well as the files that were created in the process. Files are verified by looking at their meta data, as well as performing visual diffs on images against expected fixtures. For videos we are using thumbnails to do the visual comparisons.

Once that test is written (and failing), we start writing unit tests. Let me explain what we mean by "unit". A unit to us is the tiniest atomic piece of functionality we can possibly test. As soon as there is a function call, creation of a class, or even a callback closure - we stub it. Everything that can be isolated and tested separately is handled by itself. As you can imagine, that's the part where the pain and discipline come in.

However, there is a certain undeniable beauty resulting from it. Writing those tests feels like proofing mathematical theorems. Every step is just small enough to be broken down to raw logic. Everything builds upon the previous step. There is no need to test all numbers in between 1 - 1000000, unless there is a logical reason for expecting a different result.

Of course there is a good chance of you going crazy in the process and losing track of the original goal. That's where the system test comes to rescue. Whenever you don't know what needs to be done next, you simply run the system test and it will tell you what's missing.

Is this process perfect? No. We have had a few bugs here and there, but I can still count them on two hands. Last weekend we pushed a rather insane change into production: We replaced our underlaying MySql driver (we were using that of PHP, don't ask how that worked) with node-mysql. I've been working on node-mysql with the same care and TDD-masochism as we have on transloadit, so the update went without a single hickup (so far). Not bad for a change that involved 6000 lines of code.

And that's it. Our service is not necessarily better than our competitors' because we use TDD. However, our risk of breaking things is much smaller, and we can perform heart-surgeries like this without breaking a sweat. This gives us the confidence to charge money, and the ability to spend the rest of our limited resources (we are bootstrapping) on innovative new features rather than dealing with the shortcomings of our software all the time.

So, is TDD right for you? It depends. Writing code the TDD way requires a ton of discipline, and the unicorns aren't always around to make you feel better. So make sure you have good reasons to do it. Sometimes a few system tests can get you 80% of the results for 20% of the work. Other times you need the remaining 20%. Also keep in mind that we're lucky at Transloadit for using a flexible language and most of our interfaces are JSON. This makes testing cheaper for us than it might be for you.

--fg

PS: If I find some more time, I'd love to go into more details of our node.js unit testing. Leave a comment if you're interested. Meanwhile make sure to check out the tests for node-mysql, they are done exactly like our internal tests.

Node.js slides from @rug_b meetup last night

felix@debuggable.com — Fri, 08 Oct 2010 08:25:26 +0000

Last night I had the honor of speaking about node.js in front of Berlin's ruby community at the @rug_b user group meetup.

The meeting was well attended (~50 geeks), but wooga did a great job hosting the event and even let us use their main room after it became clear that we couldn't fit everybody in the initial room.

The slides of my talk are available on slideshare and as a pdf download. For those of you who have seen the previous version of the talk, this is a very much updated version. Basically I had to replace all previous "The Future" slides with new content since node has implemented all of those things by now : ).

Anyway, Tim and I had a great time talking to everybody and also enjoyed the other talks about Mongrel2 and Phusion Passenger 3.

--fg