good coders code, great reuse

I published another 10 of my projects to GitHub

2013-04-17T16:20:28Z

Remember my previous two blog posts about publishing 30 and 20 of my projects to github? Here are another 10 projects in no particular order that I've written since then.

I really love writing open source projects. Everyone should publish all their projects to github. Good or bad. Like my friend just said: "Publish more open source, not less. "Too much noise" is a terrible excuse. Publish wisdom so we may use. Publish mistakes so we may learn."

If you find my projects interesting, consider following me on github! Thank you!

Shorten urls with bitly without api

This is a Perl module that shortens urls via bitly without using their API. I once had to shorten more than a few hundred urls quickly so I wrote this.

Here's an example of how to use it:

use bitly;

my $bitly = bitly->new('username', 'password');
my $url = bitly->shorten('http://www.url.com');

unless ($url) {
    say $bitly->{error};
}
else {
    say $url;
}

bitly shortener on github

Speak text files to wav via Microsoft Speech API

I once had this idea of converting my blog posts to mp3. So I wrote a C++ program that uses the Microsoft's speech api. It takes a txt file as input and produces a wav file output.

Usage:

speak.exe <voice name> <text file> <wav file>
-or-
speak.exe --list-voices

Compile this project with Microsoft Visual Studio 2008 or later.

speak-text-files-to-wav on github

Bonus: At first I tried using Loquendo SDK and created a TextToWav project but their speech engine wasn't too great so I created speak-text-files-to-wav.

TextToWav on github

Load status server for Windows

This simple C++ program creates a single threaded TCP server that replies with information in JSON format about Window's CPU, Disk, and Memory usage. I created this because I had to monitor Windows servers at Browserling.

Here's an example:

C:\bin\> load-status-server.exe
Started server on port 7000

Now if you connect to server.com:7000, it will send you JSON with the current server load info:

$ nc server.com 7000
{
    "memory": {
        "usage": 71,
        "total_physical": 1068388352,
        "free_physical": 302112768,
        "total_paging_file": 3096842240,
        "free_paging_file": 2155446272,
        "total_virtual": 2147352576,
        "free_virtual": 2130919424,
        "free_extended_virtual": 0
    },
    "cpu": {
        "load": 5
    },
    "disk": {
        "free_user": 22858027008,
        "free_total": 22858027008,
        "total": 42947571712
    }
}

Compile this project with Visual Studio or mingw. It uses just Win32 calls.

load-status-server on github

HWND finder

We started working on a new product at Browserling that takes screenshots so I wrote a bunch of code that finds browser window HWNDs. Then I decided to open source some parts of it and created HWND finder. The idea is to have something like jQuery's syntax for finding HWNDs. We delayed launching this screenshot product so I haven't made any updates.

Here's an example:

#include "hwnd-finder.h"

HwndFinder hf;
HWND rendererHwnd = hf.find("Chrome_WidgetWin_1 > Chrome_WidgetWin_0 > Chrome_RenderWidgetHostHWND");

This finds Chrome renderer's window handle. Here Chrome_RenderWidgetHostHWND is a child of Chrome_WidgetWin_0 is a child of Chrome_WidgetWin_1 which is the top window.

hwnd-finder on github

node-number-range

This is a node.js module that streams number ranges. Here are all the ranges it supports:

var range = require('number-range');

* range(10) - range from 0 to 9
* range(-10, 10) - range from -10 to 9 (-10, -9, ... 0, 1, ... 9)
* range(-10, 10, 2) - range from -10 to 8, skipping every 2nd element (-10, -8, ... 0, 2, 4, 6, 8)
* range(10, 0, 2) - reverse range from 10 to 1, skipping every 2nd element (10, 8, 6, 4, 2)
* range(10, 0) - reverse range from 10 to 1
* range('5..50') - range from 5 to 49
* range('50..44') - range from 50 to 45
* range('1,1.1..4') - range from 1 to 4 with increment of 0.1 (1, 1.1, 1.2, ... 3.9)
* range('4,3.9..1') - reverse range from 4 to 1 with decerement of 0.1
* range('[1..10]') - range from 1 to 10 (all inclusive)
* range('[10..1]') - range from 10 to 1 (all inclusive)
* range('[1..10)') - range grom 1 to 9
* range('[10..1)') - range from 10 to 2
* range('(1..10]') - range from 2 to 10
* range('(10..1]') - range from 9 to 1
* range('(1..10)') - range from 2 to 9
* range('[5,10..50]') - range from 5 to 50 with a step of 5 (all inclusive)
* range('10..') - infinite range starting from 10
* range('(10..') - infinite range starting from 11

Very cool stuff. Especially the infinite ranges, which use the process.nextTick trick.

node-number-range on github

HTTP::Async::Retry

It's almost HTTP::Async::Retry. It's actually just a async_retry.pm file that you can drop into your project to do a quick hack. With a bit of effort it could be HTTP::Async::Retry.

I once had to scrape a lot of information so I used my favorite language Perl and used the HTTP::Async module. A lot of URLs would time out as I was creating thousands of connections per second. At first I simply copied the retry code from hack to hack but then at one moment I had enough so I simply wrote async_retry.pm that abstracts away the retries.

Here's an example:

use warnings;
use strict;

use HTTP::Request;
use async_retry qw/async_retry/;

my @urls = (
    'http://www.google.com/1';,
    'http://www.google.com/2';,
    'http://www.google.com/3';,
    'http://www.google.com/';,
    'http://www.google.com/5';,
);

async_retry(
    {
        retries => 5
    },
    [
        map { HTTP::Request->new(GET => $_) } @urls
    ],
    sub {
        my ($req, $res) = @_;
        print $res->base, "\n";
    }
);

This code tries to get all those Google urls and retries to get them 5 times. If a url succeeds or fails after retries, it calls the callback with HTTP::Request and HTTP::Response objects.

http-async-retry on github

HTML Keyboard Widget

This is just an on-screen keyboard widget for Browserling. I wrote it because people with weird keyboard layouts couldn't input various English characters in Browserling. We'll add it to Browserling soon (it's a planned feature.)

Here's how it looks like:

You can try a live demo here.

keyboard on github

Cached browser badges

This project just creates cached browser badges for Testling, so that we don't have to generate them again as it's costly.

browser-badge-cached on github

~/.ssh/authorized_keys ssh key manager

This project manages public ssh keys in ~/.ssh/authorized_keys file. We'll use this at Browserling to manage the ssh keys for tunnels so that you can add, remove and list the keys. (It's a planned feature.)

Here's an example:

var sshManager = require('ssh-key-manager');
sshManager.addKey('pkrumins', 'ssh-rsa AAAAB3NzaC1y...', function (err) {
    if (err) {
        console.log(err);
        return;
    }
});

ssh-key-manager on github

node-tree-kill

This is a node.js module that kills all processes in the process tree, including the given root process.

Here's an example:

var kill = require('tree-kill');
kill(301, 'SIGKILL');

In this example we kill all the children processes of the process with pid 301, including the process with pid 301 itself.

This module currently works on Linux only as it uses ps -o pid --no-headers --ppid PID to find the parent pids of PID.

node-tree-kill on github

GitHub is awesome!

Push all your projects to github all the time! Don't let your project rot on your hard drive! Publish it to github! Publish wisdom so we may use. Publish mistakes so we may learn.

And just another reminder, I'd love if you followed me on github and twitter! :)

A Simple LD_PRELOAD Tutorial

2013-04-01T10:13:40Z

This is going to be a super short and super simple tutorial for beginners about LD_PRELOAD. If you're familiar with LD_PRELOAD, you'll learn nothing new. Otherwise keep reading!

Did you know you could override the C standard library's functions (such as printf, fopen, etc) with your own version of these functions in any program? In this article I'll teach you how this can be done through the LD_PRELOAD environment variable.

Let's start with a simple C program (prog.c):

#include <stdio.h>

int main(void) {
    printf("Calling the fopen() function...\n");

    FILE *fd = fopen("test.txt","r");
    if (!fd) {
        printf("fopen() returned NULL\n");
        return 1;
    }

    printf("fopen() succeeded\n");

    return 0;
}

The code above simply makes a call to the standard fopen function and then checks its return value. Now, let's compile and execute it:

$ ls
prog.c  test.txt

$ gcc prog.c -o prog

$ ls
prog  prog.c  test.txt

$ ./prog
Calling the fopen() function...
fopen() succeeded

Now let's write our own version of fopen and compile it as a shared library:

#include <stdio.h>

FILE *fopen(const char *path, const char *mode) {
    printf("Always failing fopen\n");
    return NULL;
}

Let's call this file myfopen.c, and let's compile it as a shared library:

gcc -Wall -fPIC -shared -o myfopen.so myfopen.c

Now we can simply modify LD_PRELOAD:

$ LD_PRELOAD=./myfopen.so ./prog
Calling the fopen() function...
Always failing fopen
fopen() returned NULL

As you can see the fopen got replaced with our own version that is always failing. This is really handy if you've to debug or replace certain parts of libc or any other shared library.

Next time I'll write about how the LD_PRELOAD works internally.

Visualization of Regular Expression Character Classes

2013-02-25T19:28:13Z

We all know the regular expression character classes, right? There are 12 standard classes:

[:alnum:]  [:digit:]  [:punct:]
[:alpha:]  [:graph:]  [:space:]
[:blank:]  [:lower:]  [:upper:]
[:cntrl:]  [:print:]  [:xdigit:]

But have you seen a visual representation of what these classes match? Probably not. Therefore I created a visualization that illustrates which part of the ASCII set each character class matches. Call it a cheat sheet if you like:

small version, large version

A bunch of programs that I used

Just for my own reference, in case I ever need them again, here are the one-liners I used to create this cheat sheet:

perl -nle 'printf "%08b - %08b\n", map { hex "0x".(split / /)[0], hex "0x".(split / /)[1] } $_ '
perl -nle 'printf "%03o - %03o\n", map { (split / /)[0], (split / /)[1] } $_'

And I used this perl program to generate and check the red/green matches:

use warnings;
use strict;

my $red = "\e[31m";
my $green = "\e[32m";
my $clear = "\e[0m";

my ($start, $end) = @ARGV;

die 'start or end not given' unless defined $start && defined $end;

my @classes = qw/alnum alpha blank cntrl digit graph lower print punct space upper xdigit/;

for (map { chr } $start..$end) {
    for my $class (@classes) {
        print "${green}1${clear}" if /[[:$class:]]/;
        print "${red}0${clear}" unless /[[:$class:]]/;
    }
    print "\n"
}

Credits

I was inspired to create this visualization when I saw a similar table for C's ctype.h character classification functions.

TCP Traceroute

2013-02-25T02:26:06Z

Did you know you could traceroute over the TCP protocol?

The regular traceroute usually uses either ICMP or UDP protocols. Unfortunately firewalls and routers often block the ICMP protocol completely or disallow the ICMP echo requests (ping requests), and/or block various UDP ports.

However you'd rarely have firewalls and routers drop TCP protocol on port 80 because it's the web's port.

Check this out. Let's try to traceroute www.microsoft.com using ICMP protocol:

# traceroute -I www.microsoft.com  
traceroute to www.microsoft.com (65.55.57.27), 30 hops max, 60 byte packets
 1  50.57.125.2 (50.57.125.2)  0.552 ms  0.647 ms  0.742 ms
 2  core1-aggr701a-3.ord1.rackspace.net (184.106.126.50)  0.415 ms  0.555 ms  0.653 ms
 3  corea.ord1.rackspace.net (184.106.126.128)  0.707 ms  0.873 ms  0.984 ms
 4  bbr1.ord1.rackspace.net (184.106.126.147)  1.345 ms  1.341 ms  1.337 ms
 5  * * *
 6  204.152.140.33 (204.152.140.33)  3.614 ms  3.747 ms  3.244 ms
 7  xe-0-2-0-0.ch1-96c-2b.ntwk.msn.net (207.46.46.49)  3.319 ms  4.019 ms  4.010 ms
 8  ge-7-0-0-0.co1-64c-1a.ntwk.msn.net (207.46.40.94)  53.543 ms  53.105 ms  53.074 ms
 9  xe-5-2-0-0.co1-96c-1b.ntwk.msn.net (207.46.40.165)  52.942 ms  52.710 ms  52.670 ms
10  * * *
11  * * *
12  * * *
13  * * *

We get lots of * * * and we've no idea how the packets reach www.microsoft.com.

Now let's try UDP traceroute:

# traceroute -U www.microsoft.com
traceroute to www.microsoft.com (65.55.57.27), 30 hops max, 60 byte packets
 1  50.57.125.2 (50.57.125.2)  0.529 ms  0.599 ms  0.662 ms
 2  core1-aggr701a-3.ord1.rackspace.net (184.106.126.50)  0.480 ms  0.571 ms  0.658 ms
 3  corea.ord1.rackspace.net (184.106.126.128)  0.507 ms corea.ord1.rackspace.net (184.106.126.124)  0.463 ms  0.569 ms
 4  bbr1.ord1.rackspace.net (184.106.126.145)  1.345 ms  1.322 ms  1.290 ms
 5  * * *
 6  * 204.152.140.35 (204.152.140.35)  2.697 ms *
 7  xe-0-2-0-0.ch1-96c-2b.ntwk.msn.net (207.46.46.49)  3.665 ms ge-7-0-0-0.co1-64c-1a.ntwk.msn.net (207.46.40.94)  53.363 ms  52.597 ms
 8  xe-3-1-0-0.co1-96c-1b.ntwk.msn.net (207.46.33.190)  52.284 ms  52.643 ms xe-0-1-0-0.co1-96c-1a.ntwk.msn.net (207.46.33.177)  52.665 ms
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *

Same. Finally let's try traceroute over TCP protocol port 80:

# traceroute -T -p 80 www.microsoft.com
traceroute to www.microsoft.com (65.55.57.27), 30 hops max, 60 byte packets
 1  50.57.125.2 (50.57.125.2)  0.540 ms  0.629 ms  0.709 ms
 2  core1-aggr701a-3.ord1.rackspace.net (184.106.126.50)  0.486 ms  0.604 ms  0.691 ms
 3  corea.ord1.rackspace.net (184.106.126.128)  0.511 ms corea.ord1.rackspace.net (184.106.126.124)  0.564 ms  0.810 ms
 4  bbr1.ord1.rackspace.net (184.106.126.147)  1.339 ms  1.310 ms bbr1.ord1.rackspace.net (184.106.126.145)  1.307 ms
 5  chi-8075.msn.net (206.223.119.27)  3.619 ms  2.560 ms  2.528 ms
 6  * 204.152.140.35 (204.152.140.35)  3.640 ms *
 7  ge-7-0-0-0.co1-64c-1a.ntwk.msn.net (207.46.40.94)  52.523 ms xe-0-2-0-0.ch1-96c-2b.ntwk.msn.net (207.46.46.49)  3.825 ms xe-1-2-0-0.ch1-96c-2b.ntwk.msn.net (207.46.46.53)  3.355 ms
 8  xe-0-1-0-0.co1-96c-1a.ntwk.msn.net (207.46.33.177)  61.042 ms  61.032 ms  60.457 ms
 9  * * xe-5-2-0-0.co1-96c-1b.ntwk.msn.net (207.46.40.165)  100.069 ms
10  65.55.57.27 (65.55.57.27)  53.868 ms  53.038 ms  52.097 ms

A full network path to www.microsoft.com!

There are various different traceroute implementations and if your system doesn't have one that supports tcp protocol, I suggest you either get the new modern implementation of traceroute, or get the tcptraceroute by Michael Toren.

Browserling now has Internet Explorer 10!

2013-01-22T21:44:08Z

Just a quick update on Browserling. We just launched IE 10s! IE 10s are available to all our paying customers. The free browserling version offers IE 9.

Internet Explorer 10.0.9200.16466

Next we're getting Mac OSX browsers and iPhone emulators. Stay tuned! Follow us on twitter for updates - @browserling, @testling, @substack, @pkrumins.

Try Browserling!

Browserling is an interactive cross-browser testing tool. It lets you test your website in older Internet Explorers such as IE 6, IE 7, IE 8 and IE 9, IE 10, Firefox, Chrome, Opera and Safari!

Idea for later - a 1:1 teaching meetup network

2013-01-21T18:01:59Z

I thought I'd share this idea I've had for a while and that I want to build in the future. It's a 1:1 teaching network where people meet up and have 1:1 sessions teaching each other things they don't know. I've found that 1:1 teaching sessions are much more effective than a group teaching session or learning on your own. In a 1:1 teaching session you can keep asking the other person questions instantly and infinitely and you can learn really advanced topics really quickly.

Here's an example of how this meetup network would work. Let's say I'm an expert in Python. And let's say you're an expert in Ruby. And let's say you want to master Python and I want to master Ruby. So we schedule a meet up and I teach you Python for a few hours, and you teach me Ruby for a few hours. And we rescheduling meetups whenever we're both available.

The way I imagine it could be built is having a website where you indicate where you're at, what you're a master at, indicate what you want to master, and indicate the time that you've available. The system then matches you with someone and suggests a meetup.

I'd teach and learn and teach and learn forever this way!

There are so many other ideas that I want to build and I'll just keep sharing them in my "ideas for later" blog post series! See ya!

How to write Testling-CI tests

2013-01-19T00:32:46Z

Writing testling-ci tests is super easy. The only thing your test runner needs to do is to produce TAP output on console.log or process.stdout.

Let's create a new node module called testling-ci-test-example and let's add cross-browser tests to it. Once the tests run, you get a badge like this that shows in which browsers your code works:

If you click the badge you go to the test status page.

First, let's create index.js that exports a few functions, some of which work in some browsers and fail in others:

// doesn't work in IE<9
//
exports.timesTwo = function (list) {
    return list.map(function (x) { return x*2 });
}

// works everywhere
//
exports.timesThree = function (list) {
    var ret = [];
    for (var i = 0; i < list.length; i++) {
        ret[i] = list[i]*3;
    }
    return ret;
}

Now let's write tests for this module. We'll put tests in tests/test.js:

var test = require('tape');
var exampleLib = require('../index.js');

test('timesTwo test', function (t) {
  t.plan(1);
  t.deepEqual(exampleLib.timesTwo([1,2,3]), [2,4,6]);
});

test('timesThree test', function (t) {
  t.plan(1);
  t.deepEqual(exampleLib.timesThree([1,2,3]), [3,6,9]);
});

The tests use tape module that console.log's TAP formatted output.

Now let's add testling field to package.json:

"testling" : {
    "files" : "test/*.js",
    "browsers" : {
        "ie" : [ 6, 7, 8, 9, 10 ],
        "ff" : [ 3.5, 10, 15.0, 16.0, 17.0 ],
        "chrome" : [ 10, 20, 21, 22, 23 ],
        "safari" : [ 5.1 ],
        "opera" : [ 10, 11, 12 ]
    }
}

Finally, let's setup the github hook and point it to git.testling.com. First go to the settings of the repository,

Next, go to the service hooks,

Then, choose WebHook URLs,

Finally, type git.testling.com in the webhook URL field and press update settings,

And we're done. If we visit http://ci.testling.com/pkrumins/testling-ci-test-example we'll see the badge and which browsers the tests succeeded and in which browsers the code failed:

We can instantly see that the code fails in IE6, IE7, and IE8 because these versions of IE don't have Array.prototype.map, and that the code works everywhere else. So awesome!

For more information about testling-ci see ci.testling.com.

ci.testling.com

About Browserling

Our mission at Browserling Inc is to make the developers' life easier. Follow us on twitter for updates - @browserling, @testling, @substack, @pkrumins.

Announcing Testling-CI

2013-02-27T01:00:47Z

We at Browserling are proud to announce Testling-CI! Testling-CI lets you write continuous integration cross-browser tests that run on every git push!

ci.testling.com

There are a ton of modules on npm and github that aren't just for node.js but for browsers, too. However, figuring out which browsers these modules work with can be tricky. It's often that case that some module used to work in browsers but has accidentally stopped working because the developer hadn't checked that their code still worked recently enough. If you use npm for frontend and backend modules, this can be particularly frustrating.

Testling-CI (http://ci.testling.com) is a service for running your browser tests on every commit. Just add a "testling" field to your package.json like this:

"testling" : {
    "files" : "test/*.js",
    "browsers" : {
        "iexplore" : [ "6.0", "7.0", "8.0", "9.0" ],
        "chrome" : [ "20.0" ],
        "firefox" : [ "10.0", "15.0" ],
        "safari" : [ "5.1" ],
        "opera" : [ "12.0" ]
    }
}

Then add a github webhook for http://git.testling.com and your browser tests will be run on every commit in exactly the browsers you've specified!

Once the tests run you get a badge you can put in your readme with the current browser test status. Here's how it looks:

If you click the badge you go to the test status page.

All your tests need to do is output TAP-formatted text output. Many test harnesses already support this format and the protocol is simple enough that you can just console.log() valid output yourself.

We recommend using tape to output TAP or if you use Mocha, we've a testling-ci example using Mocha. We'll support many more test harnesses later.

Your browser tests can even use node-style require()s bundled with browserify so that you can use exactly the same tests for node and the browser. If your browser tests aren't written the node way you can still run them with the "scripts" field. See testling-ci homepage for full documentation.

This service is free for open source modules published to github. If you want to run this service on your private repos, contact us and we can enroll you in our testling professional edition beta!

For full documentation and more examples visit ci.testling.com!

About Browserling

Our mission at Browserling Inc is to make the developers' life easier. Testling-CI is our third product. Take a look our first two products Browserling and Testling.

Browserling lets you do manual cross-browser testing in all major browsers, such as Internet Explorer 6, 7, 8, 9; Chrome; Safari; Firefox and Opera.

Testling lets you do automated cross-browser JavaScript testing. You write your tests in JavaScript and we run them on the browsers and tell you the results. Testling-CI is built on top of Testling.

We're also soon launching our fourth amazing product. Stay tuned! Follow us on twitter for updates - @browserling, @testling, @substack, @pkrumins.

Important Update:

We've changed goals are we're not launching a fourth product.

We've retired Testling as we built Testling-CI.

All questions about Testling-CI: feedback@browserling.com

The `sudo chroot /chroot su - user -c "cmd args"` trick

2012-12-27T23:08:52Z

Ok, this is going to be an article for beginners but with a spin at the end even for the most advanced users. Let's say you want to run a command inside of chroot as another user all in a single command and be back at your shell. How do you do it? Turns out it can be easily done by combining several commands in a beautiful way:

sudo chroot /chroot su - user -c "cmd args"

It may seem a pretty crazy combination of commands but it's all really easy to undestand. First, let's take a look at man chroot. It says,

sudo chroot /chroot command

Okay, so we can run command inside of chroot as root, but how do we run the command inside of chroot as another user? How about using su for command? Let's try that:

sudo chroot /chroot su - user

This runs su inside of chroot as user but we get an interactive shell. Let's take a closer look at man su. It says that we could try:

sudo su - user -c "cmd args"

This runs the cmd args through the shell. Great! So let's use this together with chroot again:

sudo chroot /chroot su - user -c "cmd args"

Excellent! Almost what we needed. However there is still one small problem. If the user has a custom environment (such as different PATHs to executables or some other customizations), it won't get initialized, so we've to source the initialization file(s) ourselves:

sudo chroot /chroot su - user -c ". ~/.bash_profile; cmd args"

This does what we wanted and runs cmd args inside of chroot located at /new-root-path as user and after running the command it quits and we're back at our shell.

What a beautiful way to combine commands!

On Sourcing ~/.bash_profile

Some people couldn't understand why I was sourcing ~/.bash_profile. The reason is because I was unable to find another way to invoke the login shell and have the environment initialized.

I tried this:

sudo chroot /chroot su -l user -c "cmd args"

And it just wouldn't execute ~/.bash_profile. Su's man page says that -l invokes a login shell, however I don't see that happening. Here's the transcript that demonstrates it:

pkrumins$ pwd
/chroot

pkrumins$ sudo cat ./home/testuser1/.bash_profile 
echo "moo"
PATH="/test/bin"

pkrumins$ sudo chroot /chroot su -l testuser1 -c 'echo $PATH'
/bin:/usr/bin

pkrumins$ sudo chroot /chroot su -l testuser1 
moo

testuser1$ echo $PATH
/test/bin

testuser1$ ^D
logout

pkrumins$ sudo chroot /chroot su -l testuser1 -c '. ~/.bash_profile; echo $PATH'       
moo
/test/bin

pkrumins$ sudo chroot /chroot su -l testuser1
moo

testuser1$ /bin/bash -l -c 'echo $PATH'
moo
/test/bin

This definitely looks like a bug in su to me when -l and -c are used together. So the only way to get the environment loaded is by sourcing the initialization files yourself.

Update

I went through su's source code (can be found in util-linux package) and it turns out it's not a bug in su!

Here are the arguments that su passes to execv when I run su user -c 'echo $PATH' (see run_shell function in su.c):

shell: /bin/bash
args[0]: -bash
args[1]: -c
args[2]: echo $PATH

As you can see, args[0][0] is -, which makes bash a login shell, so bash should be executing the startup files, but it does not!

To figure out what was happening, I built a custom version of bash and added a bunch of debugging statements. I found that there are two different login-shell states! Who'd have thought? One that you get when you've set args[0][0] to -, the other that you get when you've used the -l argument (to bash, not su!).

In case args[0][0] is -, and you're using -c to execute code, bash will not execute startup files because it's in this "non-interactive positive-login-shell" state. However if you use -l and -c, it goes into "non-interactive negative-login-shell" state, and it will execute startup files (see run_startup_files function in bash's shell.c source file).

That was one hell of an adventure debugging this. So if you want bash to execute startup files through su, use my ". ~/.bash_profile" sourcing trick! (Or define NON_INTERACTIVE_LOGIN_SHELLS when building bash).

Bash One-Liners Explained, Part V: Navigating around

2012-11-27T18:24:23Z

This is the fifth part of the Bash One-Liners Explained article series. In this part I'll teach you how to quickly navigate around the command line in bash using emacs-style keyboard shortcuts. That's right, emacs-style keyboard shortcuts. It might surprise you but by default bash uses emacs-style keyboard shortcuts for navigating around the command line. Keep reading to learn more!

See the first part of the series for introduction. After I'm done with the series I'll release an ebook (similar to my ebooks on awk, sed, and perl), and also bash1line.txt (similar to my perl1line.txt).

Also see my other articles about working fast in bash:

Parts of this post are based on my earlier post Working Productively in Bash's Emacs Command Line Editing Mode. Check it out, too!

And grab the emacs keyboard shortcut cheat sheet!

Let's start.

Part V: Navigating around in emacs mode

0. Introduction to input editing modes

Bash uses the readline library for input editing. The readline library supports emacs style key bindings, vi style key bindings as well as custom key bindings. By default readline will use the emacs style key bindings but you can easily switch to vi editing mode or customize them.

You can switch between the emacs and vi editing modes through set -o emacs and set -o vi commands.

The key bindings can be customized through the ~/.inputrc file or the bind command. For example, bind '"\C-f": "ls\n"' binds CTRL+f to execute ls command. You can learn more about readline's key binding syntax by consulting the readline section of the bash man page.

I'll cover the emacs editing mode in this article. In the next two articles I'll cover the vi editing mode and customizing readline.

1. Move to the beginning of the line

Ctrl+a moves the cursor to the beginning of the line. Here's an illustration. Let's say you've typed cd info in the terminal:

Pressing Ctrl+a moves the cursor to the beginning of the line:

2. Move to the end of the line

Ctrl+e moves the cursor to the end of the line. Here's an illustration. Let's say you've typed mkdir foo bar baz in the terminal and your cursor is somewhere in the middle:

Pressing Ctrl+e moves the cursor to the end of the line:

3. Move one word backward

Esc+b or Alt+b moves the cursor one word backward. You'll often see Meta+b but there is no such key on the keyboards anymore. Therefore either Esc+b or Alt+b will work, depending on how your terminal is configured.

Here's an illustration. Let's say you've typed echo 'hello world' in the terminal and your cursor is after hello:

Pressing Esc+b or Alt+b moves the cursor one word backward:

4. Move one word forward

Esc+f or Alt+f moves the cursor one word forward. Here's an illustration. Let's say you've typed echo 'hello world' in the terminal and your cursor is before hello:

Pressing Esc+b or Alt+b moves the cursor one word forward:

5. Delete the last word

Ctrl+w deletes the last word. Deleting a word is also known as killing a word. Each killed word gets stored in the kill ring buffer. If you accidentally killed a word press Ctrl+y to paste it back. Pasting from the kill ring buffer is also known as yanking.

Here's an illustration. Let's say you've typed cd /foo:

Pressing Ctrl+w deletes /foo:

6. Paste the deleted word(s) back

Ctrl+y pastes whatever is in the kill buffer back to the terminal. Here's an illustration. Let's say you had typed cd /foo (see the previous example), and you killed the last word, which is /foo so your command line looks like:

Pressing Ctrl+y brings /foo back:

7. Move one character backward

Ctrl+b moves the cursor one char backward. Here's an illustration. Let's say you had typed echo foo bar baz on the command line:

Pressing Ctrl+b moves the cursor one character backward:

8. Move one character forward

Ctrl+f moves the cursor one char forward. Here's an illustration. Let's say you moved one character backward (as in the previous example):

Pressing Ctrl+f moves the cursor one character forward:

9. Delete the whole line

Ctrl+u kills the whole line and puts it in the kill buffer. Same as with killed words, you can paste the whole line back by pressing Ctrl+y.

Here's an illustration. Let's say you've typed echo moo in your command line:

Pressing Ctrl+u deletes the whole line:

10. Search the history backward

This is probably one of the most used keyboard shortcuts in bash. Pressing Ctrl+r searches the command history backwards. For example, you can press Ctrl+r and then type the first few characters of some command that you executed earlier to quickly find it.

Here's an illustration. Let's say you had executed a complicated command such as this one:

joinlines () { sed ':a; N; s/\n/'"$1"'/; ba'; }

And now you wish to modify it but you don't want to keep going through the history to find it. Press Ctrl+r and type something you remember from the command like joi:

11. Search the history forward

If you press Ctrl+s, most likely your terminal will freeze because by default your terminal interprets Ctrl+s as the stop-flow signal. When I was less experienced this was driving me crazy. I'd accidentally press Ctrl+s and my terminal would freeze. And I had no idea what was happening. Later I learned that I can press CTRL+q to unfreeze the terminal (Ctrl+q starts the flow again.)

The right way to go is to change the terminal behavior for Ctrl+s via the stty command:

$ stty stop 'undef'

This will undefine the key binding for the stop-flow signal and you'll be able to use bash's Ctrl+s.

Ctrl+s comes handy when you've searched too far with Ctrl+r. Then you can just simply reverse the search direction by pressing Ctrl+r.

Here's an illustration. Let's say you typed awk and pressed Ctrl+r a few times and you skipped past the awk command that you wanted to find:

Pressing Ctrl+s reverses the history search direction:

12. Exchange two adjacent characters quickly

Ctrl+t transposes two characters (exchanges them) and moves the cursor one character forward. Here's an illustration. Let's say you've mistyped echo in ehco bar baz and your cursor is at the letter c:

Pressing Ctrl+t exchanges c with h and moves the cursor one char forward:

13. Exchange two adjacent words quickly

Esc+t or Alt+t transposes two words (exchanges them) and moves the cursor one word forward. Here's an illustration. Let's say you've typed foo bar baz and your cursor is at the word bar:

Pressing Esc+t or Alt+t exchanges foo with bar and moves the cursor one word forward:

14. Uppercase the rest of the word

Esc+u or Alt+u uppercases the rest of the word. Here's an illustration. Let's say you've typed foo bar baz and your cursor is at the beginning of bar:

Pressing Esc+u or Alt+u uppercases the whole word and bar becomes BAR:

15. Lowercase the rest of the word

Esc+t or Alt+t uppercases the rest of the word. Here's an illustration. Let's say you've typed foo BAR baz and your cursor is at the beginning of BAR:

Pressing Esc+l or Alt+l lowercases the whole word and BAR becomes bar:

16. Capitalize a word

Esc+c or Alt+c properly capitalizes a word. Here's an illustration. Let's say you've typed foo bar baz and your cursor is at the beginning of bar:

Pressing Esc+c or Alt+c capitalizes the first letter of the word and bar becomes Bar:

17. Insert a raw character (such as TAB or Ctrl+c)

Ctrl+v inserts the next character typed verbatim. For example, Ctrl+v followed by <TAB> would insert a literal tab in the command line, or Ctrl+v followed by Ctrl+m would insert a Windows newline (aka carriage return CR).

Here's an illustration. Let's say you've typed echo foo:

Pressing Ctrl+v followed by Ctrl+m inserts a literal Ctrl+m:

18. Comment the current line (append # at the beginning quickly)

Esc+# or Alt+# quickly comments the line. Here's an illustration. Let's say you typed echo foo bar baz:

Pressing Ctrl+# inserts the comment symbol # at the beginning of the line:

19. Open the current command in a text editor quickly

Pressing CTRL+x followed by CTRL+e opens the current command in your favorite text editor. Exiting the editor will execute the command.

20. Delete a character to the left

Ctrl+h deletes the character to the left of the cursor. Here's an illustration. Let's say you've typed echo qwerty:

Pressing Ctrl+h deletes the character to the left:

21. Delete a character to the right

Ctrl+d deletes the character to the right of the cursor. Here's an illustration. Let's say you've typed echo qwerty:

Pressing Ctrl+d deletes the character to the right:

22. Incremental undo

Pressing Ctrl+x followed by Ctrl+u undoes a change. Here's an illustration. Let's say you typed foo bar baz and then deleted baz and typed moo:

Pressing Ctrl+x, Ctrl+u a few times undoes the last changes and you end up with foo bar baz again:

23. Insert the last argument from the previous command

Esc+. or Alt+. inserts the last argument from the previous command at the current cursor position. Here's an illustration. Let's say you had run ls archive.tgz:

And now you want to extract the archive. So all you've to do is type tar -xzf and press Esc+. or Alt+.:

24. Undo all changes to the line

Esc+r or Alt+r undoes all changes to the line. It's useful when you're going through the command history with Ctrl+r and make changes. If you mess up, you can quickly revert back to the original command by pressing Esc+r or Alt+r.

Here's an illustration. Let's say you searched for grep in history:

And let's say you wanted to make some changes to the command but messed up:

Pressing Esc+r or Alt+r undoes all the changes and you end up with the original grep command that was in the history:

25. Clear the screen

Ctrl+l clears the screen. Alternatively you can type reset.

26. Change input mode to vi

$ set -o vi

This command changes the key bindings to vi's. If vi's your favorite editor, you'll love this. I'll cover the vi mode in more details in the next part of the article.

Cheat Sheet

I made a cheat sheet that lists all the default emacs mode keyboard shortcuts. Download the emacs keyboard shortcut cheat sheet.

Enjoy!

Enjoy the article and let me know in the comments what you think about it!

My favorite regex of all time

2012-11-12T22:36:04Z

I thought I'd share my favorite regex of all time:

[ -~]

Any ideas what this regexp matches?

It matches all ASCII characters from the space to the tilde. What are those characters? They're all printable characters!

Take a look at the ASCII table. The printable characters start at the space and end at the tilde:

[ -~] matches all printable ascii characters (light blue background)

I love this.

Update:

Would you like to wear this regular expression? I just had an idea to make it a t-shirt as this blog post is getting so popular. Check out this t-shirt that I made:

This shirt is available through Teespring. Teespring is like Kickstarter for t-shirts. If 50 people buy the shirt, everyone gets their shirt. Otherwise no one gets charged and no one gets the shirt.

Monthly Catonmat Geek T-Shirt #2: a.pipe(b).pipe(a)

2012-11-01T17:50:45Z

This month I'm presenting the limited-edition node.js duplex stream tees designed by the stream master and node.js wizard SubStack.

This shirt is done in collaboration with Teespring. Teespring is basically Kickstarter for t-shirts. If enough people commit to buying a shirt, everyone gets their shirt. Otherwise nobody gets a shirt (and nobody gets charged).

This tee illustrates the duplex stream pattern a.pipe(b).pipe(a) that sends the messages around like a telephone.

The shirt is available on Hanes tees and is priced at $15.

Get Your Limited Edition Duplex Stream T-Shirt Now!

Enjoy!

Do you love this shirt and want to get more geek shirts in the future? Subscribe to my blog and follow me on twitter for updates!

ASCII bit trick to convert lowercase to uppercase and back

2012-10-22T21:06:56Z

I just remembered that I had forgotten the ASCII trick to convert lowercase to uppercase and back. The trick is super simple, you xor the 6th bit (or 5th depending on how you count bits) and that changes the case!

Check this out:

a = 01100001
A = 01000001

See, just the 6th bit changed.

Why is it this way? Simply because the people who invented ASCII thought it was a great idea. If you look at characters a..z, you'll see that all of them have the 6th bit set to 1. The ASCII inventors though, hey, let's set 6th bit to 0 for upper case letters A..Z, then it will be super easy to change case. So they did.

a = 01100001    A = 01000001 
b = 01100010    B = 01000010 
c = 01100011    C = 01000011 
d = 01100100    D = 01000100 
e = 01100101    E = 01000101 
f = 01100110    F = 01000110 
g = 01100111    G = 01000111 
h = 01101000    H = 01001000 
i = 01101001    I = 01001001 
j = 01101010    J = 01001010 
k = 01101011    K = 01001011 
l = 01101100    L = 01001100 
m = 01101101    M = 01001101 
n = 01101110    N = 01001110 
o = 01101111    O = 01001111 
p = 01110000    P = 01010000 
q = 01110001    Q = 01010001 
r = 01110010    R = 01010010 
s = 01110011    S = 01010011 
t = 01110100    T = 01010100 
u = 01110101    U = 01010101 
v = 01110110    V = 01010110 
w = 01110111    W = 01010111 
x = 01111000    X = 01011000 
y = 01111001    Y = 01011001 
z = 01111010    Z = 01011010

Also check this out, if you xor a character with a space, you invert the case:

$ perl -le 'print "a"^" "'   
A
$ perl -le 'print "A"^" "' 
a

Why is that? Because a space ' ' has the value 32, which is 1<<5, which is the 6th bit, which swaps the case!

Bonus

Here are the commands I used for this blog post:

$ perl -e 'printf "%08b\n", ord("K")'
$ perl -e 'printf "%s = %08b    %s = %08b\n", $_, ord, uc, ord uc for a..z'

Simple articles best articles.

Browserling reaches 200 paying customers!

2012-10-16T23:11:22Z

Amazing news everyone! Browserling just reached 200 paying customers! It took us 2 years to get to 100 customers, and now it took just 5 months to get to 200 customers. We're looking forward to doubling our customer base again soon.

Our new customers include companies such as:

VirtualNerd, Fluent Concepts, BabyList, MapBox, Helicoid, DabApps, Next Social, My Bank Tracker

As well as many individuals.

Thank you all!

Browserling is an interactive cross-browser testing tool. It lets you test your website in older Internet Explorers such as IE 6, IE 7, IE 8 and IE 9, Firefox, Chrome, Opera and Safari!

Try Browserling!

Bash One-Liners Explained, Part IV: Working with history

2012-10-13T16:43:40Z

This is the fourth part of the Bash One-Liners Explained article series. In this part I'll teach you how to work with bash history. I'll use only the best bash practices, various bash idioms and tricks. I want to illustrate how to get various tasks done with just bash built-in commands and bash programming language constructs.

I'll break this part into several sub-parts as it's very tiring to write long articles, and I'd rather publish many short articles and make quick progress.

Also see my other articles about working fast in bash from 2007 and 2008:

Parts of this post are based on my earlier post Definitive Guide to Bash Command Line History. Check it out, too!

Let's start.

Part IV: Working with history

1. Erase all shell history

$ rm ~/.bash_history

Bash keeps the shell history in a hidden file called .bash_history. This file is located in your home directory. To get rid of the history, just delete it.

Note that if you logout after erasing the shell history, this last rm ~/.bash_history command will be logged. If you want to hide that you erased shell history, see the next one-liner.

2. Stop logging history for this session

$ unset HISTFILE

The HISTFILE special bash variable points to the file where the shell history should be saved. If you unset it, bash won't save the history.

Alternatively you can point it to /dev/null,

$ HISTFILE=/dev/null

3. Don't log the current command to history

Just start the command with an extra space:

$  command

If the command starts with an extra space, it's not logged to history.

Note that this only works if the HISTIGNORE variable is properly configured. This variable contains : separated values of command prefixes that shouldn't be logged.

For example to ignore spaces set it to this:

HISTIGNORE="[ ]*"

My HISTIGNORE looks like this:

HISTIGNORE="&:[ ]*"

The ampersand has a special meaning - don't log repeated commands.

4. Change the file where bash logs command history

$ HISTFILE=~/docs/shell_history.txt

Here we simply change the HISTFILE special bash variable and point it to ~/docs/shell_history.txt. From now on bash will save the command history in that file.

5. Add timestamps to history log

$ HISTTIMEFORMAT="%Y-%m-%d %H:%M:%S"

If you set the HISTTIMEFORMAT special bash variable to a valid date format (see man 3 strftime) then bash will log the timestamps to the history log. It will also display them when you call the history command (see the next one-liner).

6. Show the history

$ history

The history command displays the history list with line numbers. If HISTTIMEFORMAT is set, it also displays the timestamps.

7. Show the last 50 commands from the history

$ history 50

If you specify a numeric argument, such as 50, to history, it prints the last 50 commands from the history.

7. Show the top 10 most used commands from the bash history

$ history |
    sed 's/^ \+//;s/  / /' |
    cut -d' ' -f2- |
    awk '{ count[$0]++ } END { for (i in count) print count[i], i }' |
    sort -rn |
    head -10

This one-liner combines bash with sed, cut, awk, sort and head. The perfect combination. Let's walk through this to understand what happens. Let's say the output of history is:

$ history
    1  rm .bash_history 
    2  dmesg
    3  su -
    4  man cryptsetup
    5  dmesg

First we use the sed command to remove the leading spaces and convert the double space after the history command number to a single space:

$ history | sed 's/^ \+//;s/  / /'
1 rm .bash_history 
2 dmesg
3 su -
4 man cryptsetup
5 dmesg

Next we use cut to remove the first column (the history numbers):

$ history |
    sed 's/^ \+//;s/  / /' |
    cut -d' ' -f2-

rm .bash_history 
dmesg
su -
man cryptsetup
dmesg

Next we use awk to record how many times each command has been seen:

$ history |
    sed 's/^ \+//;s/  / /' |
    cut -d' ' -f2- |
    awk '{ count[$0]++ } END { for (i in count) print count[i], i }'

1 rm .bash_history 
2 dmesg
1 su -
1 man cryptsetup

Then we sort the output numerically and reverse it:

$ history |
    sed 's/^ \+//;s/  / /' |
    cut -d' ' -f2- |
    awk '{ count[$0]++ } END { for (i in count) print count[i], i }' |
    sort -rn

2 dmesg
1 rm .bash_history 
1 su -
1 man cryptsetup

Finally we take the first 10 lines that correspond to 10 most frequently used commands:

$ history |
    sed 's/^ \+//;s/  / /' |
    cut -d' ' -f2- |
    awk '{ count[$0]++ } END { for (i in count) print count[i], i }' |
    sort -rn |
    head -10

Here is what my 10 most frequently used commands look like:

2172 ls
1610 gs
252 cd ..
215 gp
213 ls -las
197 cd projects
155 gpu
151 cd
119 gl
119 cd tests/

Here I've gs that's an alias for git status, gp is git push, gpu is git pull and gl is git log.

8. Execute the previous command quickly

$ !!

That's right. Type two bangs. The first bang starts history substitution, and the second one refers to the last command. Here is an example:

$ echo foo
foo
$ !!
foo

Here the echo foo command was repeated.

It's especially useful if you wanted to execute a command through sudo but forgot. Then all you've to do is run:

$ rm /var/log/something
rm: cannot remove `/var/log/something': Permission denied
$
$ sudo !!   # executes `sudo rm /var/log/something`

9. Execute the most recent command starting with the given string

$ !foo

The first bang starts history substitution, and the second one refers to the most recent command starting with foo.

For example,

$ echo foo
foo
$ ls /
/bin /boot /home /dev /proc /root /tmp
$ awk -F: '{print $2}' /etc/passwd
...
$ !ls
/bin /boot /home /dev /proc /root /tmp

Here we executed commands echo, ls, awk, and then used !ls to refer to the ls / command.

10. Open the previous command you executed in a text editor

$ fc

Fc opens the previous command in a text editor. It's useful if you've a longer, more complex command and want to edit it.

For example, let's say you've written a one-liner that has an error, such as:

$ for wav in wav/*; do mp3=$(sed 's/\.wav/\.mp3/' <<< "$wav"); ffmpeg -i "$wav" "$m3p"; done

And you can't see what's going on because you've to scroll around, then you can simply type fc to load it in your text editor, and then quickly find that you mistyped mp3 at the end.

Enjoy!

Enjoy the article and let me know in the comments what you think about it. In the next sub-part I'll continue discussing history keyboard shortcuts, history shell options and more.