good coders code, great reuse

Reflections on Node.js Knockout Competition

2010-08-30T18:23:07Z

So I participated in the 48 hour Node.js Knockout competition together with James Halliday and Joshua Holbrook. Our team was called Dark Knights and we created an online chess application called Node Chess.

We didn't quite manage to completely finish the game and it has several bugs, like the turns don't alternate and the king can be captured, but it's crazy awesome anyway. If both players follow the rules, it all works correctly. Castling works, pawn promotion works, capture en-passant also. Try it and if you find it awesome, please vote! Oh, and it works only Chrome. Ancient-browsers-please-be-gone!

Here is how the game looks,

A chess game between pkrumins and someone. King's Indian Defence.

Joshua did all the awesome vector graphics work. I did the chess engine work, and James used his amazing dnode node.js module to blend client and server code together. James has actually redefined how web development happens. Instead of writing server code and client code, as we are so used to, with his dnode framework it's now possible to use the same code both server and client side! Much less hustle and purely ingenious!

Here is the same game in perspective view, the highlighted squares are the available moves,

The same game in perspective view.

And the moves are animated, too! The pawns shoot the opponent pieces and the queen stabs them. Try it!

Right, so my reflections on the competition.

It was well organized, and we were sent access to a Joyent deployment server and a Heroku server early on together with instructions. It turned out that Heroku's didn't support Socket.IO or websockets. Win for Joyent. Pretty much everyone went with Joyent as far as I know. We had some technical difficulties at the start with deploying our code, but guys at #node.js helped us and we got our app running pretty quickly.

We used 3 Git repositories to push the code to, our own GitHub repositories (pkrumins, substack, jesusabdullah), then the node knockout's private repository for judges, and deployment repository on Joyent. Joyent was configured so that as you push your code to its Git repo, the hooks in it would restart the node.js service and the you'd be instantly running the latest version of your code.

So I'd make changes push to my GitHub repo, James would pull from me. He'd make changes, I'd pull from him, and same for Joshua. It went pretty flawless. We had like 12 merge errors total, but those were all resolved within a minute or two.

Now some numbers. We're actually amazed by our performance. Check out these numbers:

$ git log | grep Author | wc -l
429

429 commits! Can you believe that? 429 commits in 2 days! That's 9 commits per hour on average! That is what I call hacking!

My commits:

$ git log | grep Author | grep Peteris | wc -l
169

I did 3.5 commits per hour on average. And funnily, James and Joshua each had 130 commits:

$ git log | grep Author | grep James | wc -l
130
$ git log | grep Author | grep Joshua | wc -l
130

That's 2.7 commits per hour on average! Amazing! But we also slept between the competition days. On the both days we did about 4 hours of clean sleep, shrinking our competition time to 40 hours. Then our average becomes 10.7 commits per hour! Wowsers!

Here is a graph, made with Raphael.js, that shows our git commit activity by hour, starting from 3am UTC Aug 28 to 3am UTC Aug 30:

Team "Dark Knights" git commit activity by hour.

Our peak commit intensity was at 9pm the last night, when we did 23 commits in one hour. Our team was also widely spread out. I am in Riga, Latvia, James is in Kenai, Alaska, and Joshua is in Fairbanks, Alaska. Yet we managed to keep the same schedule. I'd go to bed at noon (noon for me is 3pm UTC, see the graph above), while James and Joshua at midnight, and we'd wake up several hours later and keep hacking!

Total number of code lines written:

$ wc -l `find . -name '*.js' -o -name '*.html' -o -name '*.css' | \
  egrep -v '(jquery|raphael)'`
3074 total

So we wrote 3074 lines in two days, which according to git break up into added vs. deleted as following:

$ git log --numstat | grep '^[0-9]' | \
  egrep '(\.js|\.css|\.html|jquery|raphael)' | \
  awk '{a+=$1;d+=$2}END{print "Added: " a, "Deleted: " d}'
Added: 5210 Deleted: 2042

Hmm, 5210-2042 doesn't quite add up to 3074 but is close enough. From these 3074 lines of code non-empty were:

$ cat `find . -name '*.js' -o -name '*.html' -o -name '*.css' | \
  egrep -v '(jquery|raphael)'` | perl -nle 'print if /\S/' | \
  wc -l
2659

So 2659 real lines of code in 2 days! Talk about productivity! And that's just code alone. Joshua also did 50 artworks,

$ find . -name '*.svg' -o -name '*.png' | wc -l
50

Total number of file changes:

$ git log --shortstat | grep 'files changed' | \
  awk '{t+=$1}END{print t}'  
724

We communicated in IRC, in our #stackvm startup channel. Here are some statistics on how much stuff went on in our IRC channel:

$ (
  grep -v '^0[012]:' '#stackvm.08-28.log';
  cat '#stackvm.08-29.log';
  grep '^0[012]' '#stackvm.08-30.log'
  ) | wc -l
5069

So 5069 events happened during the challenge. That's 105 events per hour on average. We have a special lulbot in it who tells us when we commit, for example:

05:59 < lulzbot-X> Whoa Nelly! New commits to pkrumins/node-chess (master)!
05:59 < lulzbot-X>     * Peteris Krumins: MoveGenerator stub
05:59 < lulzbot-X>     * Peteris Krumins: abstract pieces
05:59 < lulzbot-X> githubs: http://github.com/pkrumins/node-chess/tree/master

Here lulzbot informed us that I committed MoveGenerator stub and abstracted pieces in node-chess repo.

Out of these 5069 events, we talked this much,

$ (
  grep -v '^0[012]:' '#stackvm.08-28.log';
  cat '#stackvm.08-29.log';
  grep '^0[012]' '#stackvm.08-30.log'
  ) | egrep -i '^< pkrumins|substack|jesus' | wc -l
2682

So we spoke 2682 times or 83.8 times per hour. We also asked quite a lot questions:

$ (
  grep -v '^0[012]:' '#stackvm.08-28.log';
  cat '#stackvm.08-29.log';
  grep '^0[012]' '#stackvm.08-30.log';
  ) | grep '?$' | wc -l
246

246 questions, for example (random selection):

< pkrumins> wait, are we including a version of socket.io.js in dnode?
< jesusabdullah> but: simplified pieces for thumbs--yea or nay?
< pkrumins> is anyone working ont he problem where the opponent cant make moves?
< SubStack> pkrumins did you see how I just dumped the node EventEmitter code into our lib/?
< SubStack> does S create a row?
< jesusabdullah> pkrumins: You fixing the board?
< pkrumins> how does resizing in raphael happen?

My chess code wasn't the easiest to write and to make sure it works correctly, I wrote 52 expresso tests,

$ expresso 

   100% 52 tests

Without tests I would have never got that chess code right.

That's about it. The competition was awesome, A++ would participate again. Hope they organize node.js knockout the next year, too!

I hope you enjoyed my post and don't forget to vote for our project! Your vote is so important to us. Thank you!

StackVM Demo Video #2

2010-08-10T10:20:30Z

Hey everyone. We at StackVM just finished recording the 2nd demo video. The 2nd video shows all the cool new features we have recently built - user login system, chatting and sharing of virtual machines by just dragging and dropping. Also this time James Halliday joins me from Fairbanks, Alaska!

Here is the video #2,

StackVM brings virtual machines to the web. Join #stackvm on FreeNode to discuss!

If you haven't seen the first video, see my StackVM startup announcement post!

Also, if you wish to be the first to try StackVM when we launch, please leave your email below. We'll send you a free demo as soon as we can!

During the past few weeks we have also written two new node.js libraries for use at StackVM:

node-gif - for producing GIF images and animated GIFs (gifcasts).
node-image - unifies node-png, node-gif and node-jpeg.

We did not demo gifcasts in this video but I am going to do a separate video in the next week or two showing just that. They're pretty awesome!

In a few weeks we'll also post the 3rd demo video. In that video we have planned to show virtual network editor that allows to network virtual machines by just dragging and dropping! Be sure to subscribe to catonmat's rss feed and follow me on twitter to know when the video is out!

See you!

Ps. Join #stackvm on FreeNode to discuss StackVM with me and James! We're there 24/7!

Announcement: I am doing a startup - StackVM!

2010-08-10T10:21:12Z

Hey everyone,

I have exceptional news - I am doing a startup together with James Halliday!

I met James on FreeNode several years ago and it turned out that we had very similar ideas about doing great hacking. So we teamed up as equal co-founders to do a startup. We're on different continents but we use IRC, GitHub and WeDoist to get stuff done!

Check out an early demo of our software:

StackVM brings virtual machines to the web. Join #stackvm on FreeNode to discuss!

We can't present a live demo of our startup right now because we don't have that much computing power, however we'll soon be able to send out time-limited demos to several people a day. If you wish to try our software before anyone else does, fill in this form and we'll send you an invite code as soon as we have the hardware:

The startup we're doing is called StackVM. StackVM makes virtual machines much more accessible over the web, makes them easier to use, and makes them embeddable in webpages (like putting a virtual machine in a blog post). This is just the first step that we're starting with. Next we're making networking between them very easy, just drag and drop to create any virtual network topology you wish, with firewalls, switches, etc. (fun for hacking competitions and learning networking). Then we're making what we call "vmcasts" - much like a screencasts, except the computation is recorded, meaning that at any point you can break into the playing vmcast and change the course of computation (and return back to it later).

Our plan is to host the virtual machines and our awesome software for you, so that you don't have to worry about anything. We'll make sure it all works! At the moment we're actually already at our third major iteration of the software.

Here are a few use cases:

Suppose you're selling software and you want your users to try it before they buy it. Perfect use of StackVM - put your software in the virtual machine and embed it on your products page. The potential customers can try your software before they buy it right from your website!
Suppose you're an application developer and have written a program that should work cross-platform. You can easily rent 10 virtual machines with Linux, Windows, MacOS, and other operating systems and test your software. Just drag and drop it into the virtual machines, and you can test your software!
Suppose you want to teach someone how to work in Perl in a series of blog posts (like I do), you can embed the terminal with a vmcast in your blog post, and everyone can follow your tutorial, and also try out the examples interactively, in a real shell!
You can build a virtual honeypot network and have hackers break into it, then analyse how they did breakins. Or, you can build a huge network and learn routing and networking concepts!
Suppose you want to share your work with a group of people. You can easily do it in stackvm! Just send the other people link to your VM and they can connect to it with any web browser. They'll be able to see what you're doing, comment on your work, and if you allow fix your bugs (think pair programming!)

Hosting virtual machines requires a lot of infrastructure, so we plan to start collecting revenue as soon as possible with affordable paid user accounts. We're not going to do the common nonsense among startups of first building up the project for years and only then trying to figure out how to make it profitable. We're going to offer virtual machines together with an awesome interface and features to them as a monthly paid service from the day we launch. We're also going to have an open API to our software, so that you can build on top of it or customize it (for example script virtual machines to do something specific).

We're also testing out the idea of complete openness with this startup. It's going to be 100% open-source and 100% idea-open, meaning that anyone can read what we're up to, and run our code if they want to run stackvm on their own. So far we have shared all the code on github (my stackvm repo, James's stackvm repo) and documented most of the ideas and experiments on stackvm github wiki.

We're applying for YC funding later this year so that we can focus on hacking and not paperwork.

Now a little bit about the tech we use at StackVM. I won't go into much details right now because many of the future posts will detail the tech at StackVM.

At the moment StackVM is in its 3rd major iteration already and is almost entirely built on node.js. Since we're doing everything open-source, we've written a bunch of reusable node.js modules:

dnode - Simple asynchronous remote method invocation for node.js.
node-bufferlist - Abstraction of node.js's buffers and monadic binary parsing.
node-rfb - Implements the client-side of the RFB protocol that VNC uses.
node-png - Produces PNG images from RGB or RGBA values.
node-jpeg - Produces JPEG images from RGB or RGBA values.
node-video - Records Theora/Ogg videos from RGB values.
node-base64 - Encodes binary data to base64 to send across websockets and ajax.
node-jsmin - Minify JavaScript on the server side, before sending it to the browser.
node-bufferdiff - A module to compare two buffers quickly.

I'll keep everyone updated on the progress and technical insight into our development. If you're not yet subscribed to my posts, please do it - catonmat rss feed. Going to make this happen!

And if you wish to talk about our software, come join #stackvm on FreeNode!

Three Years of Blogging

2010-07-08T16:09:03Z

Hey everyone! Another year has passed and it's now 3 years since I've been blogging here on catonmat! In this post I wish to summarize this year's statistics.

See the one year of blogging and two years of blogging for previous year statistics.

First of all traffic statistics,

Traffic statistics for the period 2009-07-01 - 2010-07-01 from Google Analytics.

Alright, so catonmat has received 1.43 million visitors and 2.11 million page views during this year. That's 120k visits and 175k page views per month. Or 4,000 visitors per day and 5,800 page views per day. Good numbers.

Let's look at the whole traffic picture from the day one of blogging,

Traffic statistics for the period 2007-07-01 - 2010-07-01 from Google Analytics.

Looks pretty random but seems to have a hidden linear trend upwards.

Now the FeedBurner subscriber stats,

Feedburner statistics for the period 2009-07-01 - 2010-07-01.

Last year I left off with 7000 subscribers, and now I have around 12,000. That's 5000 new subscribers, or 13 new subscribers per day on average.

And the whole picture of subscriber dynamics since the beginning of blogging,

Feedburner statistics for the period 2007-07-01 - 2010-07-01.

Also a nice positive trend, if it keeps going the same way, I expect to have around 17,000 subscribers the next year.

Now to articles. During this year I have written 43 articles. Here are ten most popular ones:

Using Fibonacci Numbers to Convert from Miles to Kilometers and Vice Versa (194,417 views).
Top Ten One-Liners from CommandLineFu Explained (155,079 views).
A Unix Utility You Should Know About: lsof (59,204 views).
ldd arbitrary code execution (55,833 views).
Summary of all the MIT Introduction to Algorithms lectures (54,954 views).
Must-Have Windows Software (or Windows Programs that I use) (53,825 views).
The Busy Beaver Problem (52,238 views).
Vim Plugins You Should Know About, Part IV: snipmate.vim (35,735 views).
Secret Perl Operators (34,346 views).
A HTTP Proxy Server in 20 Lines of node.js Code (31,822 views).

Here are my personal favorites that didn't make it into top ten:

It's now time for delicious cake:

Let's meet for cake the next year again! See you!

The Four Polymorphisms in C++

2010-06-20T05:20:52Z

When people talk about polymorphism in C++ they usually mean the thing of using a derived class through the base class pointer or reference, which is called subtype polymorphism. But they often forget that there are all kinds of other polymorphisms in C++, such as parametric polymorphism, ad-hoc polymorphism and coercion polymorphism.

These polymorphisms also go by different names in C++,

Subtype polymorphism is also known as runtime polymorphism.
Parametric polymorphism is also known as compile-time polymorphism.
Ad-hoc polymorphism is also known as overloading.
Coercion is also known as (implicit or explicit) casting.

In this article I'll illustrate all the polymorphisms through examples in C++ language and also give insight on why they have various other names.

Subtype Polymorphism (Runtime Polymorphism)

Subtype polymorphism is what everyone understands when they say "polymorphism" in C++. It's the ability to use derived classes through base class pointers and references.

Here is an example. Suppose you have various cats like these felines,

Polymorphic cats on a mat by James Halliday.

Since they are all of Felidae biological family, and they all should be able to meow, they can be represented as classes inheriting from Felid base class and overriding the meow pure virtual function,

// file cats.h

class Felid {
public:
 virtual void meow() = 0;
};

class Cat : public Felid {
public:
 void meow() { std::cout << "Meowing like a regular cat! meow!\n"; }
};

class Tiger : public Felid {
public:
 void meow() { std::cout << "Meowing like a tiger! MREOWWW!\n"; }
};

class Ocelot : public Felid {
public:
 void meow() { std::cout << "Meowing like an ocelot! mews!\n"; }
};

Now the main program can use Cat, Tiger and Ocelot interchangeably through Felid (base class) pointer,

#include <iostream>
#include "cats.h"

void do_meowing(Felid *cat) {
 cat->meow();
}

int main() {
 Cat cat;
 Tiger tiger;
 Ocelot ocelot;

 do_meowing(&cat);
 do_meowing(&tiger);
 do_meowing(&ocelot);
}

Here the main program passes pointers to cat, tiger and ocelot to do_meowing function that expects a pointer to Felid. Since they are all Felids, the program calls the right meow function for each felid and the output is:

Meowing like a regular cat! meow!
Meowing like a tiger! MREOWWW!
Meowing like an ocelot! mews!

Subtype polymorphism is also called runtime polymorphism for a good reason. The resolution of polymorphic function calls happens at runtime through an indirection via the virtual table. Another way of explaining this is that compiler does not locate the address of the function to be called at compile-time, instead when the program is run, the function is called by dereferencing the right pointer in the virtual table.

In type theory it's also known as inclusion polymorphism.

Parametric Polymorphism (Compile-Time Polymorphism)

Parametric polymorphism provides a means to execute the same code for any type. In C++ parametric polymorphism is implemented via templates.

One of the simplest examples is a generic max function that finds maximum of two of its arguments,

#include <iostream>
#include <string>

template <class T>
T max(T a, T b) {
 return a > b ? a : b;
}

int main() {
 std::cout << ::max(9, 5) << std::endl;     // 9

 std::string foo("foo"), bar("bar");
 std::cout << ::max(foo, bar) << std::endl; // "foo"
}

Here the max function is polymorphic on type T. Note, however, that it doesn't work on pointer types because comparing pointers compares the memory locations and not the contents. To get it working for pointers you'd have to specialize the template for pointer types and that would no longer be parametric polymorphism but would be ad-hoc polymorphism.

Since parametric polymorphism happens at compile time, it's also called compile-time polymorphism.

Ad-hoc Polymorphism (Overloading)

Ad-hoc polymorphism allows functions with the same name act differently for each type. For example, given two ints and the + operator, it adds them together. Given two std::strings it concatenates them together. This is called overloading.

Here is a concrete example that implements function add for ints and strings,

#include <iostream>
#include <string>

int add(int a, int b) {
 return a + b;
}

std::string add(const char *a, const char *b) {
 std::string result(a);
 result += b;
 return result;
}

int main() {
 std::cout << add(5, 9) << std::endl;
 std::cout << add("hello ", "world") << std::endl;
}

Ad-hoc polymorphism also appears in C++ if you specialize templates. Returning to the previous example about max function, here is how you'd write a max for two char *,

template <>
const char *max(const char *a, const char *b) {
 return strcmp(a, b) > 0 ? a : b;
}

Now you can call ::max("foo", "bar") to find maximum of strings "foo" and "bar".

Coercion Polymorphism (Casting)

Coercion happens when an object or a primitive is cast into another object type or primitive type. For example,

float b = 6; // int gets promoted (cast) to float implicitly
int a = 9.99 // float gets demoted to int implicitly

Explicit casting happens when you use C's type-casting expressions, such as (unsigned int *) or (int) or C++'s static_cast, const_cast, reinterpret_cast, or dynamic_cast.

Coercion also happens if the constructor of a class isn't explicit, for example,

#include <iostream>

class A {
 int foo;
public:
 A(int ffoo) : foo(ffoo) {}
 void giggidy() { std::cout << foo << std::endl; }
};

void moo(A a) {
 a.giggidy();
}

int main() {
 moo(55);     // prints 55
}

If you made the constructor of A explicit, that would no longer be possible. It's always a good idea to make your constructors explicit to avoid accidental conversions.

Also if a class defines conversion operator for type T, then it can be used anywhere where type T is expected.

For example,

class CrazyInt {
 int v;
public:
 CrazyInt(int i) : v(i) {}
 operator int() const { return v; } // conversion from CrazyInt to int
};

The CrazyInt defines a conversion operator to type int. Now if we had a function, let's say, print_int that took int as an argument, we could also pass it an object of type CrazyInt,

#include <iostream>

void print_int(int a) {
 std::cout << a << std::endl;
}

int main() {
 CrazyInt b = 55;
 print_int(999);    // prints 999
 print_int(b);      // prints 55
}

Subtype polymorphism that I discussed earlier is actually also coercion polymorphism because the derived class gets converted into base class type.

Have Fun!

Have fun with all the new knowledge about polymorphism!

Yet Another Ten One-Liners from CommandLineFu Explained

2010-06-09T20:10:01Z

Hey everyone, this is the fourth article in the series on the most popular commandlinefu one-liners explained.

Here are the first three parts:

And here are today's one-liners:

31. Quickly access ASCII table.

$ man 7 ascii

Ever forgot a keycode for some ASCII character or escape code? Look no further, man ascii contains the 7-bit ASCII table. Take a look at it online.

Linux man pages are full of gems like these. One day I actually went through all the man pages to find the most interesting ones. An article about them is upcoming but before I get it published, here are a few interesting ones:

man 1 intro - a tutorial that gets you started with Linux if you have never used it.
man 2 syscalls - lists all Linux system calls by kernel version.
man 2 select_tut - select() system call tutorial.
man 3 string - lists all <string.h> string manipulation functions.
man 3 stdio - lists and describes all <stdio.h> standard input/output library functions.
man 3 errno - lists and describes all errno error numbers.
man 4 console_codes - Linux console escape and control sequences.
man 4 full - /dev/full - a device that simulates a full device.
man 5 proc - lots of info about the /proc filesystem.
man 5 filesystems - lists various Linux filesystems.

And finally the section 7 man pages that are most packed with wild and cool info,

man 7 bootparam - a tutorial and reference of Linux kernel boot parameters.
man 7 charsets - a Linux programmer's view of character sets and internationalization.
man 7 glob - how pathname globbing works.
man 7 hier - description of the Linux file system hierarchy.
man 7 operator - C language operator precedence and associativity table.
man 7 regex - basic and extended regular expression overview.
man 7 suffixes - lists common Linux file suffixes and the associated file types.
man 7 time - overview of time and timers.
man 7 units - kilo, kibi, mega, mebi, giga, gibi, ... - decimal (SI) and binary system of units.
man 7 utf8 - description of UTF-8 encoding.
man 7 url - description of URIs, URLs and URNs.

There are a lot more interesting man pages but these stood out from the rest.

32. Simple timer.

$ time read

This one-liner can be used a simple timer. For example, if you wish to time something, you can execute it when the event starts and press the return key when the event ends. It will output the time the event took to finish.

Here is how this one-liner works. First the time command times any command that is supplied to it. In this case the command supplied to it is read that reads a line from the standard input. As soon as you press enter, read finishes and time reports how long it took.

If you get annoyed by having to press enter, you can specify that read should return after having read 1 character,

$ time read -N 1

Now you can press any key to stop the timer.

If you wish to run a timer for a specific number of seconds, you can add -t flag for timeout,

$ time read -t 60

This would stop the timer after 60 seconds.

33. Shutdown a Windows machine.

$ net rpc shutdown -I IP_ADDRESS -U username%password

Everyone knows the net command, right? We all used to net use \\ip\ipc$ *, right? :)

Anyway, the net command comes as part of Samba, which allows you, for example, to use Linux in a Windows workgroup.

This particular one-liner executes the shutdown command on a Windows computer located at IP_ADDRESS with as the user username with password password.

To reboot a machine use the -r switch to net rpc:

$ net rpc shutdown -r -I IP_ADDRESS -U username%password

If you're on an unsecured network, don't forget about the good old nmblookup and smbclient tools that come with Samba.

34. Execute a command independently from the current shell.

$ (cd /tmp && ls)

This one-liner illustrates subshells. Here the commands cd /tmp and ls are executed but they do not affect the current shell. If you had done just cd /tmp && ls, your current shell would have changed directory to /tmp but in this one-liner it happens in a subshell and your current shell is not affected.

Surely, this is only a toy example. If you wanted to know what's in /tmp, you'd do just ls /tmp.

Actually, talking about cd, be aware of pushd and popd commands. They allow you to maintain a stack of directories you want to return to later. For example,

/long/path/is/long$ pushd .
/long/path/is/long$ cd /usr
/usr$ popd 
/long/path/is/long$

Or even shorter, passing the directory you're gonna cd to directly to pushd,

/long/path/is/long$ pushd /usr
/usr$ popd 
/long/path/is/long$

Another cool trick is to use cd - to return to the previous directory. Here is an example,

/home/pkrumins$ cd /tmp
/tmp$ cd -
/home/pkrumins$

35. Tunnel your SSH connection via intermediate host.

$ ssh -t reachable_host ssh unreachable_host

This one-liner creates an ssh connection to unreachable_host via reachable_host. It does it by executing the ssh unreachable_host on reachable_host. The -t forces ssh to allocate a pseudo-tty, which is necessary for working interactively in the second ssh to unreachable_host.

This one-liner can be generalized. You can tunnel through arbitrary number of ssh servers:

$ ssh -t host1 ssh -t host2 ssh -t host3 ssh -t host4 ...

Now catch me if you can. ;)

36. Clear the terminal screen.

$ CTRL+l

Pressing CTRL+l (that's small L) clears the screen leaving the current line at the top of the screen.

If you wish to clear just some line, you can use argumented version of CTRL+l - first press ESC, then the line you want to clear, let's say 21 (21st line), and then press the same CTRL+l. That will clear the 21st line on the screen without erasing the whole screen.

$ ESC 21 CTRL+l

This command outputs a special "clear-screen" sequence to the terminal. The same can be achieved by tput command,

$ tput clear

Another way to clear the terminal (usually when the screen gets garbled) is to use the reset command,

$ reset

37. Hear when the machine comes back online.

$ ping -a IP

Ever had a situation when you need to know when the system comes up after a reboot? Up until now you probably launched ping and either followed the timeouts until the system came back, or left it running and occasionally checked its output to see if the host is up. But that is unnecessary, you can make ping -a audible! As soon as the host at IP is back, ping will beep!

38. List 10 most often used commands.

$ history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

The person who wrote it has the Unix mindset right. He's combining several shell commands to get the result he/she wants.

First, history outputs all the commands the person has executed. Next, awk counts how many times the second column $2 appears in the output. Once history has output all the commands and awk has counted them, awk loops over all the commands and outputs the count a[i] separated by space, followed by the command itself. Then sort takes this input and sorts numerically -n and reverses the output -r, so that most frequent commands were on top. Finally head outputs the first 10 most frequent history commands.

If you want to see more than 10 commands (or less), change head to head -20 for 20 commands or head -5 for 5 commands.

39. Check gmail for new mail.

$ curl -u you@gmail.com --silent "https://mail.google.com/mail/feed/atom" |
  perl -ne \
  '
    print "Subject: $1 " if /<title>(.+?)<\/title>/ && $title++;
    print "(from $1)\n" if /<email>(.+?)<\/email>/;
  '

Gmail is cool because they offer an Atom feed for the new mail. This one-liner instructs curl to retrieve the feed and authenticate as you@gmail.com. You'll be prompted a password after you execute the command. Next it feeds the output to perl. Perl extracts the title (subject) of each email and the sender's email. These two items are printed to stdout.

Here is a the output when I run the command,

Subject: i heard you liked windows! (from gates@microsoft.com)
Subject: got root? (from bofh@underground.org)

40. Watch Star-Wars via telnet.

$ telnet towel.blinkenlights.nl

Needs no explaining. Just telnet to the host to watch ASCII Star-Wars.

And here is another one,

$ telnet towel.blinkenlights.nl 666

Connecting on port 666 will spit out BOFH excuses.

That's it for today.

I hope you enjoyed the 4th part of the article. Tune in next time for the 5th part.

Oh, and I'd love if you followed me on Twitter!

On Functors

2010-05-17T14:31:53Z

It's interesting how the term "functor" means completely different things in various programming languages. Take C++ for example. Everyone who has mastered C++ knows that you call a class that implements operator() a functor. Now take Standard ML. In ML functors are mappings from structures to structures. Now Haskell. In Haskell functors are just homomorphisms over containers. And in Prolog functor means the atom at the start of a structure. They all are different. Let's take a closer look at each one.

Functors in C++

Functors in C++ are short for "function objects." Function objects are instances of C++ classes that have the operator() defined. If you define operator() on C++ classes you get objects that act like functions but can also store state. Here is an example,

#include <iostream>
#include <string>

class SimpleFunctor {
    std::string name_;
public:
    SimpleFunctor(const char *name) : name_(name) {}
    void operator()() { std::cout << "Oh, hello, " << name_ << endl; }
};

int main() {
    SimpleFunctor sf("catonmat");
    sf();  // prints "Oh, hello, catonmat"
}

Notice how I was able to call sf() in the main function, even though sf was an object? That's because I defined operator() in SimpleFunctor's class.

Most often functors in C++ are used as predicates, fake closures or comparison functions in STL algorithms. Here is another example. Suppose you have a list of integers and you wish to find the sum of all even ones, and the sum of all odd ones. Perfect job for a functor and for_each algorithm.

#include <algorithm>
#include <iostream>
#include <list>

class EvenOddFunctor {
    int even_;
    int odd_;
public:
    EvenOddFunctor() : even_(0), odd_(0) {}
    void operator()(int x) {
        if (x%2 == 0) even_ += x;
        else odd_ += x;
    }
    int even_sum() const { return even_; }
    int odd_sum() const { return odd_; }
};

int main() {
    EvenOddFunctor evenodd;
    
    int my_list[] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
    evenodd = std::for_each(my_list,
                  my_list+sizeof(my_list)/sizeof(my_list[0]),
                  evenodd);

    std::cout << "Sum of evens: " << evenodd.even_sum() << "\n";
    std::cout << "Sum of odds: " << evenodd.odd_sum() << std::endl;

    // output:
    // Sum of evens: 30
    // Sum of odds: 25
}

Here an instance of an EvenOddFunctor gets passed to for_each algorithm. The for_each algorithm iterates over each element in my_list and calls the functor. After it's done, it returns a copy of evenodd functor that contains the sum of evens and odds.

Functors in Standard ML

Vaguely talking in object-oriented terms, functors in ML are generic implementations of interfaces. In ML terms, functors are part of ML module system and they allow to compose structures.

Here is an example, suppose you want to write a plugin system and you wish all the plugins to implement the required interface, which, for simplicity, includes only the perform function. In ML you can first define a signature for plugins,

signature Plugin =
sig
    val perform : unit -> unit
end;

Now that we have defined an interface (signature) for plugins, we can implement two Plugins, let's say LoudPlugin and SilentPlugin. The implementation is done via structures,

structure LoudPlugin :> Plugin =
struct
    fun perform () = print "DOING JOB LOUDLY!\n"
end;

And the SilentPlugin,

structure SilentPlugin :> Plugin =
struct
    fun perform () = print "doing job silently\n"
end;

Now we get to functors. Remember functors in ML take structures as their arguments, so we can write one that takes Plugin as its argument,

functor Performer(P : Plugin) =
struct
    fun job () = P.perform ()
end;

This functor takes Plugin as P argument, and uses it in the job function, that calls P plugin's perform function.

Now let's use the Performer functor. Remember that a functor returns a structure,

structure LoudPerformer = Performer(LoudPlugin);
structure SilentPerformer = Performer(SilentPlugin);

LoudPerformer.job ();
SilentPerformer.job ();

This outputs,

DOING JOB LOUDLY!
doing job silently

This is probably the simplest possible example of Standard ML functors.

Functors in Haskell

Functors in Haskell is what real functors are supposed to be. Haskell functors resemble mathematical functors from category theory. In category theory a functor F is a mapping between two categories such that the structure of the category being mapped over is preserved, in other words, it's a homomorphism between two categories.

In Haskell this definition is implemented as a simple type class,

class Functor f where
  fmap :: (a -> b) -> f a -> f b

Looking back at ML example, a type class in Haskell is like a signature, except it's defined on types. It defines what operations a type has to implement to be an instance of this class. In this case, however, the Functor is not defined over types but over type constructors f. It says, a Functor is something that implements the fmap function that takes a function from type a to type b, and a value of type f a (a type constructed from type constructor f applied to type a) and returns a value of type f b.

To understand what it does, think of fmap as a function that applies an operation to each element in some kind of a container.

The simplest example of functors is regular lists and the map function that maps a function to each element in the list.

Prelude> map (+1) [1,2,3,4,5]
[2,3,4,5,6]

In this simple example, the Functor's fmap function is just map and type constructor f is [] - the list type constructor. Therefore the Functor instance for lists is defined as

instance Functor [] where
  fmap = map

Let's see if it really is true by using fmap instead of map in the previous example,

Prelude> fmap (+1) [1,2,3,4,5]
[2,3,4,5,6]

But notice that Functor definition does not say anything about preserving the structure! Therefore any sensible functor must satisfy the functor laws, which are part of the definition of the mathematical functor, implicitly. There are two rules on fmap:

fmap id = id
fmap (g . h) = fmap g . fmap h

The first rule says that mapping the identity function over every element in a container has no effect. The second rule says that a composition of two functions over every item in a container is the same as first mapping one function, and then mapping the other.

Another example of Functors that illustrate them the most vividly is operations over trees. Think of a tree as a container, then fmap maps a function over tree values, while preserving the tree's structure.

Let's define a Tree first,

data Tree a = Node (Tree a) (Tree a)
            | Leaf a
              deriving Show

This definition says that a Tree of type a is either a Node of two Trees (left and right branches) or a Leaf. The deriving Show expression allows us to inspect the Tree via show function.

Now we can define a Functor over Trees,

instance Functor Tree where
    fmap g (Leaf v) = Leaf (g v)
    fmap g (Node l r) = Node (fmap g l) (fmap g r)

This definition says, that fmap of function g over a Leaf with value v is just a Leaf of g applied to v. And fmap of g over a Node with left l and right r branches is just a Node of fmap applied to the values of left and right branches.

Now let's illustrate how fmap works on trees. Let's construct a tree with String leaves and map the length function over them to find out the length of each leaf.

Prelude> let tree = (Node (Node (Leaf "hello") (Leaf "foo")) (Leaf "baar"))
Prelude> fmap length tree
Node (Node (Leaf 5) (Leaf 3)) (Leaf 4)

Here I constructed the following tree,

           *
          / \
         /   \
        *  "baar"
       / \
      /   \
     /     \
    /       \
 "hello"  "foo"

And mapped length function over it, producing,

           *
          / \
         /   \
        *     4
       / \     
      /   \
     /     \
    /       \
   5         3

Another way of saying what fmap does is that is lifts a function from the "normal world" into the "f world."

In fact Functor is the most fundamental type class in Haskell because Monads, Applicatives and Arrows are all Functors. As I like to say it, Haskell starts where the functors start.

If you wish to learn more about Haskell type classes, start with the excellent article Typeclassopedia (starts at page 17).

Functors in Prolog

Finally, functors in Prolog. Functors in Prolog are the simplest of all. They refer to two things. The first is the atom at the start of the structure. Here is an example, given an expression,

?- likes(mary, pizza)

the functor is the first atom - likes.

The second is built-in predicate called functor. It returns the arity and the functor of a structure. For example,

?- functor(likes(mary, pizza), Functor, Arity).
Functor = likes
Arity = 2

That's it for functors in Prolog.

Conclusion

This article demonstrated how a simple term like "functor" can refer to completely different things in various programming languages. Therefore when you hear a the term "functor", it's important to know the context it's being mentioned in.

Turn any Linux computer into SOCKS5 proxy in one command

2010-05-07T12:18:07Z

I thought I'd do a shorter article on catonmat this time. It goes hand in hand with my upcoming article series on "100% technical guide to anonymity" and it's much easier to write larger articles in smaller pieces. Then I can edit them together and produce the final article.

This article will be interesting for those who didn't know it already -- you can turn any Linux computer into a SOCKS5 (and SOCKS4) proxy in just one command:

ssh -N -D 0.0.0.0:1080 localhost

And it doesn't require root privileges. The ssh command starts up dynamic -D port forwarding on port 1080 and talks to the clients via SOCSK5 or SOCKS4 protocols, just like a regular SOCKS5 proxy would! The -N option makes sure ssh stays idle and doesn't execute any commands on localhost.

If you also wish the command to go into background as a daemon, then add -f option:

ssh -f -N -D 0.0.0.0:1080 localhost

To use it, just make your software use SOCKS5 proxy on your Linux computer's IP, port 1080, and you're done, all your requests now get proxied.

Access control can be implemented via iptables. For example, to allow only people from the ip 1.2.3.4 to use the SOCKS5 proxy, add the following iptables rules:

iptables -A INPUT --src 1.2.3.4 -p tcp --dport 1080 -j ACCEPT
iptables -A INPUT -p tcp --dport 1080 -j REJECT

The first rule says, allow anyone from 1.2.3.4 to connect to port 1080, and the other rule says, deny everyone else from connecting to port 1080.

Surely, executing iptables requires root privileges. If you don't have root privileges, and you don't want to leave your proxy open (and you really don't want to do that), you'll have to use some kind of a simple TCP proxy wrapper to do access control.

Here, I wrote one in Perl. It's called tcp-proxy.pl and it uses IO::Socket::INET to abstract sockets, and IO::Select to do connection multiplexing.

#!/usr/bin/perl
#

use warnings;
use strict;

use IO::Socket::INET;
use IO::Select;

my @allowed_ips = ('1.2.3.4', '5.6.7.8', '127.0.0.1', '192.168.1.2');
my $ioset = IO::Select->new;
my %socket_map;

my $debug = 1;

sub new_conn {
    my ($host, $port) = @_;
    return IO::Socket::INET->new(
        PeerAddr => $host,
        PeerPort => $port
    ) || die "Unable to connect to $host:$port: $!";
}

sub new_server {
    my ($host, $port) = @_;
    my $server = IO::Socket::INET->new(
        LocalAddr => $host,
        LocalPort => $port,
        ReuseAddr => 1,
        Listen    => 100
    ) || die "Unable to listen on $host:$port: $!";
}

sub new_connection {
    my $server = shift;
    my $client = $server->accept;
    my $client_ip = client_ip($client);

    unless (client_allowed($client)) {
        print "Connection from $client_ip denied.\n" if $debug;
        $client->close;
        return;
    }
    print "Connection from $client_ip accepted.\n" if $debug;

    my $remote = new_conn('localhost', 55555);
    $ioset->add($client);
    $ioset->add($remote);

    $socket_map{$client} = $remote;
    $socket_map{$remote} = $client;
}

sub close_connection {
    my $client = shift;
    my $client_ip = client_ip($client);
    my $remote = $socket_map{$client};
    
    $ioset->remove($client);
    $ioset->remove($remote);

    delete $socket_map{$client};
    delete $socket_map{$remote};

    $client->close;
    $remote->close;

    print "Connection from $client_ip closed.\n" if $debug;
}

sub client_ip {
    my $client = shift;
    return inet_ntoa($client->sockaddr);
}

sub client_allowed {
    my $client = shift;
    my $client_ip = client_ip($client);
    return grep { $_ eq $client_ip } @allowed_ips;
}

print "Starting a server on 0.0.0.0:1080\n";
my $server = new_server('0.0.0.0', 1080);
$ioset->add($server);

while (1) {
    for my $socket ($ioset->can_read) {
        if ($socket == $server) {
            new_connection($server);
        }
        else {
            next unless exists $socket_map{$socket};
            my $remote = $socket_map{$socket};
            my $buffer;
            my $read = $socket->sysread($buffer, 4096);
            if ($read) {
                $remote->syswrite($buffer);
            }
            else {
                close_connection($socket);
            }
        }
    }
}

To use it, you'll have to make a change to the previous configuration. Instead of running ssh SOCKS5 proxy on 0.0.0.0:1080, you'll need to run it on localhost:55555,

ssh -f -N -D 55555 localhost

After that, run the tcp-proxy.pl,

perl tcp-proxy.pl &

The TCP proxy will start listening on 0.0.0.0:1080 and will redirect only the allowed IPs in @allowed_ips list to localhost:55555.

Another possibility is to use another computer instead of your own as exit node. What I mean is you can do the following:

ssh -f -N -D 1080 other_computer.com

This will set up a SOCKS5 proxy on localhost:1080 but when you use it, ssh will automatically tunnel your requests (encrypted) via other_computer.com. This way you can hide what you're doing on the Internet from anyone who might be sniffing your link. They will see that you're doing something but the traffic will be encrypted so they won't be able to tell what you're doing.

That's it. You're now the proxy king!

Thanks To Vitalik

Special thanks to Vitalik from "Itransition company" for helpful tips!

Download tcp-proxy.pl

Download link: tcp proxy (tcp-proxy.pl)
Download URL: http://www.catonmat.net/download/tcp-proxy.pl
Downloaded: 594 times

I also pushed the tcp-proxy.pl to GitHub: tcp-proxy.pl on GitHub. This project is also pretty nifty to generalize and make a program that redirects between any number of hosts:ports, not just two.

PS. I will probably also write "A definitive guide to ssh port forwarding" some time in the future because it's an interesting but little understood topic.

A HTTP Proxy Server in 20 Lines of node.js Code

2010-06-27T16:55:31Z

var http = require('http');

http.createServer(function(request, response) {
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

This is just amazing. In 20 lines of node.js code and 10 minutes of time I was able to write a HTTP proxy. And it scales well, too. It's not a blocking HTTP proxy, it's event driven and asynchronous, meaning hundreds of people can use simultaneously and it will work well.

To get the proxy running all you have to do is download node.js, compile it, and run the proxy program via the node program:

$ ./configure --prefix=/home/pkrumins/installs/nodejs-0.1.92
$ make
$ make install

$ PATH=$PATH:/home/pkrumins/installs/nodejs-0.1.92/bin

$ node proxy.js

And from here you can take this proxy wherever your imagination takes. For example, you can start by adding logging:

var http = require('http');
var sys  = require('sys');

http.createServer(function(request, response) {
  sys.log(request.connection.remoteAddress + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function (proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

Next, you can add a regex-based host blacklist in 15 additional lines:

var http = require('http');
var sys  = require('sys');
var fs   = require('fs');

var blacklist = [];

fs.watchFile('./blacklist', function(c,p) { update_blacklist(); });

function update_blacklist() {
  sys.log("Updating blacklist.");
  blacklist = fs.readFileSync('./blacklist').split('\n')
              .filter(function(rx) { return rx.length })
              .map(function(rx) { return RegExp(rx) });
}

http.createServer(function(request, response) {
  for (i in blacklist) {
    if (blacklist[i].test(request.url)) {
      sys.log("Denied: " + request.method + " " + request.url);
      response.end();
      return;
    }
  }

  sys.log(request.connection.remoteAddress + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function(proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary);
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

update_blacklist();

Now to block proxy users from using Facebook, just echo facebook.com to blacklist file:

$ echo 'facebook.com' >> blacklist

The proxy server will automatically notice the changes to the file and update the blacklist.

Surely, a proxy server without IP control is no proxy server, so let's add that as well:

var http = require('http');
var sys  = require('sys');
var fs   = require('fs');

var blacklist = [];
var iplist    = [];

fs.watchFile('./blacklist', function(c,p) { update_blacklist(); });
fs.watchFile('./iplist', function(c,p) { update_iplist(); });

function update_blacklist() {
  sys.log("Updating blacklist.");
  blacklist = fs.readFileSync('./blacklist').split('\n')
              .filter(function(rx) { return rx.length })
              .map(function(rx) { return RegExp(rx) });
}

function update_iplist() {
  sys.log("Updating iplist.");
  iplist = fs.readFileSync('./iplist').split('\n')
           .filter(function(ip) { return ip.length });
}

http.createServer(function(request, response) {
  var allowed_ip = false;
  for (i in iplist) {
    if (iplist[i] == request.connection.remoteAddress) {
      allowed_ip = true;
      break;
    }
  }

  if (!allowed_ip) {
    sys.log("IP " + request.connection.remoteAddress + " is not allowed");
    response.end();
    return;
  }

  for (i in blacklist) {
    if (blacklist[i].test(request.url)) {
      sys.log("Denied: " + request.method + " " + request.url);
      response.end();
      return;
    }
  }

  sys.log(request.connection.remoteAddress + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function(proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary');
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

update_blacklist();
update_iplist();

By default the proxy server will not allow any connections, so add all the IPs you want the proxy to be accessible from to iplist file:

$ echo '1.2.3.4' >> iplist

Finally, let's refactor the code a little:

var http = require('http');
var sys  = require('sys');
var fs   = require('fs');

var blacklist = [];
var iplist    = [];

fs.watchFile('./blacklist', function(c,p) { update_blacklist(); });
fs.watchFile('./iplist', function(c,p) { update_iplist(); });

function update_blacklist() {
  sys.log("Updating blacklist.");
  blacklist = fs.readFileSync('./blacklist').split('\n')
              .filter(function(rx) { return rx.length })
              .map(function(rx) { return RegExp(rx) });
}

function update_iplist() {
  sys.log("Updating iplist.");
  iplist = fs.readFileSync('./iplist').split('\n')
           .filter(function(rx) { return rx.length });
}

function ip_allowed(ip) {
  for (i in iplist) {
    if (iplist[i] == ip) {
      return true;
    }
  }
  return false;
}

function host_allowed(host) {
  for (i in blacklist) {
    if (blacklist[i].test(host)) {
      return false;
    }
  }
  return true;
}

function deny(response, msg) {
  response.writeHead(401);
  response.write(msg);
  response.end();
}

http.createServer(function(request, response) {
  var ip = request.connection.remoteAddress;
  if (!ip_allowed(ip)) {
    msg = "IP " + ip + " is not allowed to use this proxy";
    deny(response, msg);
    sys.log(msg);
    return;
  }

  if (!host_allowed(request.url)) {
    msg = "Host " + request.url + " has been denied by proxy configuration";
    deny(response, msg);
    sys.log(msg);
    return;
  }

  sys.log(ip + ": " + request.method + " " + request.url);
  var proxy = http.createClient(80, request.headers['host'])
  var proxy_request = proxy.request(request.method, request.url, request.headers);
  proxy_request.addListener('response', function(proxy_response) {
    proxy_response.addListener('data', function(chunk) {
      response.write(chunk, 'binary');
    });
    proxy_response.addListener('end', function() {
      response.end();
    });
    response.writeHead(proxy_response.statusCode, proxy_response.headers);
  });
  request.addListener('data', function(chunk) {
    proxy_request.write(chunk, 'binary);
  });
  request.addListener('end', function() {
    proxy_request.end();
  });
}).listen(8080);

update_blacklist();
update_iplist();

Again, it's amazing how fast you can write server software in node.js and JavaScript. It would have probably taken me a day to write the same in C. I love how fast you can prototype the software nowadays.

Download proxy.js

Download link: proxy server written in node.js
Download URL: http://www.catonmat.net/download/proxy.js
Downloaded: 1018 times

I am gonna build this proxy up, so I also put it on GitHub: proxy.js on GitHub

Happy proxying!

Another Ten One-Liners from CommandLineFu Explained

2010-04-30T11:54:34Z

Another week and another top ten one-liners from commandlinefu explained.

This is the third post in the series already, covering one-liners 21-30. See the previous two posts for the introduction of the series and one-liners 1-20:

Update: Russian translation now available.

#21. Display currently mounted file systems nicely

$ mount | column -t

The file systems are not that important here. The column -t command is what is important. It takes the input and formats it into multiple columns so that all columns were aligned vertically.

Here is how the mounted filesystem list looks without column -t command:

$ mount

/dev/root on / type ext3 (rw)
/proc on /proc type proc (rw)
/dev/mapper/lvmraid-home on /home type ext3 (rw,noatime)

And now with column -t command:

$ mount | column -t

/dev/root                 on  /      type  ext3   (rw)
/proc                     on  /proc  type  proc   (rw)
/dev/mapper/lvmraid-home  on  /home  type  ext3   (rw,noatime)

You can improve this one-liner now by also adding column titles:

$ (echo "DEVICE - PATH - TYPE FLAGS" && mount) | column -t

DEVICE                    -   PATH   -     TYPE   FLAGS
/dev/root                 on  /      type  ext3   (rw)
/proc                     on  /proc  type  proc   (rw)
/dev/mapper/lvmraid-home  on  /home  type  ext3   (rw,noatime)

Columns 2 and 4 are not really necessary. We can use awk text processing utility to get rid of them:

$ (echo "DEVICE PATH TYPE FLAGS" && mount | awk '$2=$4="";1') | column -t

DEVICE                    PATH   TYPE   FLAGS
/dev/root                 /      ext3   (rw)
/proc                     /proc  proc   (rw)
/dev/mapper/lvmraid-home  /home  ext3   (rw,noatime)

Finally, we can make it an alias so that we always enjoyed the nice output from mount. Let's call this alias nicemount:

$ nicemount() { (echo "DEVICE PATH TYPE FLAGS" && mount | awk '$2=$4="";1') | column -t; }

Let's try it out:

$ nicemount

DEVICE                    PATH   TYPE   FLAGS
/dev/root                 /      ext3   (rw)
/proc                     /proc  proc   (rw)
/dev/mapper/lvmraid-home  /home  ext3   (rw,noatime)

It works!

#22. Run the previous shell command but replace every "foo" with "bar"

$ !!:gs/foo/bar

I explained this type of one-liners in one-liner #5 already. Please take a look for a longer discussion.

To summarize, what happens here is that the !! recalls the previous executed shell command and :gs/foo/bar substitutes (the :s flag) all (the g flag) occurrences of foo with bar. The !! construct is called an event designator.

#23. Top for files

$ watch -d -n 1 'df; ls -FlAt /path'

This one-liner watches for file changes in directory /path. It uses the watch command that executes the given command periodically. The -d flag tells watch to display differences between the command calls (so you saw what files get added or removed in /path). The -n 1 flag tells it to execute the command every second.

The command to execute is df; ls -FlAt /path that is actually two commands, executed one after other. First, df outputs the filesystem disk space usage, and then ls -FlAt lists the files in /path. The -F argument to ls tells it to classify files, appending */=>@| to the filenames to indicate whether they are executables *, directories /, sockets =, doors >, symlinks @, or named pipes |. The -l argument lists all files, -A hides . and .., and -t sorts the files by time.

Special note about doors - they are Solaris thing that act like pipes, except they launch the program that is supposed to be the receiving party. A plain pipe would block until the other party opens it, but a door launches the other party itself.

Actually the output is nicer if you specify -h argument to df so it was human readable. You can also join the arguments to watch together, making them -dn1. Here is the final version:

$ watch -dn1 'df -h; ls -FlAt /path'

Another note - -d in BSD is --differences

#24. Mount a remote folder through SSH

$ sshfs name@server:/path/to/folder /path/to/mount/point

That's right, you can mount a remote directory locally via SSH! You'll first need to install two programs however:

FUSE that allows to implement filesystems in userspace programs, and
sshfs client that uses FUSE and sftp (secure ftp - comes with OpenSSH, and is on your system already) to access the remote host.

And that's it, now you can use sshfs to mount remote directories via SSH.

To unmount, use fusermount:

fusermount -u /path/to/mount/point

#25. Read Wikipedia via DNS

$ dig +short txt <keyword>.wp.dg.cx

This is probably the most interesting one-liner today. David Leadbeater created a DNS server, which when queried the TXT record type, returns a short plain-text version of a Wikipedia article. Here is his presentation on he did it.

Here is an example, let's find out what "hacker" means:

$ dig +short txt hacker.wp.dg.cx

"Hacker may refer to: Hacker (computer security), someone involved
in computer security/insecurity, Hacker (programmer subculture), a
programmer subculture originating in the US academia in the 1960s,
which is nowadays mainly notable for the free software/" "open
source movement, Hacker (hobbyist), an enthusiastic home computer
hobbyist http://a.vu/w:Hacker"

The one-liner uses dig, the standard sysadmin's utility for DNS troubleshooting to do the DNS query. The +short option makes it output only the returned text response, and txt makes it query the TXT record type.

This one-liner is actually alias worthy, so let's make an alias:

wiki() { dig +short txt $1.wp.dg.cx; }

Try it out:

$ wiki hacker

"Hacker may refer to: Hacker (computer security), ..."

It works!

If you don't have dig, you may also use host that also performs DNS lookups:

host -t txt hacker.wp.dg.cx

#26. Download a website recursively with wget

$ wget --random-wait -r -p -e robots=off -U Mozilla www.example.com

This one-liner does what it says. Here is the explanation of the arguments:

--random-wait - wait between 0.5 to 1.5 seconds between requests.
-r - turn on recursive retrieving.
-e robots=off - ignore robots.txt.
-U Mozilla - set the "User-Agent" header to "Mozilla". Though a better choice is a real User-Agent like "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729)".

Some other useful options are:

--limit-rate=20k - limits download speed to 20kbps.
-o logfile.txt - log the downloads.
-l 0 - remove recursion depth (which is 5 by default).
--wait=1h - be sneaky, download one file every hour.

#27. Copy the arguments of the most recent command

ALT + . (or ESC + .)

This keyboard shortcut works in shell's emacs editing mode only, it copies the last argument form the last command to the current command. Here is an example:

$ echo a b c
a b c

$ echo <Press ALT + .>
$ echo c

If you repeat the command, it copies the last argument from the command before the last, then if you repeat again, it copies the last argument from command before the command before the last, etc.

Here is an example:

$ echo 1 2 3
1 2 3
$ echo a b c
a b c

$ echo <Press ALT + .>
$ echo c

$ echo <Press ALT + .> again
$ echo 3

However, if you wish to get 1st or 2nd or n-th argument, use the digit-argument command ALT + 1 (or ESC + 1) or ALT + 2 (or ESC +2), etc. Here is an example:

$ echo a b c
a b c

$ echo <Press ALT + 1> <Press ALT + .>
$ echo a
a

$ echo <Press ALT + 2> <Press ALT + .>
$ echo b
b

See my article on Emacs Editing Mode Keyboard Shortcuts for a tutorial and a cheat sheet of all the shortcuts.

#28. Execute a command without saving it in the history

$ <space>command

This one-liner works at least on bash, I haven't tested other shells.

If you start your command by a space, it won't be saved to bash history (~/.bash_history file). This behavior is controlled by $HISTIGNORE shell variable. Mine is set to HISTIGNORE="&:[ ]*", which means don't save repeated commands to history, and don't save commands that start with a space to history. The values in $HISTIGNORE are colon-separated.

If you're interested, see my article "The Definitive Guide to Bash Command Line History" for a short tutorial on how to work with shell history and a summary cheat sheet.

#29. Show the size of all sub folders in the current directory

$ du -h --max-depth=1

The --max-depth=1 causes du to summarize disk usage statistics for directories that are depth 1 from the current directory, that is, all directories in the current directory. The -h argument makes the summary human-readable, that is, displays 5MB instead of 5242880 (bytes).

If you are interested in both sub folder size and file size in the current directory, you can use the shorter:

$ du -sh *

#30. Display the top ten running processes sorted by memory usage

$ ps aux | sort -nk +4 | tail

This is certainly not the best way to display the top ten processes that consume the most memory, but, hey, it works.

It takes the output of ps aux, sorts it by 4th column numerically and then uses tail to output the last then lines which happen to be the processes with the biggest memory consumption.

If I was to find out who consumes the most memory, I'd simply use htop or top and not ps.

Bonus one-liner: Start an SMTP server

python -m smtpd -n -c DebuggingServer localhost:1025

This one-liner starts an SMTP server on port 1025. It uses Python's standard library smtpd (specified by -m smtpd) and passes it three arguments - -n, -c DebuggingServer and localhost:1025.

The -n argument tells Python not to setuid (change user) to "nobody" - it makes the code run under your user.

The -c DebuggingServer argument tells Python to use DebuggingServer class as the SMTP implementation that prints each message it receives to stdout.

The localhost:1025 argument tells Python to start the SMTP server on locahost, port 1025.

However, if you wish to start it on the standard port 25, you'll have to use sudo command, because only root is allowed to start services on ports 1-1024. These are also known as privileged ports.

sudo python -m smtpd -n -c DebuggingServer localhost:25

This one-liner was coined by Evan Culver. Thanks to him!

That's it for today,

but be sure to come back the next time for "Yet Another Ten One-Liners from CommandLineFu Explained!"

A Real Turing Machine Running The Busy Beaver

2010-03-29T09:45:37Z

Remember my article on The Busy Beaver Problem? Well, someone built a real Turing Machine and decided to run the busy beaver with 4 states on it. Here is the video.

Video URL: http://www.youtube.com/watch?v=2PjU6DJyBpw

The Turing Machine in this video runs for 107 steps and halts with the total of 13 ones, as expected.

In my article on The Busy Beaver Problem, I also wrote a program that visualizes the tape changes. If you follow the video closely, you'll see that they match the visualization (black square stands for 1, white for 0).

Tape changes for 4 state busy beaver.

See A Turing Machine website for more videos and information about how this machine was actually built. Also see my article on Busy Beaver for a Turing Machine implementation in Python and C++.

The Next Ten One-Liners from CommandLineFu Explained

2010-04-30T11:53:56Z

Here are the next ten top one-liners from the commandlinefu website. The first post about the topic became massively popular and received over 100,000 views in the first two days.

Before I dive into the next ten one-liners, I want to take the chance and promote the other three article series on one-liners that I have written:

Update: Russian translation now available.

Alright, so here are today's one-liners:

#11. Edit the command you typed in your favorite editor

$ command <CTRL-x CTRL-e>

This one-liner opens the so-far typed command in your favorite text editor for further editing. This is handy if you are typing a lengthier shell command. After you have done editing the command, quit from your editor successfully to execute it. To cancel execution, just erase it. If you quit unsuccessfully, the command you had typed before diving into the editor will be executed.

Actually, I have to educate you, it's not a feature of the shell per se but a feature of the readline library that most shells use for command line processing. This particular binding CTRL-x CTRL-e only works in readline emacs editing mode. The other mode is readline vi editing mode, in which the same can be accomplished by pressing ESC and then v.

The emacs editing mode is the default in all the shells that use the readline library. The usual command to change between the modes is set -o vi to change to vi editing mode and set -o emacs to change back to emacs editing mode.

To change the editor, export the $EDITOR shell variable to your preference. For example, to set the default editor to pico, type export EDITOR=pico.

Another way to edit commands in a text editor is to use fc shell builtin (at least bash has this builtin). The fc command opens the previous edited command in your favorite text editor. It's easy to remember the fc command because it stands for "fix command."

Remember the ^foo^bar^ command from the first top ten one-liners? You can emulate this behavior by typing fc -s foo=bar. It will replace foo with bar in the previous command and execute it.

#12. Empty a file or create a new file

$ > file.txt

This one-liner either wipes the file called file.txt empty or creates a new file called file.txt.

The shell first checks if the file file.txt exists. If it does, the shell opens it and wipes it clean. If it doesn't exist, the shell creates the file and opens it. Next the shell proceeds to redirecting standard output to the opened file descriptor. Since there is nothing on the standard output, the command succeeds, closes the file descriptor, leaving the file empty.

Creating a new empty file is also called touching and can be done by $ touch file.txt command. The touch command can also be used for changing timestamps of the commands. Touch, however, won't wipe the file clean, it will only change the access and modification timestamps to the current time.

#13. Create a tunnel from localhost:2001 to somemachine:80

$ ssh -N -L2001:localhost:80 somemachine

This one-liner creates a tunnel from your computer's port 2001 to somemachine's port 80. Each time you connect to port 2001 on your machine, your connection gets tunneled to somemachine:80.

The -L option can be summarized as -L port:host:hostport. Whenever a connection is made to localhost:port, the connection is forwarded over the secure channel, and a connection is made to host:hostport from the remote machine.

The -N option makes sure you don't run shell as you connect to somemachine.

To make things more concrete, here is another example:

$ ssh -f -N -L2001:www.google.com:80 somemachine

This one-liner creates a tunnel from your computer's port 2001 to www.google.com:80 via somemachine. Each time you connect to localhost:2001, ssh tunnels your request via somemachine, where it tries to open a connection to www.google.com.

Notice the additional -f flag - it makes ssh daemonize (go into background) so it didn't consume a terminal.

#14. Reset terminal

$ reset

This command resets the terminal. You know, when you have accidentally output binary data to the console, it becomes messed up. The reset command usually cleans it up. It does that by sending a bunch of special byte sequences to the terminal. The terminal interprets them as special commands and executes them.

Here is what BusyBox's reset command does:

printf("\033c\033(K\033[J\033[0m\033[?25h");

It sends a bunch of escape codes and a bunch of CSI commands. Here is what they mean:

\033c: "ESC c" - sends reset to the terminal.
\033(K: "ESC ( K" - reloads the screen output mapping table.
\033[J: "ESC [ J" - erases display.
\033[0m: "ESC [ 0 m" - resets all display attributes to their defaults.
\033[?25h: "ESC [ ? 25 h" - makes cursor visible.

#15. Tweet from the shell

$ curl -u user:pass -d status='Tweeting from the shell' http://twitter.com/statuses/update.xml

This one-liner tweets your message from the terminal. It uses the curl program to HTTP POST your tweet via Twitter's API.

The -u user:pass argument sets the login and password to use for authentication. If you don't wish your password to be saved in the shell history, omit the :pass part and curl will prompt you for the password as it tries to authenticate. Oh, and while we are at shell history, another way to omit password from being saved in the history is to start the command with a space! For example, <space>curl ... won't save the curl command to the shell history.

The -d status='...' instructs curl to use the HTTP POST method for the request and send status=... as POST data.

Finally, http://twitter.com/statuses/update.xml is the API URL to POST the data to.

Talking about Twitter, I'd love if you followed me on Twitter! :)

#16. Execute a command at midnight

$ echo cmd | at midnight

This one-liner sends the shell command cmd to the at-daemon (atd) for execution at midnight.

The at command is light on the execution-time argument, you may write things like 4pm tomorrow to execute it at 4pm tomorrow, 9pm next year to run it on the same date at 9pm the next year, 6pm + 10 days to run it at 6pm after 10 days, or now +1minute to run it after a minute.

Use atq command to list all the jobs that are scheduled for execution and atrm to remove a job from the queue.

Compared to the universally known cron, at is suitable for one-time jobs. For example, you'd use cron to execute a job every day at midnight but you would use at to execute a job only today at midnight.

Also be aware that if the load is greater than some number (for one processor systems the default is 0.8), then atd will not execute the command! That can be fixed by specifying a greater max load to atd via -l argument.

#17. Output your microphone to other computer's speaker

$ dd if=/dev/dsp | ssh username@host dd of=/dev/dsp

The default sound device on Linux is /dev/dsp. It can be both written to and read from. If it's read from then the audio subsystem will read the data from the microphone. If it's written to, it will send audio to your speaker.

This one-liner reads audio from your microphone via the dd if=/dev/dsp command (if stands for input file) and pipes it as standard input to ssh. Ssh, in turn, opens a connection to a computer at host and runs the dd of=/dev/dsp (of stands for output file) on it. Dd of=/dev/dsp receives the standard input that ssh received from dd if=/dev/dsp. The result is that your microphone gets output on host computer's speaker.

Want to scare your colleague? Dump /dev/urandom to his speaker by dd if=/dev/urandom.

#18. Create and mount a temporary RAM partition

# mount -t tmpfs -o size=1024m tmpfs /mnt

This command creates a temporary RAM filesystem of 1GB (1024m) and mounts it at /mnt. The -t flag to mount specifies the filesystem type and the -o size=1024m passes the size sets the filesystem size.

If it doesn't work, make sure your kernel was compiled to support the tmpfs. If tmpfs was compiled as a module, make sure to load it via modprobe tmpfs. If it still doesn't work, you'll have to recompile your kernel.

To unmount the ram disk, use the umount /mnt command (as root). But remember that mounting at /mnt is not the best practice. Better mount your drive to /mnt/tmpfs or a similar path.

If you wish your filesystem to grow dynamically, use ramfs filesystem type instead of tmpfs. Another note: tmpfs may use swap, while ramfs won't.

#19. Compare a remote file with a local file

$ ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

This one-liner diffs the file /path/to/localfile on local machine with a file /path/to/remotefile on host machine.

It first opens a connection via ssh to host and executes the cat /path/to/remotefile command there. The shell then takes the output and pipes it to diff /path/to/localfile - command. The second argument - to diff tells it to diff the file /path/to/localfile against standard input. That's it.

#20. Find out which programs listen on which TCP ports

# netstat -tlnp

This is an easy one. Netstat is the standard utility for listing information about Linux networking subsystem. In this particular one-liner it's called with -tlnp arguments:

-t causes netstat to only list information about TCP sockets.
-l causes netstat to only list information about listening sockets.
-n causes netstat not to do reverse lookups on the IPs.
-p causes netstat to print the PID and name of the program to which the socket belongs (requires root).

To find more detailed info about open sockets on your computer, use the lsof utility. See my article "A Unix Utility You Should Know About: lsof" for more information.

That's it for today.

Tune in the next time for "Another Ten One-Liners from CommandLineFu Explained". There are many more nifty commands to write about. But for now, have fun and see ya!

PS. Follow me on twitter for updates!

Top Ten One-Liners from CommandLineFu Explained

2010-06-01T18:44:19Z

I love working in the shell. Mastery of shell lets you get things done in seconds, rather than minutes or hours, if you chose to write a program instead.

In this article I'd like to explain the top one-liners from the commandlinefu.com. It's a user-driven website where people get to choose the best and most useful shell one-liners.

But before I do that, I want to take the opportunity and link to a few of my articles that I wrote some time ago on working efficiently in the command line:

Update: Russian translation now available.

And now the explanation of top one-liners from commandlinefu.

#1. Run the last command as root

$ sudo !!

We all know what the sudo command does - it runs the command as another user, in this case, it runs the command as superuser because no other user was specified. But what's really interesting is the bang-bang !! part of the command. It's called the event designator. An event designator references a command in shell's history. In this case the event designator references the previous command. Writing !! is the same as writing !-1. The -1 refers to the last command. You can generalize it, and write !-n to refer to the n-th previous command. To view all your previous commands, type history.

This one-liner is actually really bash-specific, as event designators are a feature of bash.

I wrote about event designators in much more detail in my article "The Definitive Guide to Bash Command Line History." The article also comes with a printable cheat sheet for working with the history.

#2. Serve the current directory at http://localhost:8000/

$ python -m SimpleHTTPServer

This one-liner starts a web server on port 8000 with the contents of current directory on all the interfaces (address 0.0.0.0), not just localhost. If you have "index.html" or "index.htm" files, it will serve those, otherwise it will list the contents of the currently working directory.

It works because python comes with a standard module called SimpleHTTPServer. The -m argument makes python to search for a module named SimpleHTTPServer.py in all the possible system locations (listed in sys.path and $PYTHONPATH shell variable). Once found, it executes it as a script. If you look at the source code of this module, you'll find that this module tests if it's run as a script if __name__ == '__main__', and if it is, it runs the test() method that makes it run a web server in the current directory.

To use a different port, specify it as the next argument:

$ python -m SimpleHTTPServer 8080

This command runs a HTTP server on all local interfaces on port 8080.

#3. Save a file you edited in vim without the needed permissions

:w !sudo tee %

This happens to me way too often. I open a system config file in vim and edit it just to find out that I don't have permissions to save it. This one-liner saves the day. Instead of writing the while to a temporary file :w /tmp/foobar and then moving the temporary file to the right destination mv /tmp/foobar /etc/service.conf, you now just type the one-liner above in vim and it will save the file.

Here is how it works, if you look at the vim documentation (by typing :he :w in vim), you'll find the reference to the command :w !{cmd} that says that vim runs {cmd} and passes it the contents of the file as standard input. In this one-liner the {cmd} part is the sudo tee % command. It runs tee % as superuser. But wait, what is %? Well, it's a read-only register in vim that contains the filename of the current file! Therefore the command that vim executes becomes tee current_filename, with the current directory being whatever the current_file is in. Now what does tee do? The tee command takes standard input and write it to a file! Rephrasing, it takes the contents of the file edited in vim, and writes it to the file (while being root)! All done!

#4. Change to the previous working directory

$ cd -

Everyone knows this, right? The dash "-" is short for "previous working directory." The previous working directory is defined by $OLDPWD shell variable. After you use the cd command, it sets the $OLDPWD environment variable, and then, if you type the short version cd -, it effectively becomes cd $OLDPWD and changes to the previous directory.

To change to a directory named "-", you have to either cd to the parent directory and then do cd ./- or do cd /full/path/to/-.

#5. Run the previous shell command but replace string "foo" with "bar"

$ ^foo^bar^

This is another event designator. This one is for quick substitution. It replaces foo with bar and repeats the last command. It's actually a shortcut for !!:s/foo/bar/. This one-liner applies the s modifier to the !! event designator. As we learned from one-liner #1, the !! event designator stands for the previous command. Now the s modifier stands for substitute (greetings to sed) and it substitutes the first word with the second word.

Note that this one-liner replaces just the first word in the previous command. To replace all words, add the g modifer (g for global):

$ !!:gs/foo/bar

This one-liner is also bash-specific, as event designators are a feature of bash.

Again, see my article "The Definitive Guide to Bash Command Line History." I explain all this stuff in great detail.

#6. Quickly backup or copy a file

$ cp filename{,.bak}

This one-liner copies the file named filename to a file named filename.bak. Here is how it works. It uses brace expansion to construct a list of arguments for the cp command. Brace expansion is a mechanism by which arbitrary strings may be generated. In this one-liner filename{,.bak} gets brace expanded to filename filename.bak and puts in place of the brace expression. The command becomes cp filename filename.bak and file gets copied.

Talking more about brace expansion, you can do all kinds of combinatorics with it. Here is a fun application:

$ echo {a,b,c}{a,b,c}{a,b,c}

It generates all the possible strings 3-letter from the set {a, b, c}:

aaa aab aac aba abb abc aca acb acc
baa bab bac bba bbb bbc bca bcb bcc
caa cab cac cba cbb cbc cca ccb ccc

And here is how to generate all the possible 2-letter strings from the set of {a, b, c}:

$ echo {a,b,c}{a,b,c}

It produces:

aa ab ac ba bb bc ca cb cc

If you liked this, you may also like my article where I defined a bunch of set operations (such as intersection, union, symmetry, powerset, etc) by using just shell commands. The article is called "Set Operations in the Unix Shell." (And since I have sets in the shell, I will soon write articles on on "Combinatorics in the Shell" and "Algebra in the Shell". Fun topics to explore. Perhaps even "Topology in the Shell" :))

#7. mtr - traceroute and ping combined

$ mtr google.com

MTR, bettern known as "Matt's Traceroute" combines both traceroute and ping command. After each successful hop, it sends a ping request to the found machine, this way it produces output of both traceroute and ping to better understand the quality of link. If it finds out a packet took an alternative route, it displays it, and by default it keeps updating the statistics so you knew what was going on in real time.

#8. Find the last command that begins with "whatever," but avoid running it

$ !whatever:p

Another use of event designators. The !whatever designator searches the shell history for the most recently executed command that starts with whatever. But instead of executing it, it prints it. The :p modifier makes it print instead of executing.

This one-liner is bash-specific, as event designators are a feature of bash.

Once again, see my article "The Definitive Guide to Bash Command Line History." I explain all this stuff in great detail.

#9. Copy your public-key to remote-machine for public-key authentication

$ ssh-copy-id remote-machine

This one-liner copies your public-key, that you generated with ssh-keygen (either SSHv1 file identity.pub or SSHv2 file id_rsa.pub) to the remote-machine and places it in ~/.ssh/authorized_keys file. This ensures that the next time you try to log into that machine, public-key authentication (commonly referred to as "passwordless authentication.") will be used instead of the regular password authentication.

If you wished to do it yourself, you'd have to take the following steps:

your-machine$ scp ~/.ssh/identity.pub remote-machine:
your-machine$ ssh remote-machine
remote-machine$ cat identity.pub >> ~/.ssh/authorized_keys

This one-liner saves a great deal of typing. Actually I just found out that there was a shorter way to do it:

your-machine$ ssh remote-machine 'cat >> .ssh/authorized_keys' < .ssh/identity.pub

#10. Capture video of a linux desktop

$ ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq /tmp/out.mpg

A pure coincidence, I have done so much video processing with ffmpeg that I know what most of this command does without looking much in the manual.

The ffmpeg generally can be descibed as a command that takes a bunch of options and the last option is the output file. In this case the options are -f x11grab -s wxga -r 25 -i :0.0 -sameq and the output file is /tmp/out.mpg.

Here is what the options mean:

-f x11grab makes ffmpeg to set the input video format as x11grab. The X11 framebuffer has a specific format it presents data in and it makes ffmpeg to decode it correctly.
-s wxga makes ffmpeg to set the size of the video to wxga which is shortcut for 1366x768. This is a strange resolution to use, I'd just write -s 800x600.
-r 25 sets the framerate of the video to 25fps.
-i :0.0 sets the video input file to X11 display 0.0 at localhost.
-sameq preserves the quality of input stream. It's best to preserve the quality and post-process it later.

You can also specify ffmpeg to grab display from another x-server by changing the -i :0.0 to -i host:0.0.

If you're interested in ffmpeg, here are my other articles on ffmpeg that I wrote while ago:

PS. This article was so fun to write, that I decided to write several more parts. Tune in the next time for "The Next Top Ten One-Liners from CommandLineFu Explained" :)

Have fun. See ya!

PSS. Follow me on twitter for updates.

This article was sponsored by:

Canadian domain names available through Netfirms.ca
Hosting.com offers Dedicated server hosting
Advanceware's Inventory management software can help you organize your resources.
Sherweb provides Hosted Exchange solutions for your small business.

Contact me, if you wish to sponsor any other of my existing posts or future posts!

Visual Math Friday

2010-03-05T15:58:12Z

Hey everyone, I'm starting a new article series here on catonmat that will be published on Fridays. It's called "Visual Math Friday." The goal of this article series is to assemble all the wonderful mathematical proofs and identities that can be proven in a single picture or drawing. I have been collecting them for years from various books, math journals, websites and video lectures and have collected several hundred of them.

If you have been following my blog, you'll have noticed that I love to explain things. I'll use the same writing style in this series as well. I'll post the visual proof together with my explanation. In the explanations I'll walk you through the proof, suggesting how to think about the visual image. But it's best if you try to understand the proof without reading my explanation and only then read it.

The proofs come from algebra, number theory, geometry, calculus, complex analysis, topology and various other fields of mathematics. I'll do one proof a week and will try to alternate between really easy, moderate and difficult proofs every week so that readers with various mathematical backgrounds can enjoy the series.

I'll start with a really easy integer sum proof this week. Without further ado, here it is.

Now try to figure what it proves without reading further! The colors play an important role in the visual proofs, as do the shape of the visual drawing, the shapes on it and their arrangement!

The Proof

This picture proves that the sum of first n odd numbers is equal to n². In other words:

1 + 3 + 5 + 7 + 9 + 11 + ... + (2n-1) = n².

Explanation

Imagine that the blue and green dots are marbles. Now let's look at the lower left corner. One blue marble is located there. Next if we follow the diagonal to the upper right corner, we see that the 1 blue marble is kind-of wrapped in 3 green marbles, then they in turn are wrapped in 5 blue marbles again, then those are wrapped in 7 green ones, which in turn are wrapped in 9 blue ones, then 11 green ones, and finally 13 ones.

This suggests that we could look at the sum 1 + 3 + 5 + 7 + 9 + 11 + 13. Well what is it? It's 49. Also what does this sum involve? It involves only odd numbers. Interesting.

Now let's count how many marbles we have on each side. 7 on one side and 7 on the other. And what is 7x7? It's 49! And how many numbers did we sum together? Well, seven - 1, 3, 5, 7, 9, 11, 13. Interesting.

Could it be that the sum of the first n odd numbers is equal to n²? Or in other words, is 1 + 3 + 5 + 7 + ... + (2n-1) equal to n²?

Let's try to prove this hypothesis by going one number further and adding 15 green marbles - 13 filled marbles and 2 semi-filled marbles:

Adding 15 marbles created a square of marbles again - now we have 64 marbles, 8 on each side.

The 2 semi-filled marbles are very important here. Each time we go further from n odd marbles to n+1 odd marbles, the difference changes by 2. This difference of two marbles can always be put where the semi-filled ones are in this picture.

This actually concludes the proof. We showed that the proof worked for several marbles, and showed how it still works if we went one step further. Therefore it works for all future steps. This method of proof is called mathematical induction and it's a very powerful and important tool in mathematics for proving sums like these.

Before we go

Can you figure out the two proofs hidden in the logo of this post?

Let me know in the comments if you can!

Get ahead of me

By the way, if you wish to get ahead of me in this series, you can get these three amazing books on visual proofs:

Here are these books on Amazon.com:

Vim Plugins You Should Know About, Part VII: ragtag.vim (formerly allml.vim)

2010-03-08T17:19:49Z

This is the seventh post in the article series "Vim Plugins You Should Know About". This time I am going to introduce you to a plugin called "ragtag.vim". A month ago it was still known as "allml.vim" but now it has been renamed to ragtag.vim.

The best parts of RagTag are mappings for editing HTML tags. It has a mapping for quickly closing open HTML tags, a mapping for quickly turning the typed word into a pair of open/close HTML tags, several mappings for inserting HTML doctype, linking to CSS stylesheets, loading JavaScript <script src="...">...</script> and it includes mappings for wrapping the typed text in a pair of <?php ... ?> tags for PHP, or <% ... %> for ASP or eRuby, and {% .. %} for Django.

RagTag is written by Tim Pope. He's the master of Vim plugin programming. I have already written about two of his plugins - surround.vim and repeat.vim and more articles about his plugins are coming!

Previous articles in the series.

Part I: surround.vim - find and edit surrounding parens, quotes, html tags, etc.
Part II: repeat.vim - repeat the previous surrounding command.
Part III: matchit.vim - extends the functionality of % key.
Part IV: snipmate.vim - the best snippet plugin for vim
Part V: a.vim - quickly change between source and header files (.c/.h).
Part VI: nerd_tree.vim - awesome filesystem exploration plugin.

Here are examples of using the RagTag plugin.

Quickly closing an open HTML tag.

Suppose you have typed <div> and you want to close it without typing the whole closing tag </div> yourself.

The quick way to do it with RagTag is to press CTRL+X /. This mapping automatically closes the last open HTML tag.

Extra tip: If you didn't have this plugin you could quickly close the tag by typing </ and pressing CTRL+X CTRL+O. The default Vim mapping CTRL+X CTRL+O guesses the item that may come after the cursor. In case of an open HTML tag it's the close tag.

Creating a pair of open/close HTML tags from a word.

Suppose you want to quickly create a pair of <div></div> and place the cursor between the two tags.

The quick way to do it with RagTag is to type div and press CTRL+X SPACE. This mapping takes the typed word and creates a pair of HTML tags, one closing tag and one open tag on the same line.

However, if you wish to create open/close tag pair separated by a newline, type CTRL+X ENTER.

Here is an example, if you just typed div and then press CTRL+X ENTER it will produce the following output:

<div>
|
</div>

| indicates the position of cursor.

Insert HTML doctype.

If you type CTRL+X ! RagTag will display a list of HTML doctypes to choose from. Defaults to HTML 4.01 Strict.

This mapping is not that useful, given that I already introduced snipmate.vim plugin for creating snippets. Using snipmate.vim you can create a snippet "h" that would insert the whole HTML structure, including doctype, html, body, title, meta tags, etc.

Link to a CSS stylesheet.

Typing CTRL+X @ inserts the snippet for linking to a CSS stylesheet.

<link rel="stylesheet" type="text/css" href="/stylesheets/|.css">

This is again not that useful, given that we have snipmate.vim.

The mapping is easy to remember because @ is used for importing in CSS.

Insert meta content-type tag.

Typing CTRL+X # inserts the HTML meta tag for document's content type and encoding.

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">

The charset depends on document's charset. If it's utf-8, the mapping will set the charset to utf-8 in the meta tag.

Load JavaScript document.

Typing CTRL+X $ links to a JavaScript file.

<script type="text/javascript" src="/javascripts/|.js">

The mapping is easy to remember because $ is a valid char in identifiers in many languages.

Wrap the typed text in PHP, Django, eRuby template tags.

There are several different mappings for wrapping text in template tags. It's best to summarize them in the following table. The table assumes you had just typed "foo" and you are editing an eRuby document:

Mapping    Result
---------  -----------
CTRL+X =   foo<%= | %>
CTRL+X +   <%= foo| %>
CTRL+X -   foo<% | %>
CTRL+X _   <% foo| %>
CTRL+X '   foo<%# | %>
CTRL+X "   <%# foo| %>

What this table shows us is, for example, that if you have typed "foo" and press CTRL+X _, the plugin will wrap "foo" in <% %> tags and place the cursor after "foo".

Summary of all the RagTag mappings.

CTRL+X /       Close the last open HTML tag
CTRL+X SPACE   Create open/close HTML tags from the typed word
CTRL+X CR      The same as CTRL+X SPACE but puts a newspace in between
CTRL+X !       Insert HTML doctype
CTRL+X @       Insert CSS stylesheet
CTRL+X #       Insert meta content-type meta tag
CTRL+X $       Load JavaScript document

For the following mappings, suppose that
you have typed "foo".

Mapping        Result
---------      -----------
CTRL+X =       foo<%= | %>
CTRL+X +       <%= foo| %>
CTRL+X -       foo<% | %>
CTRL+X _       <% foo| %>
CTRL+X '       foo<%# | %>
CTRL+X "       <%# foo| %>

How to install ragtag.vim?

To get the latest version:

1. Download ragtag.zip.
2. Extract ragtag.zip to ~/.vim (on Unix/Linux) or ~\vimfiles (on Windows).
3. Run :helptags ~/.vim/doc (on Unix/Linux) or :helptags ~\vimfiles\doc (on Windows) to rebuild the tags file (so that you can read help :help ragtag.)
4. Restart Vim or source ragtag.vim by typing :so ~/.vim/plugin/ragtag.vim (on Unix/Linux) or :so ~\vimfiles\plugin\ragtag.vim

Have Fun!

Have fun closing those HTML tags and until next time!