A few months ago, while I was creating the Reddit River website, I noticed that Python’s standard library included a curses module. Having worked with curses and Curses Development Kit in C, I decided to refresh my curses skills, this time in Python.

Coding the application started with creating two separate Python modules for retrieving stories from Hacker News and Reddit.

If you look at the source code of Hacker Top and Reddit Top programs, you’ll notice two Python modules called “pyhackerstories.py” and “pyredditstories.py“.

Both of these modules follow the same interface and provide function get_stories(). The core functionality of this function can be easily understood from this code fragment:

stories = []
for i in range(pages):
    content = _get_page(url)
    entries = _extract_stories(content)
    stories.extend(entries)
    url = _get_next_page(content)
    if not url:
        break

The function iterates over the given number of Reddit or Hacker News pages and creates a list of objects (of type Story) containing information about stories on each page.

This function takes two optional parameters ‘pages‘ and ‘new‘ (pyredditstories.py also takes an optional ‘subreddit‘ parameter). These parameters control how many pages of (new) stories to scrape.

Here is an example of using pyhackerstories.py module to get the titles and scores of the first five most popular stories on Hacker News:

>>> from pyhackerstories import get_stories
>>> stories = get_stories()
>>> for story in stories[:5]:
...     print "%-3d - %s" % (story.score, story.title)
...
30  - Rutgers Graduate Student Finds New Prime-Generating Formula
59  - Xobni VP Engineering leaves for own startup
69  - The Pooled-Risk Company Management Company
14  - Jeff Bonforte, CEO of Xobni, explains why Gabor left
52  - Google's Wikipedia clone Knol launches.

Each Story object contains the following properties:

• position - story position
• id - identifier used by Hacker News or Reddit to identify the story
• title - title of the story
• url - web address of the story
• user - username of the user who submitted the story
• score - number of upvotes the story has received
• human_time - time the story was submitted
• unix_time - unix time the story was submitted
• comments - number of comments the story has received

Here is another example of retrieving most active Reddit users on Programming Subreddit (based on 5 pages of stories):

>>> from pyredditstories import get_stories
>>> stories = get_stories(subreddit='programming', pages=5)
>>> userdict = {}
>>> for story in stories:
...     userdict[story.user] = userdict.setdefault(story.user, 0) + 1
>>> users = [(userdict[u], u) for u in userdict]
>>> users.sort()
>>> users.reverse()
>>> for user in users[:5]:
...  print "%s: %d" % (user[1], user[0])
...
gst: 19
dons: 6
llimllib: 4
synthespian: 3
gthank: 3

The pyhackerstories.py and pyredditstories.py Python modules can also be used as standalone applications.

Executing pyredditstories.py with ‘–help‘ command line argument tells that:

$ ./pyredditstories.py  --help
usage: pyredditstories.py [options]

options:
  -h, --help   show this help message and exit
  -sSUBREDDIT  Subreddit to retrieve stories from. Default:
               front_page.
  -pPAGES      How many pages of stories to output. Default: 1.
  -n           Retrieve new stories. Default: nope.

Here is an example of executing pyredditstories.py to get five most popular programming stories:

$ ./pyredditstories.py -s programming | grep '^title' | head -5

title: "Turns out my nephew is really good with computers, so we're going to give him the job!"
title: Sun Microsystems funding Haskell on multicore OpenSPARC!
title: I love parser combinators [Haskell]
title: 2D collision detection for SVG - demo of intersection routines with SVG/Javascript
title: Priorities: Solaris vs Linux

That’s about enough information about these modules to create all kinds of wonderful things! Just play around a little!

Download pyhackerstories.py: pyhackerstories.py
Download pyredditstories.py: pyredditstories.py

Now I’ll briefly explain how the curses user interface was made. I’ll cover only the ideas and will not go deeply into code.

I started with reading Curses Programming with Python howto. This howto explained how to do all the basic curses operations in Python - how to get into curses mode, how to create windows, print colorful text and how to handle user input.

The user interface had two requirements - it had to be responsive while the new stories were being retrieved from the sites, and it had to be independent from the website, meaning that it should be easy to display data from a different website with minor (or no) modifications to the interface code.

The first requirement was satisfied by creating a separate thread (see Retriever class) which periodically called get_stories() and passed the stories to the interface via a queue.

The second requirement was already satisfied when I created the pyredditstories.py and pyhackerstories.py modules. As I mentioned, these modules provided the same interface for retrieving stories and could be used almost interchangeably.

That’s basically it! If you have any specific questions, feel free to ask them in the comments!

Ps. I am thinking of releasing Dzone Top and Digg Top, and then merge all these Top programs into a single social news console program. Are there any volunteers who would like to help me?

Reddit has long been my favorite site for news but very frequently I found myself reading nonsense which had nothing to do with my real passion. I decided to take a radical step and turn Reddit into the definitive source for programming news, or the way I like to say, I decided to read Reddit the fanatic programmer way.

Here is how I did it.

I said no to all the default Reddit communities.

And subscribed to various programming subreddits!

Note: I am also subscribed to math and physics subreddits as those subjects are my hobbies.

This picture includes only a fraction of the programming subreddits I subscribed to! Here is the full list:

• Programming
• Python
• Computer Science
• Ruby
• Javascript
• C++
• Lisp
• PHP
• Functional
• Perl
• Erlang
• Types
• Haksell
• Assembly Programming
• C Programming
• C Language
• Scheme
• Prolog

I am also subscribed to some related communities:

• Linux
• Network Security
• Math
• Video Lectures
• Physics
• Hacking
• Cheat Sheets
• Unix
• Geek Music
• Catonmat

Doing so turned Reddit’s front page into a definitive source for programming news.

I invite you to share your methods for following the latest programming buzz in the comments!

During this year (July 14, 2007 - July 20, 2008) I managed to write 58 posts, which received 808 comments. Based on statistics from Statcounter and Google Analytics, these posts received 574,874 views by 424,292 unique visitors.

Here is a Google Analytics graph showing monthly page views for this period (click for a larger version):

Here are the top 5 countries my blog readers came from:

• United States (195,076 visitors)
• United Kingdom (25,988 visitors)
• Canda (25,335 visitors)
• India (13,753 visitors)
• Germany (13,670 visitors)

Not surprisingly, the top 5 referring sites were all social media and bookmarking sites:

• reddit.com (84,391 visitors)
• digg.com (43,903 visitors)
• stumbleupon.com (20,234 visitors)
• del.icio.us (13,146 visitors)
• news.ycombinator.com (10,757 visitors)

From all the visitors 48,386 went to my site directly and 82,502 were sent here by my darling Google.

During the first year, approximately 1000 people subscribed to my blog. Here is the Feedburner subscriber graph for the year:

If you are interested in my blog, you may subscribe here: catonmat rss feed.

According to Technorati, my blog has received 476 blog reactions and ranks 29,521-st out of 112.8 million blogs!

Many of my posts have been submitted to Reddit, Digg and have been Stumbled. Here are the top 5 most visited posts:

• Learning JavaScript through Video Lectures (44,548 views)
• Designing Digg Picture Website in a Matter of Hours (43,962 views)
• The Definitive Guide to Bash Command Line History (31,568 views)
• Working Productively in Bash’s Vi Command Line Editing Mode (23,508 views)
• How to Extract Audio Tracks from YouTube Videos (22,588 views)

Some of my how-to posts came with downloadable cheat sheets. Here are the top 5 cheat sheets:

• Sed, UNIX Stream Editor, Cheat Sheet (downloaded 12,162 times)
• Perl’s pack()/unpack() and printf() Cheat Sheet (downloaded 10,849 times)
• AWK Programming Language Cheat Sheet (downloaded 10,303 times)
• Screen Terminal Emulator Cheat Sheet (downloaded 6,706 times)
• Bash VI Editing Mode Cheat Sheet (downloaded 5,452 times)

During the last year I also did several web projects. As I was busy with physics studies, I created only four web projects (all of these projects are open source):

• Reddit Media: intelligent fun online!
• Digpicz: the missing digg picture section!
• Picurls: buzziest pictures on the net!
• Reddit River: what flows online!

In March 2008 I started posting geek music on Fridays. Here are the top 5 geek songs:

• Crypt-o (played 5,153 times)
• Model-View-Controller Song (played 4,700 times)
• Code Monkey (played 2,186 times)
• God Wrote in Lisp (played 1,952 times)
• The Day the Routers Died (played 1,462 times)

I am satisfied with where this blog is heading. I’d like to thank all my fans and all the visitors who regularly return to the site!

To make things more challenging, I am setting myself a goal of reaching 5000 subscribers by the end of the next year of blogging (July 2009)! I know that this is very ambitious goal but I am ready to take the challenge!

The song is written and performed by Brent Simon. Brent describes himself as a super nerd who plays the synth and makes original music that’s honest and from the heart. He keeps composing and his latest music can be found on his MySpace page - brentsimon.

The song, when I first downloaded it, was actually called “Mininova”, in honor of one of the largest BitTorrent sites on the net - Mininova.org.

Here it is! The BitTorrent Song!

Download this song: the bittorrent song (musical geek friday #11)
Downloaded: 329 times

Download lyrics: the bittorrent song lyrics (musical geek friday #11)
Downloaded: 45 times

Here is the lyrics of The BitTorrent Song:

-Verse 1-
Gather ’round and hear my story
‘Bout the new old west
I sought torrents, rips, and ISO’s
Logged in under guest

Movies, music, games and porno,
Warez and killer apps,
Compressors, CODECs and keygens
Compilers, hacks and cracks

-Chorus-
Data’s streamin’ my hard drive’s screamin’
Fragmenting the night
Bit’s are flowin’ corruption’s growin’
Into every last byte

-Verse 2-
Bad command or file name
OK, now what’s next
D-I-R where is my brain
It’s coded in base hex

Alright now you’ll feel my wrath
Don’t ever mess with me
Cram this up your Gigabyte
So long Format C:

-Chorus-
Data’s streamin’ my hard drive’s screamin’
Fragmenting the night
Bit’s are flowin’ corruption’s growin’
Into every last byte

-Bonus Verse-
If you watch my lips real closely
You’ll see they don’t match what I’m saying
My typewriter emits fluids
All of which are intoxicating

Staring at my shoes all after-
Noon on heroin isn’t boring
Just to poop I need the surgical
Equivalent of apple coring

-Chorus-
Naked Lunch
Naked Lunch
Naked Lunch
Naked Lunch

Here is Brent Simon performing The Bittorrent Song:

This lecture is given by Neil Daswani, who has a Ph.D. from Stanford and currently works at Google as a security engineer. He is also an author of a book entitled “Foundations of Security: What Every Programmer Needs to Know“, which teaches you state-of-the-art software security design principles, methodology, and concrete programming techniques you need to build secure software systems.

Neil talks about top three web application vulnerabilities that cybercriminals use to steal money. These three vulnerabilities are:

• SQL Injection attacks,
• Cross-Site Request Forgery (XSRF) attacks, and
• Cross-Site Script Inclusion (XSSI) attacks.

I was surprised that he did not cover plain, old Cross-Site Scripting (XSS) attacks, but jumped right to dynamic XSS. You’ll have to get familiar with this type attack on your own. See the XSS Faq and XSS Cheat Sheet for more information!

Interesting points from the lecture:

• [01:48] Years ago cybercriminals were teenagers writing viruses and worms, today they are organized crime looking for stealing money.
• [03:19] Intermediate goals to stealing money are data theft, extortion and malware distribution.
• [04:02] Russian Business Network (RBN) is an example of organized cybercrime.
• [09:00] Attack #1: SQL Injection.
• [16:30] Preventing SQL injections.
• [17:00] Don’t blacklist (filter) characters in queries. Whitelist (allow) well-defined set of safe values for each field.
• [18:30] Take a look at mod_security if you use Apache web server. Mod_security is a Web Application Firewall. It allows you to define a set of rules the web application must follow.
• [19:30] Prepared statements and bind variables help to avoid SQL injections.
• [23:00] Other mitigations strategies include - limiting web application user’s privileges on the sql server, hardenining database server and host operating system.
• [23:45] Second order SQL injections (link to pdf) abuse data that is already in the database.
• [23:55] Blind SQL injection (link to pdf) is a technique to reverse engineer the structure of the database.
• [24:25] Attack #2: Cross-Site Request Forgery (XSRF).
• [26:00] How XSRF Works.
• [31:30] Drive-By-Pharming (pdf) is an XSRF technique where the attacker changes DNS settings of a users broadband router (fact - 50% of home users do not change default router password).
• [34:00] Preventing XSRF.
• [34:20] Check Referer HTTP header. That doesn’t always work because the user might be using a proxy.
• [36:15] Validate the user by asking him to provide his password or any other token only the user has knowledge of.
• [37:15] Validate requests via “Action Tokens” which add special tokens to forms to distinguish them from forged forms.
• [38:30] Attack #3: Cross-Site Script Inclusion (XSSI).
• [39:10] How XSSI works.
• [41:20] Dynamic script inclusion example.
• [47:25] Trends.
• [50:12] Open Web Application Security Project (OWASP) Top 10 vulnerabilities in 2007 (link).
• [53:55] Google has some material on Web Security at code.google.com/edu.

Happy hacking! (just kidding )

In this talk, professor Tao Xie speaks about his research on using public code repositories together with code search engines for finding common API usage patterns and anti-patterns.

His research software uses the following four code search engines.

• Google Code Search with billions of lines of code.
• Krugle with 2.5 billion LOC.
• Koders with 760 million LOC.
• Codase with 250 million LOC.

He suggests to view Raphael Volz’s analysis for more information about these search engines.

Tao has developed three tools, which use the aforementioned search engines:

• PARSEWeb for finding API usage patterns,
• XWeb for finding forgotten exception handlers, and
• NEGWeb for finding misuses of API calls.

See the code mining project website for more information.

The lecture is done in a very academic manner and it’s very hard to follow. Be sure that you are really interested in this topic before watching it.

Some excerpts from the lecture:

• [04:26] A problem with data mining on source code is that it might not have enough data points (usages of API) to discover common patterns.
• [04:58] It is crucial to have a lot of data points to get good results out of data mining
• [08:37] Google Code Search indexes publicly hosted SVN and CVS repositories.
• [09:20] Example of searching for C stdlib’s fopen usage on Google Code Search (query: “lang:C file:.c$ fopen\s*\(”
• [11:08] Example of the same search on Krugle.
• [16:40] Code search engines return partial code samples. Various heuristics are used for type inference.
• [22:05] Example of integrating Tao’s PARSEWeb into Eclipse.
• [28:15] Interesting idea of constructing and issuing multiple queries to find more code samples.
• [36:20] A study showed that a proper deallocation of resources after an exception resulted in 17% performance increase.

I’d like to hear some comments on websites that you use for finding code examples!

As I mentioned in the Hacker Top post, Reddit is my favorite source for news because of the great programmer community it has. I actually unsubscribed from all the default subreddits (politics, pics, etc.) and subscribed to some 20 - 30 programming subreddits (like python, erlang, compsci and many others).

Reddit Top was actually derived from Hacker Top. Hacker Top was made in such a manner that it required me just to write a new parser for Reddit to create the new program. The program is written in Python programming language and uses ncurses interface for displaying the stories.

Try the ‘m‘ keyboard shortcut to switch to other display modes

Download

Download link: reddit top program
Downloaded: 595 times

I’ll describe the process of creating the program in one of the next posts.
If you want to read about that, I suggest you subscribe to my rss feed.

Note - this program is released under GNU GPL.

How to run the program?

1) Make sure you are running a Unix type operating system.

2) Make sure you have Python installed. Any recent version will do.

3) Download and unpack the hacker top program archive.

$ wget 'http://www.catonmat.net/download/reddit-top.tgz'
$ tar -xvzf reddit-top.tgz

4) Change to 'reddit-top' directory which was created by unpacking the archive.

$ cd reddit-top

5) Give the ‘reddit_top.py’ program execute permissions.

$ chmod u+x reddit_top.py

6) Run the reddit_top.py program.

$ ./reddit_top.py

(If that does not work out, try running 'python ./reddit_top.py')

Make sure that your terminal is at least 80 columns wide, otherwise the program won’t be able to display the results nicely.

Command Line Options

If you run the program with '--help' argument, it will display the possible command line options:

Usage: ./reddit_top.py [-h|--help] - displays this
Usage: ./reddit_top.py [-s|--subreddit subreddit]
          [-i|--interval interval] [-n|--new]
          [-u|--utf8 ]

As the help message suggests, the four main options are:

• -s or –subreddit, which specifies the subreddit to monitor.
  The default is reddit’s front page (http://www.reddit.com).
  At the moment it is not possible to specify multiple subreddits. I’ll add the feature in the future. Here are a few examples of valid subreddits - ‘programming’, ‘wtf’, ‘python’, ‘politics’, and others.
• -i or –interval, which specifies refresh interval.
  The default refresh interval is 1 minutes. Here are a few examples: 10s (10 seconds), 12m (12 minutes), 2h (2 hours).
• -u or –utf8, turns on utf8 output mode.
  Default: off. Use this if you know for sure that your terminal supports it, otherwise you might get gibberish.
• -n or –new, which follows only the newest reddit stories on a given subreddit or front page.
  Default: follow front page stories.

Keyboard Shortcuts

There are several keyboard shortcuts which you should know about when using the Reddit Top program:

• q - quits the program.
• u - forces an update of the news.
• up arrow/down arrow (or alternatively j/k keys) - scrolls the news list up or down.
• m - changes the display mode. There are 5 different display modes for your taste.

Enjoy the program! I’ll write some details how I created it in one of the next posts. Stay tuned - subscribe to my rss feed!

TECO stands for Text Editor and COrrector. It was developed at Massachusetts Institute of Technology (MIT) and was one of the first text editors ever written. It grew over the years, gaining popularity and features. Along the way, it also became a Turing-complete programming language. Several sets of editor macros were developed and used. I was surprised to find that around 1975 Richard Stallman organized these Editor MACroS into the first Emacs-like text editor. Here is an article about the history of TECO.

DDT, on the other hand, was a collection of several debugger programs. These debugging programs included a command shell from which TECO was also used interchangeably with DDT. DDT stands for Dynamic Debugging Technique (initially known as DEC Debugging Tape).

Talking about the 10th geek song, it’s written and performed by a band called “red martian“.

Enough of facts, here is the TECO and DDT song:

Download this song: teco and ddt (musical geek friday #10)
Downloaded: 284 times

Download lyrics: teco and ddt lyrics (musical geek friday #10)
Downloaded: 51 times

TECO and DDT Lyrics:

Oh, you can hack anything that you want with just TECO and DDT.
You can hack anything that you want with just TECO and DDT!

Oh, no, you don’t have to tell me what would I need because it’s simply plain to see…
That I can hack anything that I want with just TECO and DDT!

Yeaha!

Oh, you can hack anything that you want with just TECO and DDT.
Oh, you can hack anything that you want with just TECO and DDT!

Oh, no, you don’t have to tell me what would I need because it’s simply plain to see…
That I can hack anything that I want with just TECO, just TECO, just TECO and DDT!

I’d like to thank Adam for letting me know about this song.

Chapter 3 of this book is written by Jon Bentley. The chapter is curiously entitled “The Most Beautiful Code I Never Wrote” and it talks about a beautiful implementation and experimental analysis of the most widely used sorting algorithm in the world called Quicksort.

Jon also decided to give a talk at Google on the same topic. In this techtalk, he talks about three Quicksort topics. In the first part Jon discusses an elegant and tiny implementation of Quicksort. In the next part he analyzes the Quick sort algorithm by instrumenting this elegant implementation. Finally, he talks about a production quality Quicksort implementation which has been included in C standard library (qsort function).

What I learned the most from this lecture is the experimental part of counting number of comparsions of an algorithm. I had never read about experimental analysis of algorithms.

Interesting ideas from the lecture:

• [02:30] What’s the most beautiful code you’ve ever written?
• [04:58] The classical Quicksort algorithm.
• [05:32] Best case (nlog(n)), worst (n²) and average case (nlog(n)) analysis results of Quicksort.
• [08:00] Randomization in Quicksort.
• [09:08] Lomuto’s idea of partitioning.
• [10:48] Partitioning algorithm implementation pseudocode.
• [11:10] Beautiful Quicksort code.
• [12:13] Strengths of Quicksort: simple to implement and explain, pretty fast. Weaknesses of Quicksort: quadratic time for increasing elements (fix by randomization), quadratic time for equal elements, not stable.
• [13:55] “vigorous writing is concise” /William Strunk Jr.’s/
• [14:49] Simplest semi-production C Quicksort implementation.
• [15:35] How much time will Quicksort take on average on indistinct elements?
• [17:00] Graph of (runtime/N, logN) for Quicksort and Heapsort.
• [18:31-32:00] Finding number of comparisons for Quicksort experimentally.
• [23:08] Doing various tricks the time to count comparisons reduced from nlog(n) to n and space reduced from n to log(n).
• [24:25] The Inventor’s Paradox: “The more ambitious plan may have more chance of success.”
• [30:13] Perlis: “Simplicity does not precede complexity, but follows it.”
• [30:45] Mathematical solution to Quicksort’s comparison complexity.
• [31:35] The solution is ~ 1.386nlog(n)
• [32:10] Analysis of comparisons in Quicksort is isomorphic to time taken to insert an element in a Binary Search Tree.
• [36:25] A beautiful bug report on the original implementation of C library’s qsort
• [39:31] Engineering qsort together with Doug McIlroy. Goals and strategy.
• [41:23] With equal elements, some Quicksorts go quadratic, some go nlog(n), some go linear.
• [41:46] Is there a quick algorithm for ternary (<, = , >) partitioning?
• [46:00] Knuth said that the overheard and comparisons took the same amount of time but the swaps really killed you, where as Jon and Doug found that time(overhead) < time(swaps) < time(comparisons), if swaps are done properly.
• [46:30] Choosing effective partitioning element was important. First, middle or random gave runtime aproximately 1.386nlog(n), where as median of three or median and three medians gave run times of 1.188nlog(n) and 1.09nlog(n), respectivey.
• [46:50] The final qsort had ternary partitioning, fast swap, adaptive sample for partition element, insertion sort for small files.
• [47:40] The complete code of C library’s qsort.
• [48:15] Beauties of qsort (beautiful algorithm, beautiful interface, beautiful bug report, beautiful cost models, beautiful tests)
• [50:40] Three beautiful Quicksorts - 1) a simple teaching tool, 2) an anaysis of Quicksort, 3) An industrial-strength Quicksort.
• [51:35] Q and A.

Here is a working C program which runs Quicksort algorithm (in bold) from the Chapter 3 of the book. The algorithm sorts the global array of integers x.

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

#define S(a) (sizeof(a)/sizeof(a[0]))

int x[] = { 56, 10, 39, 15, 20, 62, 14, 39, 77, 17 };

void swap(int i, int j) {
    int t = x[i];
    x[i] = x[j];
    x[j] = t;
}

int randint(int l, int u) {
    return rand()%(u-l+1)+l;
}

void quicksort(int l, int u) {
    int i, m;
    if (l >= u) return;
    swap(l, randint(l, u));
    m = l;
    for (i = l+1; i <= u; i++)
        if (x[i] < x[l])
            swap(++m, i);
    swap(l, m);
    quicksort(l, m-1);
    quicksort(m+1, u);
}

int main() {
    int i;
    srand(time(NULL));
    quicksort(0, S(x)-1);
    for (i = 0; i < S(x); i++) {
        printf("%d ", x[i]);
    }
    printf("\n");
    return 0;
}

In a functional programming language, Quicksort can be written even more elegantly. Check out this Quicksort written in Haskell (thanks to bayareaguy.

  quicksort (x:xs) =
      quicksort [ i | i <- xs, i < x ]
      ++ [x] ++
      quicksort [ i | i <- xs, i >= x ]

  quicksort [] = []

And what’s the most beautiful code you’ve ever written?

Ps. If you got interested, I suggest you get the Beautiful Code book. Jon Bentley carefully explains the ticks used for doing the experimental part of analysis in his chapter.

Hacker News has become the 2nd source for news after programming.reddit for me. I have a couple of ideas for a startup myself and the tips I get from Hacker News are priceless.

If you have never heard of Hacker News, I suggest you visit the site (http://news.ycombinator.com/). It’s a Digg/Reddit like social news site for startup founders and hackers created by Paul Graham.

This program works ideally for me. I spend lots of hours hacking away at my shell and all I have to do is switch the terminals to the one running the Hacker Top program to get the latest stories!

Here is a screenshot of the program running:

Try the 'm' keyboard shortcut to change display modes!

Download

Download link: hacker top program
Downloaded: 537 times

Note - this program is released under GNU GPL.

How to run the program?

1) Make sure you are running a Unix type operating system.

2) Make sure you have Python installed. Any recent version will do.

3) Download and unpack the hacker top program archive.

$ wget 'http://www.catonmat.net/download/hacker-top.tgz'
$ tar -xvzf hacker-top.tgz

4) Change to 'hacker-top' directory which was created by unpacking the archive.

$ cd hacker-top

5) Give the ‘hacker_top.py’ program execute permissions.

$ chmod u+x hacker_top.py

6) Run the hacker_top.py program.

$ ./hacker_top.py

(If that does not work out, try running 'python ./hacker_top.py')

Also make sure your terminal is at least 80 columns wide, otherwise the program won’t be able to display the results nicely.

Command Line Options

If you run the program with '--help' argument, it will display the possible command line options:

Usage: ./hacker_top.py [-h|--help] - displays this
Usage: ./hacker_top.py [-i|--interval interval]
          [-u|--utf8 ] [-n|–new]

As the help message suggests, the three main options are:

• -i or –interval, which specifies refresh interval.
  The default refresh interval is 3 minutes. Here are a few examples: 10s (10 seconds), 12m (12 minutes), 2h (2 hours).
• -u or –utf8, turns on utf8 output mode.
  Default: off. Use this if you know for sure that your terminal supports it, otherwise you might get gibberish.
• -n or –new, which follows only the newest hacker stories.
  Default: follow front page stories.

Keyboard Shortcuts

There are several keyboard shortcuts which you should know about when using the Hacker Top program:

• q - quits the program.
• u - forces an update of the news.
• up arrow/down arrow (or alternatively j/k keys) - scrolls the news list up or down.
• m - changes the display mode. There are 5 different display modes for your taste.

Enjoy the program! I’ll write some details how I created it in one of the next posts. Stay tuned - subscribe to my rss feed!

ps. Be sure to read Hacker News Guidelines and FAQ if you decide to join the hacker community and star submitting interesting hacker news.

It took me 4 years to complete the studies. During these years I took more than 50 courses from physics, mathematics and computer science departments and my average grade was 9.1/10 (students in Latvia are graded from 1 up to 10, 10 being excellent, 9 being very good, …, 4 being the lowest grade for passing a course. Anything below 4 means failure).

As I already mentioned in the about me page, I chose to study physics because I already had a broad understanding of various topics of computer science. Now I can proudly say that I understand a lot of physics as well.

Here is a picture of me with a diploma in my hands.

Here is another picture with all of us who graduated from University of Latvia with a B.Sc. degree in Physics in 2008. I’m right in the middle of the front row!

Talking about my future plans, I decided not to go to a graduate school just yet. I’ll spend the next year or two hacking on whatever interests me the most and learn exactly what I have wanted (more mathematics and more computer science). It also means that I will be able to post more frequently here.

Thanks for reading my blog!

The song is written and performed by the founder of Free Software Foundation and the most active free software advocate Richard Stallman (scroll down for a video of Mr. Stallman performing it himself).

Richard wrote this song at a filksinging session at a science fiction convention. He realized he had never written a filksong relating to free software, so he figured it was time he did. He remembered that he had never written a filksong using Bulgarian dance music, so he figured that would be a good thing to do for once. He chose Sadi Moma because it is not too fast or complicated, and is easy to sing.

Here it is - The Free Software Song!

Download this song: the free software song (musical geek friday #9)
Downloaded: 331 times

Download lyrics: the free software song lyrics (musical geek friday #9)
Downloaded: 82 times

Here is the lyrics of The Free Software Song:

Join us now and share the software;
You’ll be free, hackers, you’ll be free.
Join us now and share the software;
You’ll be free, hackers, you’ll be free.

Hoarders may get piles of money,
That is true, hackers, that is true.
But they cannot help their neighbors;
That’s not good, hackers, that’s not good.

When we have enough free software
At our call, hackers, at our call,
We’ll throw out those dirty licenses
Ever more, hackers, ever more.

Join us now and share the software;
You’ll be free, hackers, you’ll be free.
Join us now and share the software;
You’ll be free, hackers, you’ll be free.

Here is Richard Stallman himself performing The Free Software Song:

The audio quality of this video is poor. Here is a much better version by a band with a hilarious title “The GNU/Stallmans”:

Shmoocon, as they describe themselves, is an annual East coast hacker convention hell-bent on offering three days of an interesting atmosphere for demonstrating technology exploitation, inventive software & hardware solutions, and open discussions of critical infosec issues.

Here are the videos from Shmoocon 2006:

• Behavioral Malware Analysis Using Sandnets by Joe Stewart
• Asterisk: VoIP for the Masses by Damin [presentation]
• Black Ops Of TCP/IP 2005.5 by Dan Kaminsky
• Breaking LanMan Forever by Dan Moniz and Patrick Stach
• Covert Crawling: a wolf among lambs by Acidus
• The Church of Wi-Fi presents: An Evil Bastard, A Rainbow and a Great Dane! by Renderman, Thorn, Dutch, and Joshua Wright [presentation and coWPAtty tool]
• Network Policy Enforcement / Network Quarantine - Latest Security Gimmick or Good Idea? by Steve Manzuik
• Responding to Responsible (or Not) Disclosure by Jon Callas and Special Guests
• ConCon: A History of Hacker Conferences by Jason Scott
• Advanced Network Reconnaissance with Nmap by Fyodor
• FreeBSD jail(8), A Secure Virtual Machine by Ike
• Do it yourself: Building an Enterprise Class Surveillance System by Joel Wilbanks [presentation]
• Advances in Single Packet Authorization by Michael Rash [presentation, and fwknop tool]
• Network Security Monitoring with Sguil by Richard Bejtlich and David Bianco [presentation]
• Countering Attacks at Layer 2 by Eric Smith [presentation]
• “Hacking the Friendly Skies” by Simple Nomad [presentation]
• A Young Gentleman’s Primer on the Reading and Emulation of Magnetic Cards by Abend
• Anonym.OS: Security and Privacy, Everywhere You Go by dr.kaos, digunix, atlas and beth
• Bitchslapping Wireless IDS/IPS appliances by Eldon Sprickerhoff [presentation]
• Web Application Vulnerabilities and Exploits by Matt Fisher [presentation]
• Wi-Fi trickery, or how to secure, break and have fun with Wi-Fi by Laurent Butti and Franck Veysset [presentation]
• Kryptos and the Cyrillic Projector Ciphers by Elonka Dunin
• Reverse Engineering for Fun and BoF It! - 2006 by Pedram Amini, Chris Eagle [presentation]
• Trojans and Botnets and Malware, Oh My! by Lance James [presentation]
• Snort BoF by Cazz
• Windows Vista Heap by Adrian Marinescu
• Outbound Content Compliance by Jim Noble [presentation]
• Lockpicking and Physical Security Fundamentals by Deviant Ollam
• j0hnny’s Greatest Hits: The Best of Johnny Long by Johnny Long
• Securing Civil Liberties - Or Why Hackers Should Run the Government by Jennifer Granick

Here are the videos from Shmoocon 2007:

• Hacking the Airwaves with FPGA’s by H1kari
• Auditing Cached Credentials with Cachedump by Eoin Miller and Adair Collins
• Security Breaches are Good for You by Adam Shostack
• No-Tech Hacking by Johnny Long
• Boomstick-Fu: The Fundamentals of Physical Security at its Most Basic Level by Deviant Ollam, Noid and Thorn
• Hacker Potpourri by Simple Nomad
• Extend your Code into the Real World by Ryan Clarke
• Weaponizing Noam Chomsky, or Hacking with Pattern Languages by Dan Kaminsky
• Bypassing NAC Systems by Ofir Arkin [presentation]
• Web Application Incident Preparation by Matt Fisher, Cygnus and PresMike
• JavaScript Malware for a Grey Goo Tomorrow by Billy Hoffman
• Backbone Fuzzing by Raven
• Windows Mobile Software: Raw and Exposed by Seth Fogie [presentation]
• Hacking Disposable Digital Cameras by John Maushammer
• WPAD: Proxy Attack by Chris Paget
• Vulnerability Disclosure Panel Palaver (or 0-day: OK, No Way, or For Pay) by Katie Moussouris
• A Hacker Looks at 50 by G. Mark Hardy [presentation]
• The Church of WiFi presents: A Hacker in Iraq by Michael Schearer [presentation]
• Wireless (and Wired) Networks @ Security Cons by Luiz Eduardo
• The Hacker Foundation: The Ethic in Action by Jesse Krembs and Nick Farr
• Dissecting ShmooCon Labs by The Shmoo Group
• VOIP, Vonage, and Why I Hate Asterisk by Joel Bruno and Eric Smith
• Attack Detection and Response with Linux Firewalls by Michael Rash [presentation]
• Assess the Security of Your Online Bank (Without Going to Jail) by Chuck Willis
• RFIDiots by Major Malfunction
• Encrypted Protocol Identification via Statistical Analysis by Rob King and Rohlt Dhamankar [presentation]
• Three Crypto Geeks on the Current State of Cryptography and the Internet by Rodney Thayer, Jon Callas and Ben Laurie
• Standard Bodies - What are these Guys Drinking? by Al Potter, Renderman, and Russ Housley
• 0wn the Con by The Shmoo Group

Here are the videos from Shmoocon 2008:

• 21st Century Shellcode for Solaris by Tim Vidas [presentation]
• Advanced Protocol Fuzzying - What We Learned When Bringing Layer2 Logic to SPIKE Land by Enno Rey and Daniel Mende
• Baked not Fried Performing an Unauthorized Phishing Awareness Exercise by Syn Phishus [presentation]
• Forced Internet Condom by Aaron Higbee and Jaime Fuentes [presentation]
• Forensic Image Analysis to Recover Passwords by David Smith [presentation]
• Got Citrix Hack It! by Shanit Gupta [presentation]
• Hacking Windows Vista Security by Dan Griffin [presentation]
• Hacking the Samurai Spirit by Isaac Mathis [presentation]
• How do I Pwn Thee Let Me Count the Ways by RenderMan
• Intercepting Mobile PhoneGSM Traffic by H1kari
• Electronic Voting - Risks and Opportunities by Alex Halberman
• Legal Issues for Bot-net Researchers and Mitigators by Alexander Muentz [presentation]
• Malware Software Armoring Circumvention by Danny Quist [presentation]
• PEAP Pwned Extensible Authentication Protocol by Josh Wright and Brad Antoniewicz [presentation]
• Path X Explosive Security Testing Tools using XPath by Andre Gironda, Marcin Wielgoszewski and Tom Stracener [presentation]
• Practical Hacker Crypto by Simple Nomad
• SIPing Your Network by Radu State, Humberto Abdelnur, and Olivier Festor [presentation]
• TL1 Device Security by Rachel Bicknell
• The Geek and the Gumshoe or Can Mathematics and Computers Really Solve Crimes by Michael Schearer and Frank [presentation]
• They’re Hacking Our Clients! Why are We Focusing Only on the Servers by Jay Beale [presentation]
• Using Aspect Oriented Programming to Prevent Application Attacks by Rohit Sethi and Nish Bhalla [presentation]
• Virtual Worlds by Charlie Miller and Dino Dai Zovi [presentation]
• VoIP Penetration Testing Lessons Learned by John Kindervag and Jason Ostrom [presentation]
• Web Portals Gateway to Information or a Hole in our Perimeter Defenses by Deral Heiland [presentation]
• When Lawyers Attack! Dealing with the New Rules of Electronic Discovery by John Benson, Esq [presentation]
• Why are Databases so Hard to Secure by Sheeri Cabral [presentation]
• On the Social Responsibility of Hackers Panel by Bruce Potter (moderator), Simple Nomad, Johnny Long, Rick Dakan [info on panel]

Enjoy and don’t forget to comment on which videos you liked the best!

Apparently I have missed the first three puzzles (first, second and third) but I’ll give the fourth puzzle a shot.

The fourth problem is about prime numbers and it’s formulated as following:

Find the smallest number that can be expressed as
the sum of 7 consecutive prime numbers,
the sum of 17 consecutive prime numbers,
the sum of 41 consecutive prime numbers,
the sum of 541 consecutive prime numbers,
and is itself a prime number.

For example, 41 is the smallest prime number that can be expressed as
the sum of 3 consecutive primes (11 + 13 + 17 = 41) and
the sum of 6 consecutive primes (2 + 3 + 5 + 7 + 11 + 13 = 41).

The Solution

I’ll write how I got the solution as I go, so I’ll mix the past and present tenses in this post. Sometimes I’ll write what I am going to do and sometimes I’ll write what I just did.

I have no desire to generate lists of prime numbers myself as it has been done infinitely many times already. I’ll just use a publicly available list of prime numbers! Here is a list of first fifty million primes.

I’ll use my Unix-fu to find the solution.

First I noticed that the primes are zipped and split into chunks of million primes per file. The file names are like “primes1.zip”, … “primes50.zip”.

A quick loop from 1 to 50 and wget gets all these files to my hard drive:

$ for i in $(seq 50); do wget "http://primes.utm.edu/lists/small/millions/primes$i.zip"; done

Next, I unzip all these files, and remove those zips to save space:

$ for i in $(seq 50); do unzip "primes$i.zip" && rm -f "primes$i.zip"; done

After doing that and looking at what I got, I realized that they were in some strange format, 8 primes per line, space padded and with some text on the first two lines. Here is an example how the first five lines look in primes1.txt file:

                 The First 1,000,000 Primes (from primes.utm.edu)

         2         3         5         7        11        13        17        19
        23        29        31        37        41        43        47        53
        59        61        67        71        73        79        83        89

I want all my primes to be in one file and one prime per line so I can extract N-th prime by looking at that line.
I used the following command to merge all the files into a single file:

for i in $(seq 50); do (awk 'BEGIN { OFS="\n" } NR > 2 {print $1,$2,$3,$4,$5,$6,$7,$8}' primes$i.txt >> primes.txt) && rm -f primes$i.txt; done

A quick verification that I did not lose any primes:

$ wc -l primes.txt
50000000 primes.txt

Now I’ll create four files which contain sums of 7, 17, 41 and 541 consecutive primes, not exceeding the biggest prime in primes.txt file. I did that with the following AWK one-liner:

$ last=$(tail -1 primes.txt)
$ for N in 7 17 41 541
  do
   awk 'BEGIN { prev[0] = 0 } NR < '$N' {prev[NR] = $1; sum += $1 } NR >= '$N' { psum += prev[NR-'$N']; delete prev[NR-'$N']; prev[NR] = $1; sum += $1; if (sum - psum > '$last') { exit } printf "%d\n", sum - psum }' primes.txt > primes$N.txt
  done

The command created primes7.txt, primes17.txt, primes 41.txt and primes541.txt files. These files contain sums of prime numbers and just some of them are primes.

The solution, if it exists in the given data set, is the intersect of all these files. If there are multiple items in the intersect, the smallest should be chosen and checked if it really was a prime.

$ sort -nm primes541.txt primes41.txt | uniq -d | sort -nm primes17.txt - | uniq -d | sort -nm primes7.txt - | uniq -d
7830239
$ grep -m1 7830239 primes.txt
7830239

We have found the solution! It’s 7830239!

I submitted the answer and after a few minutes it was confirmed to be correct! Awesome!

Your question: [7, 17, 41, 541]
Your answer: 7830239
Time received: 2008-06-06 23:33:26.268414 UTC

Correct answer: 7830239
Your answer was: Correct

Now leave a comment and tell me how you solved this problem!