clipboarded

urban dictionary's greatest hits

2009-12-02T21:22:00.011-06:00

I was curious about what words were the most popular on urbandictionary.com. So I scraped the popular section for each letter in the alphabet and came up with the following table where the ranking is based on the number of upvotes. I didn't want to have my blog associated with any of the words below and I didn't want to give backlinks to the urban dictionary so here's an image of my results.

Here is the code I used to get to the information. It scrapes all the links provided in the popular section for each letter and writes a dump of all entries. The dumps can later be used to study the data.

require 'rubygems'
require 'hpricot'
require 'open-uri'
require 'entry'
require 'net/http'
require 'json'

def retrieve_votes(doc)
  # in-browser, votes are retrieved through an ajax call after the web page is loaded
  (doc/"td.tools").each do |tools|
    id = tools[:id]
    uncacheable_id = id.scan(/\d+/)[0]

    json_response = Net::HTTP.post_form(URI.parse('http://www.urbandictionary.com/uncacheable.php'), {'ids'=> uncacheable_id})
    thumbs = JSON.parse(json_response.body)['thumbs'][0]
    return [thumbs['thumbs_up'], thumbs['thumbs_down']]
  end
end

def retrieve_links letter
  anchors = []

  doc = Hpricot(open(letter))
  (doc/"table#columnist//tr").each do |row|
    (row/"td").each do |cell|
      (cell/"ul"/"li").each do |li|
        (li/"a").each do |anchor|
          anchors <<  anchor.get_attribute(:href)
        end
      end
    end
  end

  anchors
end

def build_entry(doc)
    word = "no words found"
    definition = "no definitions found"

    up, down = retrieve_votes(doc)
    (doc/"td.word").each do |wrd|
      word = wrd.to_plain_text
      break
    end
    (doc/"div.definition").each do |defined|
      definition = defined.to_plain_text
      break
    end

    Entry.new(word, definition, up, down)
end

"ABCDEFGHIJKLMNOPQRSTUVWXYZ".each_char do |letter|
  links = retrieve_links("http://www.urbandictionary.com/popular.php?character=#{letter}")
  entries = []
  links.each do |link|
    sleep 5
    puts "fetching #{link}"
    doc = Hpricot(open("http://www.urbandictionary.com" + link))
    entries << build_entry(doc)
  end

  File.open("letter#{letter}.dump",'w'){|file|
    file.write(Marshal.dump(entries))
  }
end

The Entry class is defined like so.

class Entry
  attr_accessor :word, :definition, :up, :down
  def initialize word, definition, up, down
    @word = word
    @definition = definition
    @up = up
    @down = down
  end

  def <=>(other)
    (other.up - other.down) - (@up - @down)
  end

  def to_s
    "#{word} (#{up} up, #{down} down): #{definition}"
  end
end

In order to get some kind of greatest hits, I used the following script. I had to filter out entries with too many down votes to get rid of the most childish entries.

require 'entry'

entries = []
Dir['letter*.dump'].sort.each do |filename|
  entries.concat(Marshal.load(File::open(filename).read))
end

entries.sort!
entries.reject! {|entry| entry.down > 1000}
html="<table>"
50.times do |i|
  i = i + 1
  html += "<tr><td><b>#{i}</b></td><td style='padding: 0 2em 0 2em;'>#{entries[i].word}</td><td><a href=\"http://www.urbandictionary.com/define.php?term=#{entries[i].word}\">read more</a></td></tr>"
end
html+="</table>"

puts html

pair programming prep

2009-11-29T23:08:00.001-06:00

What can we do before pairing to make it more effective?

Make sure both peers focus on productivity to get the ego and flowery phrases out of the way
Agree on a game plan. What problem are we solving? What type of pairing is this going to be? (ball and board or coder/navigator or ping pong)
Understand the benefits/costs of pairing
Negotiation: Roy Miller of RoleModel Software offers a suggestion:
One of the best tricks I've ever used when pairing is to start each pairing session with some 'negotiation.' I lay out what I like and don't like when I pair with someone, things I do well and things I don't do well, things I need help on, things that bug me to death. Then I listen while the other person does the same. This takes less than five minutes, usually, but the results are staggering. Both members of the pair then can use the pairing session as a time to get work done and to help the other person. I think 'pairing as negotiation' is a key concept.
Read more: Pair Programming Illuminated

change the web challenge

2009-02-28T10:02:00.000-06:00

The not-for-profit Social Actions just launched the Change the Web Challenge. The idea is to come up with the greatest application you can think of that leverages their API.

Their API allows you to search a database of social actions -- things like volunteering opportunities, fundraising events, petitions -- aggregated from contributors like kiva.org or idealist.org.

For a quick peek at what you have access to, you can try my very own search for opportunities. The code for it is available as a git repository.

test-first VS writing a test harness

2008-11-30T12:20:00.000-06:00

There is a common understanding that code written using a test-first approach (as in TDD-style) ends up being more maintainable than code that has been harnessed with tests later on. I would justify that understanding as follows:

TDD simplifies original design. This one is easy. By driving the design through small unit tests, we never write code that is unnecessary and experienced TDDers know to make the tests pass with the simplest thing that works. Further, because we gain immediately from writing testable code, the code usually ends up loosely coupled with no implicit dependencies and few explicit dependencies.
Another thing is that by writing the code that exercises the implementation, we design both sides of the API, client code and implementation at the same time. I blogged about client code as design some time ago.

The tests once failed for lack of functionality. To make sure the test is bug-free -- and actually testing the upcoming functionality -- it is imperative to make it fail before making it pass. If you're testing first, that's easy. If you're writing a test harness, you might have to comment out a bit of code to get the red bar, but then will it still compile? Will it still be coherent at runtime? A test harness is forever dubious.

TDD gives you fine-grained coverage. With very little overhead, TDD gives you fine-grained coverage since you're writing the simplest test that brings you forward, all the time. In the words of Gerard Meszaros, this gives you better Defect Localization. When writing a test harness, we usually emphasize the higher level functionality and forget about the smaller 'units' of work. These smaller units might be hard to test because they are implicit, hidden/encapsulated, or even unfathomable.

Test/Specification mismatch. The tests are the requirements for the code. The BDD people have taken this to the next level, but it remains true with TDD. Test code is the most useful documentation to a fellow developer: it's working, up to date, terse, and it serves as an example of client code. When developing TDD-style, the tests become the developers' understanding of the stories -- the ultimate reference.
If writing tests later on, we need to match the functional requirements with the actual codebase, like pieces of a puzzle. Except there will be mismatches, where the developers were not really clear about the requirements. What are we to do then? For this reason alone, test harnessing a big codebase is just really, really hard.

Interestingly, most of the benefits derived from test-first development are gone even if you write the tests right after new code.

about free culture

2008-10-26T12:53:00.000-05:00

I just finished reading The Future Of Ideas and Free Culture. I read both books avidly and I would recommend them to pretty much anyone -- programmer or not. What's more, they are both available for free online, under a creative commons license.

In both books, Lessig dissects the state and the fate of the commons in american society. The commons are to be cherished and cared for as they lead to innovation, creativity, and a free culture.

The Future Of Ideas is mostly concerned about explaining the concept of commons and the one of layers. The idea of layers is taken from Yochai Benkler and divides the network in three parts: the physical layer, the code, and the content. Lessig then explains how the balance of control at each layer affects innovation and creativity. The example stories include Napster, MP3.com, UNIX, and deCSS.

Free Culture is mostly concerned about the effects of excessive control on culture. Lessig discusses copyright law, intellectual property, and the concentration of the media. The example stories include the infamous RIAA lawsuits and the Eldred v. Ashcroft case.

A few things strike me about this free culture debate:

There is no clear stakeholder in the 'new'. The new innovators and creators that free culture helps aren't yet established. This makes the fight incredibly hard to win. It's another case of special interests violating the people.
There is no obvious way to disagree with Lessig unless you have a stake in the 'old'. Lessig explains his case rationally and is smart enough to include the whole political spectrum with his arguments.
Prolonging the term of existing copyrights is absurd. The objective of copyrights is to give an incentive to innovators. Once the innovation has seen the light of day, prolonging the term of copyrights delays their introduction in the public domain where they could give rise to even more innovation. Thus, prolonging the term of copyrights goes against their prime objective.

As a side note, another book from Lessig just came out: Remix.

skip dependencies in Rake

2008-09-15T11:58:00.002-05:00

Here is a handy rake task to skip dependencies whenever you want to:

desc 'Skip dependencies of subsequent tasks'
task :skip do
  Rake::Task.tasks.each do |t|
    t.clear_prerequisites
  end
end

pyweek 7 is over

2008-09-14T13:13:00.005-05:00

PyWeek just went by. A python programming challenge where participants have one week to make a game.

Here is our project page: google code strlen

Or you can check it out like so:

svn checkout http://strlen.googlecode.com/svn/trunk/ strlen

being more than programmers

2008-08-23T11:08:00.005-05:00

There is a recurrent theme that I see in blogs and books where it is stated that programmers have to open up, broaden their horizons and become more interested in the business perspective of software. Kent Beck dedicating Extreme Programming Explained goes: "Without you, this book would still be about programmers hiding in a corner. Without you, I would still be one of those programmers."

Dave Thomas, with usually very utilitarian views on our careers as software developers, also consistently encourages his audience to take into account the big picture and invest in our personal education and knowledge.

I'm sure this trend is nothing new but it's been very visible recently in the agile movement in general. A recent book along this trend is Chad Fowler's My Job Went To India. Its second chapter bears the title "Coding Don't Cut It Anymore". This idea of opening up is a good one and is usually well received. However, I feel that the direction in which we need to open up is often implicit in most texts. This is unfortunate as I believe this direction should be the result of a conscious decision.

There is no question that we need to be more than programmers. The mere will to write useful software needs us to be more than coders. However, I need more than the proposed, and dare I say obvious, solution. I would paraphrase this solution like so: we need to build a bigger role for ourselves by learning a few things from the way our organizations work and pay more attention to the customer's needs and the business value of our product: software.

Chad Fowler wants us to ask ourselves these questions:

"How much value did you produce last year? What was your positive impact on the company's bottom line? What did you give back? How much money did you cause the company to save? How much more in revenues did you contribute?"

or to follow the wave in search of the next big thing:

"Ten years ago, you would have been surprised to find out just how big object-oriented languages with garbage collection would become. But, there were definitely signs if you were watching. Ten years from now, who knows what the Next Big Thing will be? You've got to keep your eyes and ears open."

The shortcoming of these propositions is that their benefits are not distributed equally. Really, those solutions turn out to be very good for your company and your personal gain is only marginal. How to align the employees' incentives so that their actions benefit the company is a special case of the principal-agent problem. This is the one big problem we should be working on inside our organizations. Here is the spoiler though: this problem is only truly solved if you start your own company.

Short of doing that, there are lots of ways to pick up the slack and "open up" outside of our organizations. Open-source projects are an easy solution, you're not an "agent" when you labor out of love. This point is actually discussed in Chad's book so I will skip it. Community-oriented services are another solution: projects like the OLPC are using technology to respond to social problems and engaging programmers in their communities. Other than that, lots of issues are being publicly debated and it is our duty to understand them, if not take part in them. The copyright laws are completely out of touch with our everyday behaviors, how can we convince the politicians to work with us? The creative commons and copyleft licenses are trying to address problems with the usual licenses, how are they succeeding? How are they failing? Network neutrality boosts each citizen's right to unbiased and unrestricted information, that's a good thing but how do we do that without stifling investment and availability? Kiva.com offers microfinanced loans to people in need, how else can we harness massive audiences for the good of the poorest?

Another way to open things up and become more well-rounded is to mix disciplines. Common sense and general resourcefulness dramatically increases with the number of fields we can master or even just be interested in. This approach has known a lot of successes and will know many more. For instance, applying concepts from biology to programming problems led to genetic algorithms, ant colony optimization, agent-based modeling, etc. What else can we mix and match to increase the value of the whole?

If we do value individuals over processes and collaboration over contracts, I believe we should pay more attention to how software affects our communities. I believe we should pay more attention to how we can transform our organizations to the employee's benefits, not the other way around. Over the span of our careers, we will have many jobs and many employers. I'm only 25 and I've had 5 employers -- double that number if we are counting jobs outside the IT industry. This kind of track record is now the norm. As software experts and citizens, the only innate responsibility we have is to people and software.

for frequent commits

2008-08-18T19:25:00.009-05:00

When I code, I always commit as often as possible. I try to bring myself to check code in at least every hour. Here is the case for frequent commits.

The psychological effects on me are just amazing. The rhythmic process becomes a dragonboat's drum that pushes me forward. If I fail to commit after an hour, I feel like I dropped the ball and end up working harder. The following points are more technical, but this one alone is enough to justify frequent commits.

Breaking down the tasks into small chunks becomes essential to make the next commit on time. When a task is hard to explode, I get creative and check in tiny bits of functionality at a time. Write a small test, make it pass, check it in. Refactor, check it in. It helps me to always stay in control, I can justify every step I take toward the goal.

Fewer integration conflicts result since less code is checked in at a time. This one is well understood. Conflicts are bad.

Flexibility. It makes it easier if I want to pair, go solo, or switch pair. I can "change state" every hour if my heart so desires. Also, if someone needs a piece of code from the story I'm working on, it might already be checked in. Sharing and integration cannot happen soon enough.

Thinner diffs between commits makes the problem of finding where a bug appeared/re-appeared easier. Once I found the build that "broke the feature", it's a lot easier to dig through a small change and pinpoint the dubious line. As a bonus, the log message for each check-in usually ends up very precise.

Backups. Every time I commit, I am also making a backup of my code. Losing my local work is never a problem.

Visibility. The commit log becomes a pulse on the state of each pair or each developer. It becomes another tool for the team or the manager to assess progress. Frequent commits become a sign of good health. How hard the current task is can now be estimated by the interval between commits. A slowdown in the commit rate might raise some questions during the stand-up meeting or the retrospective...

Frequent commits are a good thing.

smart shuffle

2008-08-02T17:56:00.008-05:00

Here's a feature I've always wanted when using an audio player: shuffle the playlist so that all the songs from the same artist are as spread out as possible. You can try it out with the amarok player (apt-get users can click this link), after you've installed this script. Just right-click the playlist and choose Smart Shuffle -> Shuffle.

Here's the Ruby code that does the job, as long as the parameter to #smart_shuffle is an array containing elements that respond to #artist.

require 'utils'

def smart_shuffle playlist
  playlist.shuffle!
  groups = playlist.group_by {|item| item.artist}

  groups.inject([]) {|decorated_playlist, group|
    decorated_playlist + group.decorate {|item, index|
      every_n = playlist.length.to_f / group.length.to_f
      rank = every_n * (index + 1) - (every_n / 2.to_f)
    }
  }.sort_by_decoration.undecorate
end

Here's the file it depends on, utils.rb:

class Array
  def shuffle
    sort_by { rand }
  end

  def shuffle!
    self.replace shuffle
  end
end

module Enumerable
  def group_by &block
    groups = Hash.new(){|hash, key| hash[key] = []}
    each {|item|
      retval = block.call(item)
      retval = "Unknown" if retval == "" || retval.nil?
      groups[retval.to_sym] << item
    }
    groups.values
  end

  def decorate &block
    decorated = []
    each_with_index {|item, index|
      decorated << [block.call(item, index), item]
    }
    decorated
  end

  def sort_by_decoration
    sort {|this, that|
      this[0] <=> that[0]
    }
  end

  def undecorate
    map {|item| item[1]}
  end
end

I'm using a simple DSU algorithm where the decoration is given in those two lines:

every_n = playlist.length.to_f / group.length.to_f
rank = every_n * (index + 1) - (every_n / 2.to_f)

The variable every_n is the interval between two songs by the same artist -- a song from this artist should come back every n songs. I calculate the rank based on that, adding one to the index to prevent useless cancellation when it is zero. I subtracted (every_n / 2.to_f ) simply because it was giving me better results.

command line quizzes part 3

2008-07-24T22:59:00.016-05:00

Q: Given that svn log -q | grep '^r' gives you a list of all commits and their owners, one per line. Figure out the percentage of commits done by a particular user, since that user's first commit.

As an example, we can use the following list of 100 changes to the trunk of the subversion project:

We want to know the percentage of commits done by the user 'julianfoad' (randomly chosen) since his first commit -- 10th line from the bottom.

The following lines do the trick:


svn log -q --limit 100 | grep '^r' | cat -n | sort -nr | sed -ne '/julianfoad/,$ p' | sort -n | wc -l
svn log -q --limit 100 | grep '^r' | cat -n | sort -nr | sed -ne '/julianfoad/,$ p' | sort -n | grep julianfoad | wc -l

The first line gives the overall number of commits since julianfoad's first: 91.

The second line gives his personal number of commits: 15.

So the percentage is echo '15/91 * 100' | bc -q. In english: 16.48%.

the ideal client code as design

2008-07-14T11:12:00.024-05:00

It is still an open question how to approach most programming problems: what to do first? how much design is enough? Some methodologies are trying to answer these questions. TDD comes to mind, or bottom-up programming, or domain-driven design. What I've been trying to discipline myself to do these last few months is to ask myself: what would the perfect interface be?

I don't code, I don't test. I just ponder about the best possible interface for this new feature. What would ideal code using this feature look like? I'm thinking about the method names and what is the least amount of information that I must give in order to implement the desired behavior. For example, I want to create a window with a given title. Then I want to add a button to it that quits the application, but only quits if the current document does not have outstanding changes.

I think the following code would look great:

create_window("title") and add_button do quit if unchanged end

It doesn't matter what functions or classes already exist or how the hell am I going to get a handle to a thousand objects. I try to build the ideal client code. I am using ruby syntax at the moment but I could also use pseudocode -- it might even be more educational using pseudocode.

As another example, I want to create an email from a string and send it to Bob at 3pm:

at 3.pm do create_email("body").send_to(:bob) end

The underlying assumption is that if the client code makes sense to the domain expert, the domain code will have to bend over backwards to represent the domain better. For instance we could also fancy sending an email like this:

send_email("body").to(:bob) | 3.pm

The two forms look nice to me but if the application represents a clock triggering actions at specific times, I would choose the first one. If the application feels more like an email client, I might choose the second form. The point is that I am free to think about what makes sense in the domain, free from the current codebase, free from scope rules or whatever stupid design decision I made yesterday.

The way to go from "ideal code" to the actual implementation is usually easy. I like to write the ideal code in a test file, make it compile by whatever means, and increment slowly to hook the code up to the rest of the application. Sometimes a revelation will come and prevent the ideal code from happening and that's fine, although too many of these might be a smell.

This "client code as design" is of course nothing new. It's in the same cluster as DSLs, and UCD.

postfix / procmail / courier-imap for dummies

2008-04-26T06:24:00.004-05:00

To install a mail server on a unix machine, you can use the following stack:

postfix : listens for incoming emails and delegates to...

procmail : delivers emails to each user's mailbox

Courier-IMAP : listens for pop and imap requests and serves emails

This is just a high-level installation procedure for a mail server where the users are simply the local users.

Install postfix and make sure it delegates to procmail by adding the following line to the appropriate section in postfix's main.cf:

mailbox_command = /usr/bin/procmail -a "$EXTENSION" DEFAULT=$HOME/Maildir/

After you make any changes to main.cf, you should run:

postfix reload

Make sure /etc/procmailrc or $HOME/.procmailrc contains the following lines:

MAILDIR=$HOME/Maildir
VERBOSE=on
LOGFILE=$MAILDIR/from

:0
new

Next, build the maildir by issuing

maildirmake ~/Maildir

At this point, we should have a working system, that we can test with something like:

mail -s hello $USER
Hi,
This is a test
^D

If everything works fine, the email should be received by postfix, sent to procmail, and delivered to $HOME/Maildir/new.

To access the emails through POP or IMAP, we need to install Courier-IMAP. Once this is done, start the authdaemond and the courier daemon. Finally, netstat -a should reveal everyone listening: one smtp connection, one POP connection and one IMAP connection.

tcp 0 0 *:smtp *:* LISTEN
tcp 0 0 *:imaps *:* LISTEN
tcp 0 0 *:pop3s *:* LISTEN
tcp 0 0 *:pop3 *:* LISTEN
tcp 0 0 *:imap *:* LISTEN
tcp 0 0 localhost.localdom:imap localhost.localdo:34187 ESTABLISHED

You can fire up thunderbird and create an account with the local hostname as the POP or IMAP server. You should then be able to send yourself emails.

Once this works, you can use and test the fetcher plugin for Ruby on Rails (more info here).

hugamovie.com :: hugacity.com :: hugamusic.com

2008-03-10T05:52:00.006-05:00

My brother and I just deployed a couple of web apps that we built together.

hugamovie.com

hugacity.com

hugamusic.com

huganoodle.com

They all revolve around the same concept: random items are displayed and the user rates them from 1 to 5. We call it hugging or ditching. Once the users get bored, they can look at the top 10 or the other huga*.com sites.

Hugacity.com makes use of pictures licensed under the creative commons. The other apps pulled their initial data from Amazon.

All the sites share the same core of code and were built on top of Rails 2.

handy 'and' and 'or' in Ruby

2008-02-23T02:22:00.004-06:00

Here's a nice shortcut to verify if any member of an array satisfies a predicate, or to verify if all members of an array satisfy the predicate.


#!/usr/bin/env ruby -w

module Enumerable
    def any sym
        each {|item| return true if item.send sym }
        return false
    end

    def all sym
        each {|item| return false if !item.send sym }
        return true
    end
end

puts [1, 2, 3, 0, nil].any(:nil?)
puts [1, 2, 3, 0].any(:nil?)
puts ["", "", ""].all(:empty?)
puts ["", "", "dsa"].all(:empty?)

Output:

true
false
true
false

the craft of functional programming :: review

2008-02-21T20:59:00.000-06:00

I went through most of The Craft of Functional Programming by Simon Thompson. All in all it was a great learning experience.

The book is aimed at beginners and the text is very dense, which saves a lot of time. The exercises were very useful to me and sometimes even enlightening. The one thing I found annoying was the lack of transparency in the sample code snippets: the code given in the book sometimes needs an import from the standard library to function, or it shadows a name from the standard library. All the problems can be solved by referencing the source files or hoogle but it's not as straightforward as it could be.

The book covers the basics fairly well but is very scarce on program complexity analysis, the type system, and monads.

If I had to recommend a "track" for learning haskell, it would go like this:

Go through the haskell wikibook

Read The Craft of Functional Programming up to chapter 18

Read IO inside on the haskell wiki

Read the rest of the book

Find other more advanced resources...

All the while, there are exercises in the wikibook, in the book, on the haskell wiki at 99 problems, and at project euler.

news.yc statistics

2008-02-17T20:22:00.018-06:00

I was thinking idle thoughts about the community at news.yc and I decided to pull out a few statistics. I found the following page with some stats on it but I wanted more.

The following was deduced from a sample of about 130 users, including the 50 leaders. This is a small sample and I can only access 180 posts per user, so this is not perfect, but still interesting. Note: for a few of those statistics, I only considered users with more than 8 posts.

Total number of posts: 8317

Total number of points on those posts: 42458

Minimum points per post ratio: 0.72

Maximum points per post ratio: 20.2

Mean points per post ratio: 5.75

Std deviation on the ratio: 3.22

Overall points per post ratio: 5.10

Mean number of posts: 62.00

Std deviation on the number of posts: 68.36

When a user has a high points per post ratio, he is probably representative of the community. All kinds of guidelines could be derived from this: if a user's points per post ratio is too low below the average, they probably are posting too much, or not a strong link in the community as a submitter. If the standard deviation on the mean ratio is low, then the community is rather cohesive: lots of people interact without leading or counting on others. If the standard deviation is high, than the users are possibly separated in two groups, one reading the posts of the other. These kinds of statistics might be useful as an early index of community 'dilution'.

Here's a graph showing the points per post against the number of posts.

We can see that it's very hard to maintain a good points per post ratio when the number of posts reaches about 40. Also, if a newcomer wanted to know what kind of users are on news YC, we should point them to paul, palish, sharpshoot, or pg.

Here's the same graph but only showing the leaders.

Zoom on the users with 180 posts:

Zoom on the bottom left cluster for all users:

Read the comments on this at news YC.

Here is the ruby code that I used. You can tweak it to find out your own points per post ratio.

go.rb


#!/usr/bin/env ruby
require 'open-uri'

HOSTNAME = "http://news.ycombinator.com/"
LOG = "... "

def compute(user, url, total_points, total_posts)
    more_url = nil
    open(HOSTNAME + url) {|doc|
        doc.read.each {|line|
            more_url = find_more_url line if more_url.nil?
            line.scan(/(\d*) point[s]{0,1}<\/span>/) do |match|
                #puts "analysing match: #{match}"
                total_points += (match[0].to_i - 1) #posts start at one point
                total_posts += 1
            end
        }
    }
    begin
        sleep(5)
        puts LOG + "page done"
        $stdout.flush
        #if there is a more button, follow it and recurse
        total_points, total_posts = compute(user, more_url, total_points, total_posts) unless more_url.nil?
    rescue
        puts "caught #{$!}"
        puts "stopped after #{total_posts} posts for user #{user}"
    end
    return [total_points, total_posts]
end

def go users
    users.each do |user|
        puts LOG + "processing #{user}"
        $stdout.flush
        begin
            total_points, total_posts = compute(user, "submitted?id=#{user}", 0, 0)
            ratio = total_points.to_f / total_posts
            puts "#{user} #{total_points} #{total_posts} #{sprintf("%.2f", ratio)}"
        rescue
            puts "caught #{$!}"
            puts "user #{user} not processed"
        end
    end
end

def find_more_url line
    if line =~ /<a href="([^"]*?)" rel="nofollow">More<\/a>/
        $1
    else
        nil
    end
end

def file2list file
    users = []
    File.open(file) {|f|
        f.each {|line|
            users.push line.chomp unless line.nil?
        }
    }
    users
end

def extract_names users, url=""
    more_url = nil
    open(HOSTNAME + url) {|doc|
        puts 'transferring data'
        doc.readlines.each {|line|
            more_url = find_more_url line if more_url.nil?
            line.scan(/<a href="user\?id=([^"]*)">/) do |match|
                users.push match[0] unless users.include? match[0]
            end
        }
    }

    users = extract_names(users, more_url) unless more_url.nil?
    return users
end

#extract_names([]).each {|u| puts u}
go(file2list(ARGV[0]))

extract.rb


#[0] = name
#[1] = points
#[2] = posts
#[3] = ratio

def std_deviation data, mean, index
    sq_errors = 0.0
    data.each {|row|
        sq_errors += (row[index] - mean)**2
    }
    puts "Std deviation: #{Math::sqrt(sq_errors/data.length)}"
end

def num_posts_metrics data
    min = 99999
    max = 0
    total_posts = 0
    data.each {|row|
        total_posts += row[2]
        min = row[2] if row[2] < min
        max = row[2] if row[2] > max
    }
    puts "Total posts: #{total_posts}"
    puts "Min # posts: #{min}"
    puts "Max # posts: #{max}"
    mean_num_posts = total_posts/data.length
    puts "Mean # posts: #{sprintf("%.2f", mean_num_posts)}"
    std_deviation(data, mean_num_posts, 2)

end

def ratio_metrics data
    min = 9999
    max = 0
    total_posts = 0
    total_points = 0
    sum_ratio = 0.0
    data.each {|row|
        total_posts += row[2]
        total_points += row[1]
        min = row[3] if row[3] < min
        max = row[3] if row[3] > max
        sum_ratio += row[3]
    }
    puts "Total posts: #{total_posts}"
    puts "Total points: #{total_points}"
    puts "Min ratio: #{min}"
    puts "Max ratio: #{max}"
    mean_ratio = sum_ratio/data.length
    puts "Mean ratio: #{sprintf("%.2f", mean_ratio)}"
    overall_ratio = total_points.to_f / total_posts
    puts "Overall ratio: #{sprintf("%.2f", overall_ratio)}"

    std_deviation(data, mean_ratio, 3)
end

#munge munge
data = []
File.open(ARGV[0]){|file|
    file.each do |line|
        data << line.chomp.split
    end
}

data = data.map {|row|
    [row[0], row[1].to_i, row[2].to_i, row[3].to_f]
}

#output
ratio_metrics(data.select {|row|
    row[2] > 8
})
puts
num_posts_metrics data

command line quizzes part 2

2008-02-14T23:23:00.010-06:00

Here's part two of the command line quizzes. Part one can be found here.

Q: Given that "top -b -n 1" gives the following output, write a script which kills all processes that use more than 25% of the processing power.

Tasks: 85 total, 1 running, 84 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.7%us, 0.6%sy, 0.0%ni, 93.7%id, 0.8%wa, 0.2%hi, 0.1%si, 0.0%st
Mem: 506372k total, 495416k used, 10956k free, 12364k buffers
Swap: 0k total, 0k used, 0k free, 195588k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6159 root 19 -1 357m 38m 5380 S 2.0 7.9 7:02.21 X
6271 mats 20 0 123m 38m 25m S 2.0 7.8 1:33.09 amarokapp
6287 mats 20 0 25840 9916 7324 S 2.0 2.0 0:29.45 gkrellm
1 root 20 0 1612 544 472 S 0.0 0.1 0:00.84 init
2 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0.0 0.0 0:02.08 ksoftirqd/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
6 root 15 -5 0 0 0 S 0.0 0.0 0:00.16 events/0
7 root 15 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
42 root 15 -5 0 0 0 S 0.0 0.0 0:00.11 kblockd/0

Here's a solution.


#!/bin/bash
#script to kill all processes which use more than 25% of the cpu
top -b -n 1 | awk 'NR >= 8 && $9 > 25 {print $1}' | xargs -r kill

Thanks to Ralph for this solution.

Q: Given the following two CSS files, write a script to verify that every class is only defined once (catch duplicate declarations).

File 1: first.css


body {font-family:Verdana;}
.header {font-size:90%;  }
.article {line-height:1.1em;}
.footer {font-size:80%;  }
.main .left #contents {float:left;}

File 2: second.css


#article .title {font-size:110%;}
.header {font-weight:bold;}
.email {font-family:Courier;}
.address {line-height:0.8em;}
.left {text-align:left;}

Here's a solution.


#!/bin/bash
#script to find out if two css files contain the same class name

#Use sed to filter out the css rules and put all the ids and classes on their own line
#Use sed again to only keep the class names
#Use sort and uniq to figure out the dups
classnames=$(sed -e 's/\([^{]*\).*/\1/g' -e 's/ /\n/g' $@ | sed -n -e '/\.\w\+/ p' | sort | uniq -dc | sort -r)
echo -e "\nDuplicate   classes:\n"
echo "$classnames"
echo -e "\nGrepped     files:\n"
#grep the files in argument for the class names found above
grep --color -n "$(echo "$classnames" | awk '{print $2}' | sed -e 's/\./\\./g' -e 's/\n/\|/g')" $@

It can be called like so.

./dupcss.sh first.css second.css

And gives the following output.

Duplicate classes:

2 .left
2 .header

Grepped files:

first.css:2:.header {font-size:90%;}
first.css:5:.main .left #contents {float:left;}
second.css:2:.header {font-weight:bold;}
second.css:5:.left {text-align:left;}

Grepping the files and displaying the results is just a convenience to the user.

I don't have any other quizzes on the back burner but I do have a few good links:

Getting started with awk

Sed - an intro and tutorial

bash how-to

vim cheat sheet

cheat-sheets.org

command line quizzes part 1

2008-02-13T00:51:00.016-06:00

When I wanted to learn about bash and the unix command line tools, I looked for some easy quizzes and challenges but could not find any. Here's two for a start. (edit: part two can be found here)

Q: Find the most common 4-letter word in the following text.

Ruby is a pink to blood red gemstone, a variety of the mineral corundum (aluminium oxide). The common red color is caused mainly by the element        chromium. Its name comes from ruber, Latin for red. Other varieties of gem-quality corundum are called sapphires. It is considered one of the four     precious stones, together with the sapphire, the emerald and the diamond. Improvements used include color alteration, improving transparency by        dissolving rutile inclusions, healing of fractures (cracks) or even completely filling them.

Prices of rubies are primarily determined by color (the brightest and best "red" called Pigeon Blood Red, command a huge premium over other rubies of  similar quality). After color follows clarity: similar to diamonds, a clear stone will command a premium, but a ruby without any needle-like rutile    inclusions will indicate the stone has been treated one way or another. Cut and carat (size) will also determine the price.

Rubies have a hardness of 9.0 on the Mohs scale of mineral hardness. Among the natural gems only diamond is harder, with a Mohs 10.0 by definition.

Here's a solution.


#!/bin/bash
#a script to find the most common 4-letter word in a text file
sed 's/ /\n/g' text.txt | grep -E '^\w{4}$' | sort | uniq -c | sort -r | head -1

Giving the following output. The most common 4-letter word is 'will' with 3 occurrences.

3 will

Q: Write a 10x10 matrix with random integers from 0 to 9 to standard output.

Here's a solution.


#/bin/bash
#a script to generate a 10x10 matrix with random numbers from 0 to 9
for i in `seq 1 10`; do
 for i in `seq 1 10`; do expr $RANDOM % 10; done | xargs
done

Giving the following output.

3 1 0 3 8 5 7 7 0 8
4 5 1 8 8 0 8 7 5 4
6 1 1 3 6 4 8 8 0 9
6 6 1 8 6 9 9 6 3 4
5 1 7 2 1 4 5 4 8 5
7 5 8 7 9 9 5 4 2 1
7 0 8 2 9 4 6 2 7 8
8 6 8 6 9 9 6 3 2 3
8 1 9 3 9 6 7 3 5 8
8 5 9 8 9 0 7 1 0 3

project euler in Haskell

2008-02-09T22:27:00.001-06:00

I was looking for a way to practice my lowly haskell skills and I found Project Euler. All my solutions can be found at Google Code: myhseuler. I solved a bit more than 30 problems as of this writing.

You can check it out like so:

svn checkout http://myhseuler.googlecode.com/svn/trunk/ myhseuler-read-only

function composition in Ruby

2008-02-03T18:40:00.000-06:00

Here is an easy way to compose functions:


#!/usr/bin/env ruby

# method composition example
# > require 'compose'
# > (method(:multwo) << method(:multwo) << method(:add)).call(2,3)
# => 20

class Proc
    def << g
      lambda {|*args| self.call(g.call(*args)) }
    end
end

class Method
    def << g
      lambda {|*args| self.call(g.call(*args)) }
    end

end

def add left, right
    left + right
end

def multwo arg
    arg * 2
end

olpc: XO-1

2007-12-02T02:48:00.000-06:00

I just tried out the XO-1 system on my laptop using the qemu emulator and I am impressed. It took me about 20 minutes to get it working.

You need the development image and qemu with kqemu.

If you want you can pinch those scripts and you should be good to go.

start_olpc.sh

#!/bin/bash
su -c ./su_ops.sh && qemu -m 256 -kernel-kqemu -soundhw es1370 -net user -net nic,model=rtl8139 -redir tcp:2222::22 -hda olpc-redhat-stream-development-devel_ext3.img

su_ops.sh

#!/bin/bash
rmmod rtc_cmos
modprobe rtc

#optimiser
modprobe kqemu

#virtual memory
umount /dev/shm
mount -t tmpfs -o size=272m none /dev/shm

echo 1024 > /proc/sys/dev/rtc/max-user-freq

small windows

2007-11-29T06:29:00.001-06:00

There's something about a computer hooked to the internet: it's really all about small windows. We open up a program, we get a new window. Change the settings, there's a new window. Check out a website, new window. We move them around, we swap, we toggle, looking for something real, something that will push our buttons.

We say we are multitasking but that's a joke. Every window is there to hide everything else, and if we could see everything at once, we would never be able to work. And I'm not just turning the shirt backwards, we really need partitions, we need filters, we need structures to inject some sense in the randomness, to connect the dots. It's all about small windows.

We share, we comment, we blog, we email, we receive news feeds, we monitor stories, we screen others. It's a wire tap on the world. It's big, it's fresh, it's empowering, but how much do we really share at one time? When we email or comment on a story, 5 or 6 lines is too long: no one has time to read it. We need to minimize the input and maximize the effect. So what do we do? What do we do? Whatever the answer to that is, there is a common thread: we want to make an impact, a difference. We need feedback. We are feedback whores. So what do we do? As the window shrinks, we have less time to impress others and we get violent. We get mean. Violence is a way to control. We need it, because it will lead us to feedback.

When that happens we need to compress our thoughts. It's all about small windows. When we communicate, it's decision time, decisions decisions decisions. And nobody likes to be irrelevant. There is only so much time but we still aim to have an impact.

And it's getting harder and harder. More people get online everyday and the more people there are the less time we have for one single person. That's led to people saying they have a short attention span and that keeps them from achieving more. I think that's completely off. Small windows make us smarter. It's the rarity problem: by having less time to dedicate, we are forced to choose what we will spend our time on next. A short attention span is an adaptation, embrace it.

Look around, explore, surf, discover the world one window at a time. Whatever we are looking for, it's out there. Let's try something new, we can afford to crash, the next window is just around the corner.

Forget about being too small a player and quality versus quantity, nothing is binary: quality will come straight from quantity. There cannot be too many people and there cannot be too many windows. Cause we can stack them up, one on top of another, across the board. Explore, examine, use, evaluate, come back, don't come back. It's your choice. You're the driver. You're rich, you've got windows. A lot of them. I say cultivate your short attention span, there's nothing like it.

It's a scary world if we go against the grain. We need to admit it, we need to click: it's all about small windows.

python decorators

2007-11-15T04:26:00.000-06:00


"""
DECORATORS
PEP 318
Introduced in Python 2.4
"""

#-----------------------------------------
# simple decorator
#-----------------------------------------
def log(f):
    def new_f(*args, **kwargs):
        print f.__name__, ': called'
        print args, kwargs
        f(*args, **kwargs)
        print f.__name__, ': end of call'

    return new_f

class Simple(object):
    @log
    def do(self, *args, **kwargs):
        print '...done'


#-----------------------------------------
# decorator without sugar
#-----------------------------------------
class NoMagic(object):
    def do(self, *args, **kwargs):
        print '...done'
    do = log(do)


#-----------------------------------------
# decorator with arguments
#-----------------------------------------
def prefix(text):
    def fn(method):
        def new_fn(*args, **kwargs):
            print text,
            method(*args, **kwargs)
        #rename the method to be the same as the decorated one
        #new_fn.__name__ = method.__name__
        return new_fn

    return fn

class Args(object):
    @prefix('wait for it: ')
    def do(self, *args, **kwargs):
        print '...done'


#-----------------------------------------
# decorator with arguments and no sugar
#-----------------------------------------
class NoMagicArgs(object):
    def do(self, *args, **kwargs):
        print '...done'
    do = prefix('wait for it: ')(do)


#-----------------------------------------
# chaining decorators
#-----------------------------------------
class Chain(object):
    @log
    @prefix("I'm chaind but still: ")
    def do(self, *args, **kwargs):
        print '...done'

practical common lisp

2007-11-03T18:51:00.000-05:00

Peter Seibel's book Practical Common Lisp is a jewel.

The book introduces the reader to Lisp in a gentle way, but also covers most of the features that make Lisp so powerful: macros, multimethods, dynamic variables...

The book is also filled with historical notes that make for a very interesting read.
Towards the end of the book, entire chapters are devoted to building small apps, justifying the title.

The book taught me about Lisp, but also taught me about OO (sic) and about what a modern programming language should provide. Five stars.