Harish Mallipeddi's Blog

I.O.U.S.A. - Documentary on the US Federal Debt (30-min version)

Tue, 18 Nov 2008 16:07:59 +0800

I.O.U.S.A. - Documentary on the US Federal Debt (30-min version)

Using JInterface to communicate between Java & Erlang

Thu, 13 Nov 2008 00:53:54 +0800

I figured this stuff out this morning. JInterface ships by default with the Erlang distribution. JInterface implements the Erlang VM wire protocol and hence you can create pseudo Erlang-ish VMs using Java. Erlang code running on a different VM can then send/receive messages to the processes running on the Java node and vice versa.

Here’s the code (if you cannot view the GitHub gist embed, visit the direct link). In this example, I’m creating a process which receives messages of the format {FromPid, some_erlang_term} and echoes some_erlang_term back to FromPid.

Making of Guitar Hero World Tour

Sun, 09 Nov 2008 13:32:58 +0800

Making of Guitar Hero World Tour

The Reckoning - From Midwest to M.T.A., Pain From Global Gamble - Series - NYTimes.com

Sun, 02 Nov 2008 14:53:54 +0800

The Reckoning - From Midwest to M.T.A., Pain From Global Gamble - Series - NYTimes.com: Just shows how much of a complex and interconnected world we live in. This reminds me of what my mom mentioned a few days back on the phone. Life was much simpler (not necessarily awesome, but simpler) in India in the 80s when it was more or less isolated from the rest of the world.

User Experience Design Really liked this slide from a talk at...

Sat, 25 Oct 2008 06:22:59 +0800

User Experience Design

Really liked this slide from a talk at the recent Web2.0ExpoEU’08

Writing and deploying non-trivial Erlang apps

Tue, 21 Oct 2008 09:30:24 +0800

I recently wrote a twitter bot in Erlang which taught me a lot of things about writing Erlang programs. This was my first non-trivial Erlang program. Quite frankly it left me with a lot of new unanswered questions.

How much to OTPify?

It seems like when someone writes an Erlang app, they roughly go through the following steps:

Use vanilla Erlang processes (hook things up with links and trap-exits).
Switch to using gen_servers and supervisors.
Package as an application (implement the application behaviour and add a .app file).
Create a release (add a .rel spec file; create a boot script).
Installing a release.
Support for upgrades.

Btw Mitchell Hashimoto has a great series of articles on this exact topic on his blog which is where I learnt most of this from. For my bot, I stopped with packaging it as an application and didn’t do the rest of the steps. I start the bot by doing application:start(mybot). I get to do application:loaded_applications() in the shell to see if my app is running. When the time comes to deploy the bot, I tar up my code, scp that tarball to the production box, untar and just start the app in the shell. So far this works great for this bot. But how do folks do this in a real professional setting?

There’s very little reading material right now on topics like this. I feel like someone needs to explain and show you exactly the benefits you get by incrementally OTPifying your app. I bought the early-access edition of Manning’s new OTP book but so far I didn’t find anything interesting in that book.

Deploying Erlang daemons

Armstrong’s book and a lot of other online tutorials show you how to start/stop simple Erlang programs from within an Erlang shell. But what they don’t tell you is how to run daemons non-interactively without using the shell and I only figured this out after searching for a while in the Erlang mailing list.

-noshell flag

-noshell flag prevents the Erlang shell from being started but the output from the program is still printed out to your console. -s flag can be used to execute arbitrary MFAs. Supplied args will be sent to the function as atoms. If you want them to be sent as strings, use the -run flag instead. More info: man erl


%% supplied MFA should be a loop; otherwise it will stop immediately.
$erl -noshell -s Mod Func Arg -s init stop

%% use the -noinput flag; otherwise it enters the STOPPED state immediately.
$erl -noshell -noinput -s Mod Func Arg > myprog.out &

-detached flag

-detached flag starts an Erlang node completely detached from your console. This would mean you cannot see the output from the program unlike -noshell. But you can use the -remsh flag to start a new Erlang shell and connect to the previously started -detached node. More info: read Joe Armstrong’s tutorial on this topic.

Since you cannot see the output in the console, I replaced all my io:format/2 calls with appropriate error_logger calls and I setup multi-file rotating logs (as described in Armstrong’s book). But unfortunately when I used methods from the report browser (rb) module in order to parse the logs, the output seemed like it went to the -detached node instead of the node I made the invocation from. One patch from the mailing list confirmed that this was a bug in the report browser module.

embedded mode - run_erl and to_erl

I finally settled on using the embedded mode. run_erl takes in a $PIDDIR and a $LOGDIR, starts the Erlang program. Output is automatically logged to files in the $LOGDIR and you get log rotation for free. Supply to_erl the pid, and you get back into the shell running the program. You can execute new commands from this shell. This is sorta like using the screen command. More info: run_erl and to_erl. But this comes with its own caveat - if two users use to_erl at the same time, then the behavior is not explained.

Best of Velocity '08: Web Ops.

Tue, 07 Oct 2008 03:28:18 +0800

I finally had time tonight to look at the presentations from Velocity 08. If you’re interested, they’re all available on the Oreilly conference website.

Personally I thought the following two talks were awesome:

Capacity Management by John Allspaw (Flickr Ops Mgr) [PPT]
Building an Automated Infrastructure by Adam Jacob [PPT]

John Allspaw has a new book titled “The Art of Capacity Planning” and there’s an interview of him on the HighScalability blog mostly talking about his book. Key takeaway from his slides is “capacity planning” should be tied to application-level metrics (# of photos processed, etc) not just system-level metrics (# of busy Apache procs, disk-I/O).

Adam Jacob’s talk is a comprehensive guide to what steps need to be taken in order to achieve a 100% automated infrastructure. He has a nice checklist in one of his slides:

OS Install - Get an operating system up and on a network
DNS - Give your new system a name
Server Inventory - Have a place where you keep track of each system, and what it does (iClassify)
Identity Management - Grant your users access and privileges to your new servers (LDAP)
Version Control - Keep track of the changes to your application code, and ideally, your infrastructure too (SVN, git)
Configuration Management - Keep track of how each system is configured, and update it when you make changes (Puppet)
Monitoring - Watch your new systems for signs of trouble (Nagios)
Trending - Make graphs and charts of important metrics, so that you can tell if the infrastructure is behaving well, and predict future capacity (Ganglia, Cacti)
Application Deployment - Actually put your application on the infrastructure, and update it (Capistrano)

There are a lots of new things I need to play with and explore - LDAP, DNS servers, Nagios, Ganglia, Puppet, Capistrano. Expect a lot of follow-up posts on these topics soon!

"Tharman said Singapore’s banking system is safe amid strict central bank regulation and there..."

Mon, 06 Oct 2008 15:07:57 +0800

“Tharman said Singapore’s banking system is safe amid strict central bank regulation and there is no need to increase the minimum guarantee that insures deposits of up to S$20,000 ($13,800).”

- Singapore’s econ slowdown to last several quarters-FinMin - Forbes.com

NYTimes on the Valley's Green-Tech craze

Mon, 06 Oct 2008 01:27:47 +0800

NYTimes on the Valley's Green-Tech craze

Nassim Taleb's latest essay: A map of the limits of statistics in the fourth quadrant

Wed, 01 Oct 2008 01:09:28 +0800

Nassim Taleb's latest essay: A map of the limits of statistics in the fourth quadrant: Nice read. Extension of what he already wrote in his book, “The Black Swan”.

"If your object is responding to a method/message, and crashes halfway through, any changes you made..."

Mon, 29 Sep 2008 11:18:14 +0800

“If your object is responding to a method/message, and crashes halfway through, any changes you made to instance variables are rolled back to before the method is invoked. Transactional semantics, provided by gen_server, should hopefully help alleviate the problem of object state being corrupted by calls which error out halfway through after modifying instance variables.”

- Reia: Classes and Objects! : The Art of Science

Getting stack trace from a running Python application - Stack Overflow

Fri, 26 Sep 2008 11:33:16 +0800

Getting stack trace from a running Python application - Stack Overflow: This recipe is really cool. In order to debug a long-running Python process (if not you could just use pdb), you register a signal handler (for SIGUSR1). When you need to debug the process, send the process the signal. The signal handler then prints the stack trace, starts an interactive console (using the code module) and populates the context with the globals() and locals() from the current stack frame. Pressing Ctrl-D will quit the interactive console, and the process will continue.

Notes on Python's dict

Wed, 24 Sep 2008 10:15:36 +0800

Notes on Python's dict

Proposal: user-friendly API for multi-database support - Django developers | Google Groups

Mon, 22 Sep 2008 12:22:16 +0800

Proposal: user-friendly API for multi-database support - Django developers | Google Groups: Thanks to Cal Henderson’s talk at DjangoCon ‘08, things are moving fast wrt multiple DB support in Django, doing more frequent releases, etc.

Death and Near-Death Experiences on Wall St. - NYTimes.com

Mon, 22 Sep 2008 02:20:41 +0800

Death and Near-Death Experiences on Wall St. - NYTimes.com: Detailed recap of Lehman’s and Merrill Lynch’s CEOs long days at work over the last week.

BearHugCamp For Those Who Missed It archives | metajack.im

Tue, 16 Sep 2008 15:17:45 +0800

BearHugCamp For Those Who Missed It archives | metajack.im: Nice summary of everything that was discussed at BearHugCamp (which apparently is where all the cool micro-blogging folks hang out these days). Lots of talk about XMPP PubSub, OpenMicroBlogging (OMB) spec, BOSH, etc.

"By registering pownce:// and instructing the Service Provider to redirect the User to..."

Tue, 16 Sep 2008 14:15:03 +0800

“By registering pownce:// and instructing the Service Provider to redirect the User to pownce://access_token after authorization, the native Pownce application is automatically relaunched.”

- OAuth on the iPhone - I’m Mike

Pownce does OAuth the right way on the iPhone but a lot of users don’t get it apparently. This blog post was discussed quite a bit at the Yahoo Open Hack Day 2008 in lieu of all the OAuth-enabled APIs that Yahoo is planning to launch very soon (I’ll do a separate post on my trip later).

Barcamp Bangkok 2 talk - How to date a Jap girl

Thu, 04 Sep 2008 14:05:16 +0800

Barcamp Bangkok 2 talk - How to date a Jap girl:

31o5 is awesome…I loved the term “surface kindness” :)

Found this via Preetam

Using git with dropbox

Mon, 01 Sep 2008 11:54:45 +0800

Using git with dropbox

"The futures market that serves as a price discovery mechanism for the physical oil market is open..."

Sun, 24 Aug 2008 18:41:50 +0800

“The futures market that serves as a price discovery mechanism for the physical oil market is open only to the elite. We trust these elites to determine the prices, but who are they? Who are the so-called experts? Hedge funds, oil companies, OPEC — the very people who profit from massive, consistent increases in prices. Notice a conflict of interest? All an oil supplier would have to do to raise prices is buy up futures contracts.”

- Are Oil Prices Rigged? — Printout — TIME