The Dismal Blog

Loose Ends

2010-09-10T13:46:00.001+05:30

So the dust has settled a little bit (for those that don’t know this, I have moved from here) and I find myself in the possession of an incredible piece of hardware (no pun intended). I will also never underestimate Americans when they talk about size. They said it was big, and it is huge. And I have owned some large laptops in my time. Hence this (almost) celebratory blog post.

~*~

I have known the Dismal Blogger a long time and believe I know him well. Yet, over the last year, he has surprised me with his generosity and warmth. I thank him from the bottom of my heart for this.

~*~

Partly because of this gratitude, I believe I should find an alternate home for my often arcane material. The readers of this blog, carefully collected by the Dismal Blogger over the course of many years – discerning people all – have spoken out unequivocally against the hi-jacking of this blog, and I could not, as a statistician, ignore that evidence. I haven’t found a new home yet, but once I do, I will make it known on this blog. In cyberspace as in real life, I thank the Dismal Blogger for being so generous with his blog.

~*~

Following advice here, I am using Windows Live Writer to compose this blog post. And it is nice!

Is that intuition, or are you just happy to see me?

2010-07-17T16:29:00.001+05:30

Some intellectual wrangling with the Dismal Blogger resulted in us reaching a well-known (atleast to me) impasse. What is the role of intuition and simplicity in science and mathematics, and is it antithetical to rigour and complexity? Despite my ardent desire to learn, I am poorly versed in methodological questions. Therefore, I supply my favourite quotation on this issue. Obviously, being a quotation, it doesn't amount to much, but it is generally how I feel about the rôle of rigour.

The rôles of rigor and intuition are subject to misconceptions. As was pointed out in Volume 1, natural intuition and natural thinking are a poor affair, but they gain strength with the development of mathematical theory. Today's intuition and applications depend on the most sophisticated theories of yesterday. Furthermore, strict theory represents economy of thought rather than luxury.

- An Introduction to Probability Theory and its Applications, Volume II (2^nd edition), Feller, William (1971), pp. 3, footnote 4
In the pantheon of extraordinary textbook writers, Feller is second to none, cramming in as he does relevant example after relevant example, but never giving in to the desire to jump to conclusions based on examples - preferring to work through to confirm/deny his intuition, and using his results to inform his intuition. This last is important, because if like the American astronauts, we were to wait for all problems to be solved rigorously before we ever did statistics, we'd be here a while. The part played by experience and common sense in settings where cost, scale and speed are issues is invaluable. Just yesterday, Andrew Gelman, over on his blog dredged out this quote:

Statisticians are particularly sensitive to default settings, which makes sense considering that statistics is, in many ways, a science based on defaults. What is a "statistical method" if not a recommended default analysis, backed up by some combination of theory and experience?

I can understand that practitioners would typically choose to let the theorists battle it out amongst themselves for a decade or two to figure out relative merits of techniques, and only then move to gather up the spoils of war i.e. the most-widely applicable/median/default method. But I fear this is sub-optimal. There is little comfort in something that often works, but fails when you need it to work most (e.g. macroeconomics).
On a related note, I am little uncomfortable with the phrase - complexity for complexity's sake. There is, in my opinion, no such thing. Complexity relates to how you “do” science. Proofs can be elegant or workmanlike and algorithms & techniques can be crude or sophisticated, and you can choose one or the other based on your preferences. But the actual content of science, theorems, are exactly that, little bits of value-neutral truth which give you the power to discriminate. And just like John Locke:

To love truth for truth's sake is the principal part of human perfection in this world, and the seed-plot of all other virtues.

so we too claim.
Is all rigour a good thing? Obviously not. But useless rigour is easily seen for what it is (if assessed by a trained person) and cast aside. Not that easy to cast aside rules-of-thumb or traditions or defaults, because by definition, these things are hard to pin down.
Further reading

Reconstructing aspects of the Receiver Operator Characteristic (ROC) curve in Stata

2010-07-15T01:22:00.009+05:30

Lorenz cuvres and Gini coefficients which are typically used as summary measures of inequality, are ubiquitous in development economics. A closely-related measure, the Receiver Operating Characteristic [ROC] curve is a commonly used visual model selection tool in classification problems. In a recent conversation with the Dismal Blogger, some issues of implementation came up and since I knew nothing about these measures, I decided to look them up and see if I could figure out how they are used. In this post, I mainly discuss the ROC curve.

ROC

Predictions from a model for binary data can be summarised neatly using the so-called confusion matrix. Let the outcome of interest be $y \in \{0, 1\}$ and predictions from the model be $\hat{y}$, then we have the following 2x2 contingency table:

$\begin{tabular}{|c||c c|}
\hline
& \begin{math}y=1\end{math}&\begin{math}y=0\end{math} \\ \hline
\begin{math}\hat{y}=1\end{math}&True positive [TP]&False positive [FP]\\
\begin{math}\hat{y}=0\end{math}&False negative [FN]&True negative [TN] \\
\hline
\end{tabular} $

The ROC is a fairly intuitive summary of the cost of acquiring true positives in terms of the increase in false positives, as it plots the True Positive Rate [TPR] $\left(=\dfrac{TP}{TP+FN}\right)$ against the False Positive Rate [FPR] $\left(=\dfrac{FP}{FP+TN}\right)$ for changing values of the criteria of discrimination (to be explained below). The first term is also known as the sensitivity and the second term is $1-$specificity. The idea being that as the cost (FPR) increases, the benefits (TPR) should dominate it.

To construct these numbers, we need predicted values of the outcomes from our model, $\hat{y}_i$, where $i \in {1, \dots, N}$ is the observation index, from a sample of N individuals. Typically, models for binary data, like the logit, do not produce binary predictions, but predicted probabilities of positives, i.e., $\hat{p}_i = \widehat{Prob}(y_i=1|\mathbf{x}_i)$, where $\mathbf{x}_i$ is a vector of regressors. Therefore, a binning criteria is necessary, say, $\hat{y}_i = \mathbf{1}(\hat{p}_i> c)$ for some $c \in [0, 1]$. Allowing the binning threshold to vary in its domain allows us to estimate an ROC curve for a particular model. For $c=0$, TPR = 1 and FPR = 1, so (1, 1) is a point on the curve, and for $c=1$ both TPR and FPR are zero (all predicted outcomes are zero), thus, (0,0) is a point on the curve. Note that the the entire ROC curve is contained in the unit square.

Area Under Curve [AUC]

While the entire ROC curve, for various values of the threshold parameter, can be used to assess the fitted model (more on this in a later post), single-number summaries are popular, and one such summary number is the area under the ROC curve. The interpretation is that it is the probability with which an arbitrarily chosen positive will be classified (by the model) as having higher probability than an arbitrarily chosen negative point.

This number can be estimated by numerical integration, which due to the irregular nature of the estimated curve, is typically done by the method of trapezoids (I knew high-school math was going to come in handy one day!). Stata's default numerical integration algorithm is by fitting cubic splines to the estimated curve which better for smoother curves.

An example

Here is an example of how the ROC curve is constructed by hand in Stata using the Hosmer & Lemeshow dataset. This is not remarkable in itself, but hopefully be a building block for more complicated examples. I draw the ROC curve based on a logistic fit and compute the area under the ROC curve using numerical integration:

/*********************************************
* Reconstruct the ROC curve in Stata
**********************************************/
clear*
webuse lbw

tab low
logistic low age lwt i.race smoke ptl ht ui  // fit a logistic model
predict prob_low, pr  // predcited probabilities
cap matrix drop roc
// calculate the ROC curve using the percentiles as thresholds
forv i=1(1)99 {
 cap drop pred_low
 cap drop sens
 cap drop spec
 _pctile prob_low, nq(100)
 local cutoff = `r(r`i')'
 qui {
  g pred_low = (prob_low >= `cutoff')
  g sens = (pred_low == 1 & low==1)  // true positive
  g spec = (pred_low == 0 & low==0)  // true negative
  count if sens
  local sens_mean = `r(N)'
  count if low  // positives
  local sens_mean = `sens_mean' / `r(N)'
  count if spec  
  local spec_mean = `r(N)'
  count if !low  // negatives
  local spec_mean = `spec_mean' / `r(N)'
  local sum =`sens_mean' + `spec_mean'
 }
 mat roc = (nullmat(roc)\ (`sens_mean', 1-`spec_mean', `sum'))
}
mat roc = (1, 1, .)\ roc
mat roc = roc \ (0, 0, .)
cap erase roc_mat.dta
svmatf , mat(roc) fil("roc_mat.dta")  // save the matrix of ROC points

preserve
use roc_mat, clear
integ c1 c2, trapezoid  // calculate the area under the ROC curve using the trapezoidal rule
local integ_trap =`r(integral)'
integ c1 c2    // calculate the area under the ROC curve using cubic splines
local integ_cub =`r(integral)'
twoway line c1 c2, xtitle("1-specificity") ytitle("Sensitivity") ///
 note("The area under the curve is estimated as: " ///
 "`integ_trap' (trapezoidal) and `integ_cub' (cubic spline)") ///
 xla(0(.25)1, grid) yla(0(.25)1) ///
 title("ROC curve for estimated model") subtitle("Manual calculation")
gr export roc.png, replace 
restore

logistic low age lwt i.race smoke ptl ht ui
lroc, title("ROC curve for estimated model") recast(line) ///
 subtitle("Stata's calculation") ///
 lpattern(solid)  // calculate the area under the ROC curve using built-in function
gr export lroc.png, replace

Then, the ROC curves look like this:

Note that in the calculation of the integral, Stata only uses 79 points due to the one-many nature of the estimated function (I think).

Miscellaneous points

There were several other points I wanted to cover in this post but will have to postpone for lack of time, including,

the drawbacks of using the AUC, as covered here;
the relationship of the AUC measure to the Gini coefficient as given here;
and the Lorenz curve as the rotated ROC about the 45 degree line of symmetry.

PS. I'd like to point out again that I go to some lengths to ensure that my code runs out-of-the-box, and so is, in marketing parlance, free to try!

Fixed format data in Stata

2010-07-05T16:23:00.000+05:30

I have never needed to use fixed-format ASCII data in Stata; I typically work with CSV files. However, I was looking for data to replicate this paper by Bill Greene on a general method to incorporate selectivity into limited dependent variable models.
I was about to write to him for the data, but decided to take a quick look through his website to see if he did provide the data. Turns out he provides a subset of that dataset, Table F25.1: Expenditure and Default Data, 1319 observations, as part of the example datasets of the 6th edition of his massively bestselling Econometric Analysis.
Small problem is that that dataset is a fixed-format text file and probably formatted as an Nlogit/Limdep dataset and reading it into Stata is not straightforward.
So, I decided to figure out how to write a dictionary file and read that data in using Stata's -infile- command. The dictionary file looks like this:

dictionary {
 _first(4)  * first line of data is the fourth
 _lines(3)  * there are three lines of data per observation
 
 _line(1)   * begin with line one of each observation
 Cardhldr "Dummy variable, 1 if application for credit card accepted, 0 if not"
 Majordrg "Number of major derogatory reports"
 Age  "Age n years plus twelfths of a year"
 Income  "Yearly income (divided by 10,000)"
 Exp_Inc  "Ratio of monthly credit card expenditure to yearly income"
 
 _newline   * move to the next line of an observation
 Avgexp  "Average monthly credit card expenditure"
 Ownrent  "1 if owns their home, 0 if rent"
 Selfempl "1 if self employed, 0 if not."
 Depndt  "1 + number of dependents"
 Inc_per  "Income divided by number of dependents"
 
 _newline  * move to the next (last) line of an observation
 Cur_add  "months living at current address"
 Major  "number of major credit cards held"
 Active  "number of active credit accounts"
}

Save this file as "limdep2stata.dct". Then, this dictionary file can be used to read in the data using a do-file which looks like this:

/*
* Read in LIMDEP data in Stata
*/
infile using limdep2stata.dct, using(TableF25-1.txt) clear 
renvars _all, lower
drop if cardhldr==.  // one extra line read in

I guess the data file lacks an end-of-file delimiter and so an extra line is read in before Stata figures out that the file has ended. I will see if there is a simple solution to avoid this. But it does no harm and the extra line is easily dropped.
And that's it! You are good to go.
PS. I must mention that the command -renvars- is due to Nick Cox and Jeroen Weesie.

Blog clog

2010-07-04T04:51:00.003+05:30

No sooner had I fixed the last problem with this blog, I returned home today to discover that Alex Gorbatchev who writes and maintains SyntaxHighlighter [SH], which I use to highlight the code on this blog, had made a major revision up to ver. 3.0.83 and guess what, it broke all the highlighting on my previous R code, even though the code highlighted by other brushes looked fine. My R code looked like a dog's breakfast. Since the R brush was written by someone else, I assumed that the changes to SH had made the brush incompatible with the new SH. I looked at the changes in the new version and decided on balance that I needed the R brush fixed more than I needed the new features of SH. So I started upon fixing the mess.

I decided to download the last version of SH (where all my codes rendered properly) and host it myself. Previously, I was using the versions Alex was hosting on his server space and which he periodically updated with new versions.
Since most of SH is JavaScript [JS], and I have had enough of hosting JS at random places on the internet (most uploading sites do not accept JS, so you need to find specific places that will), I decided to muck around with hosting it on a Google Code project. I re-used the project I had created to host the JS of the $LaTeX$ renderer. I decided to wait until I had figured it out to create a separate code project to host SH.
This allowed me to do something I have always wanted to figure out - see how version control software works. I had installed Tortoise SVN eons ago but never used it. It was pretty simple as I just followed the instructions here.
The main thing was to figure out how to set mime-types for the various files whether they be CSS or JS. I learnt how to do that here.
Finally, I replaced the Blogger HTML template with the new links to the JS files associated with SH.

Then I spent half-a-day scratching my head because after checking everything, it would still not work (the R code rendered with visible errors). After much staring at the screen, I realised that the old SH versions were arranged chronologically rather than reverse-chronologically as I had expected and I had downloaded the oldest available minor revision of release 2! Quickly rectified and sanity was restored!
All that now remains to be done is to migrate all the SH code to a new code project which will contain all the JS associated with this blog, including the $LaTeX$ renderer. Then I can be free of the whims and fancies of developers and servers!

Javascript, HTML, jQuery and the Milk of Human Kindness

2010-07-02T01:09:00.000+05:30

Some of you who read this blog (optimistically) would have noticed the ugliness which was perpetrated on this blog in this previous post since I put up this other post. This was largely because the $LaTeX$ parser I use to display math was also encoding certain bits of R code with the character "$!$" as $LaTeX$.

Since then I have had a great if sometimes frustrating time googling and querying the developer forums and have learnt a lot about how HTML, Javascript and jQuery work. Thank you in particular to the anon. moderator on the webdeveloper forums who spent quite a bit of time helping me through this.
The trick is to define and send the contents of a user-defined div class to the parser on load. This involves setting up your Blogger "HTML/Javascript" widget to contain the Javascript:

and then include everything you want the parser to parse as in the following example:

$\frac{1}{2}$

Unbiasedness and 2 counterexamples

2010-06-30T23:33:00.005+05:30

The role of unbiasedness in statistical decision theory is ambiguous. Of the various criteria being considered here, it is the only one that does not depend solely on the risk function. Often we find that biased estimators perform better than unbiased ones from the point of view of, say, minimising the mean squared error. For this reason, many modern statisticians consider the whole concept of unbiasedness to be somewhere between a distraction and a total irrelevance.

- Essentials of Statistical Inference, Young, G.A. and R.L. Smith, CUP (2005), pp. 10

I will supply two examples - one innocuous and one dramatic to illustrate the problems with unbiasedness as a criterion for selecting estimators. These are both taken from here, but the second is orginally from here.

Consider the problem of estimating the variance of observations $\{X\}_{i=1}^{n}$ drawn from a Gaussian density with mean $\mu$ and variance $\sigma^2$. Consider the estimators \[s^2 =\frac{1}{n-1}\sum_{i=1}^n(x_i-\bar{x})^2\] and \[t^2 =\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2\] for $\sigma^2$ where $\bar{x} = \frac{\sum_{i=1}^n x_i}{n}$. $s^2$ is often insisted upon as it is claimed to be better due to its unbiasedness. However, on the more relevant criteria of mean square error (what good is an unbiased estimator for a parameter with true value zero (say) if it is -100 half the time and 100 the other half? I'd rather have the biased estimator that is +0.0001 with probability 0.6 and -0.0001 with probability 0.4) $t^2$ is superior in this case, as \[\mathbb{E}(t^2-\sigma^2)^2 < \mathbb{E}(s^2-\sigma^2)^2\]

The differences in the two estimators above diminishes as the sample sizes increase and they converge to the same probability limit, $\sigma^2$, and thus are both consistent. A remarkable situation where this is not the case is when $X \sim Poisson(\lambda)$, $\lambda \in \mathbb{R}^+$. An unbiased estimator $\delta(X)$ for $e^{-2\lambda}$ satisfies the requirement \[\mathbb{E}(\delta(X)) = \sum_{x=0}^{\infty}\delta(x)\frac{e^{-\lambda}\lambda^x}{x!} = e^{-2\lambda}\], where $\frac{e^{-\lambda}\lambda^x}{x!}$ is the Poisson mass function.
But from the power series expansion of $e^{-\lambda}$ we know that the only possible such function $\delta(X)$ is $(-1)^{X}$. Then if $X=100$, we are lead to estimate the parameter $e^{-2\lambda}$ as 1 whereas $\lambda$ is would need to be fairly high to produce a realisation of 100. And even more incredibly, if $X=3$, then $\hat{\delta} = -1$, an estimate for a parameter which must lie in $(0, 1]$. A better estimator for $e^{-2\lambda}$ is $e^{-2X}$ which is, in this case, the maximum likelihood estimator.

Multinomial logit in Stata and R III

2010-06-30T14:21:00.003+05:30

Another set of translations between Stata and R - calculation of the most important kind of margins (see previous post), i.e., the marginal effects. This requires us to use yet another R package for out-of-the-box calculations of marginal effects, although I do also show how to do this by hand. Note that in both the R constructions, the standard errors are missing, and in a future post I will show how to calculate the standard errors analytically using the delta method and a nonparametric bootstrap.

Here I construct the marginal effects for the covariate "age" on the predicted probabilities of each of the three categories of the outcome. For the automatic construction of marginal effects in R, I make use of another package, VGAM and its vglm() and margeff() functions.
R

########################
# calculate the marginal effects
########################
library(VGAM)
library(gregmisc)
library(foreign)
sysdn <- read.dta("D:/programming/r/sandbox/sysdn1.dta", convert.factors=TRUE)
sysdn.multinomial <- vglm(insure~age+male+nonwhite+as.factor(site), data=sysdn, multinomial)
sysdn.multinomial
margeff.sysdn.multinomial <- margeff(sysdn.multinomial)
rowMeans(margeff.sysdn.multinomial["age",,])  # marginal effects
# manual construction of ME
p <- fitted(sysdn.multinomial)  # p_ij (row `i' = individual, column `j' = choice) here, a 644x3 matrix
avgmargeff.age1 <- mean(p[,1]*(coef(sysdn.multinomial)["age:1"] - p[,2]*coef(sysdn.multinomial)["age:2"]- p[,1]*coef(sysdn.multinomial)["age:1"]))
avgmargeff.age2 <- mean(p[,2]*(coef(sysdn.multinomial)["age:2"] - p[,2]*coef(sysdn.multinomial)["age:2"]- p[,1]*coef(sysdn.multinomial)["age:1"]))
# note that insure="Uninsure" is the base category and the coefficient vector is zero
avgmargeff.age3 <- mean(p[,3]*(0 - p[,2]*coef(sysdn.multinomial)["age:2"]- p[,1]*coef(sysdn.multinomial)["age:1"]))
avgmargeff.age1 
avgmargeff.age2
avgmargeff.age3

Stata

webuse sysdsn1, clear
mlogit insure age male nonwhite i.site, base(3)
margins, dydx(age) predict(outcome(1)) // average marginal effects
margins, dydx(age) predict(outcome(2))
margins, dydx(age) predict(outcome(3))

I think this back-and-forth between languages and packages makes a good case for learning more than one programming languages. Somethings are more easily done in one or the other - a valuable bit of flexibility when working on a large project. And to round it off, here is a very good explanation of what margins are, quoted from [R], Stata's base reference manual:

What we call margins of responses are also known as predictive margins, adjusted predictions, and recycled predictions. When applied to balanced data, margins of responses are also called estimated marginal means and least-squares means. A margin is a statistic based on a fitted model calculated over a dataset in which some of or all the covariates are fixed at values different from what they really are. For instance, after a linear regression fit on males and females, the marginal mean (margin of mean) for males is the predicted mean of the dependent variable, where every observation is treated as if it represents a male; thus those observations that in fact do represent males are included, as well as those observations that represent females. The marginal mean for female would be similarly obtained by treating all observations as if they represented females.

EDIT: Thank you to Gautam C. for pointing out what happens to HTML style comments if they are in the CSS of an HTML file.

Multinomial logit in Stata and R II

2010-06-29T22:29:00.008+05:30

A couple more interesting translations:

If you wanted to change the base outcome in the estimation, the multinom() function in nnet is no longer sufficient and we use the mlogit package. Note that this package is more in line with the motivation of multinomial logit as a choice model and thus expects the data in a form where for each choice occasion, all choices are enumerated as separate rows in the data.
Stata

webuse sysdsn1, clear
mlogit insure age male nonwhite i.site, base(2)

library(effects)
library(foreign)
library(mlogit)
# using the nnet package
sysdn <- read.dta("D:/programming/r/sandbox/sysdn1.dta", convert.factors=TRUE)
sysdn.multinom <- multinom(insure~age+male+nonwhite+as.factor(site), data=sysdn)
summary(sysdn.multinom)
# change the base category of the response
sysdn$site <- as.factor(sysdn$site)
sysdn$insure <- as.factor(sysdn$insure)
sysdn.mldata <- mlogit.data(sysdn, varying=NULL, choice="insure", shape="wide")
sysdn.mlogit <- mlogit(insure ~ 1|age+male+nonwhite+site, data=sysdn.mldata, reflevel="Indemnity") 
sysdn.mlogit2 <- mlogit(insure ~ 1|age+male+nonwhite+site, data=sysdn.mldata, reflevel="Prepaid") 
summary(sysdn.mlogit)
summary(sysdn.mlogit2)

You might want to predict (identified statistics) using the estimated model at various values of the covariates. In Stata 11, the new -margins- command offers a very general framework for doing this. I would strongly recommend reading [M] margins (the pdf documentation) for a very good description of what can be achieved using -margins-. Let us see is we can encode losslessly to R. I use the example I found here.
R

# calculate predicted probabilities
mydata <- read.csv(url("http://www.ats.ucla.edu/stat/r/dae/mlogit.csv"))
attach(mydata)
names(mydata)
mlogit.model<- mlogit(brand~1|female+age, data = mldata, reflevel="1")
summary(mlogit.model)  
newdata <- data.frame(cbind(age = rep(24:38, 2), female = c(rep(0, 15), rep(1, 15))))
logit1 <- rep(0, 30)
logit2 <- -11.774655 + 0.523814*newdata$female + 0.368206*newdata$age
logit3 <- -22.721396 + 0.465941*newdata$female + 0.685908*newdata$age
logits <- cbind(logit1, logit2, logit3)
p.unscaled <- exp(logits)
p <- cbind(newdata, (p.unscaled / rowSums(p.unscaled)))
colnames(p) <- c("age", "female", "pred.1", "pred.2", "pred.3")
p

Stata

insheet using http://www.ats.ucla.edu/stat/r/dae/mlogit.csv, comma clear
mlogit brand female age, base(1)
margins, predict(outcome(1)) at(age=(24(1)38) female=(0 1))
margins, predict(outcome(2)) at(age=(24(1)38) female=(0 1))
margins, predict(outcome(3)) at(age=(24(1)38) female=(0 1))

The R code is more cumbersome, but it has the advantage of laying bare the structure of the construction, which is very valuable while learning about these models.

Multinomial logit in Stata and R

2010-06-29T19:53:00.007+05:30

Here is how you can reproduce Stata's -mlogit- in R:
Stata


websuse sysdn1, clear
mlogit insure age male nowhite i.site


install.packages("foreign")
library(foreign)
sysdn <- read.dta("D:/programming/r/sandbox/sysdn1.dta", convert.factors=TRUE)
sysdn.multinom <- multinom(insure~age+male+nonwhite+ as.factor(site), data=sysdn)
summary(sysdn.multinom)

And as of today, I am using Revolution R's optimized 64-bit version of R which they have made free for academic use. The function multinom() in R comes from the pre-loaded nnet package. Syntax highlighting for R code here . Still looking for a brush/syntax highlighter for Stata. If anyone knows a good one, please do let me know.

Flash charts in Blogger

2010-03-02T20:26:00.005+05:30

George & Ravi have a fantastic blog here, where they discuss their trading strategies for their fund. It is eminently readable and they have some great ideas.
However, George mentioned that they rely on pre-fabricated graphs and charts to illustrate their point. I thought it might be nice to have dynamic charts which could be embedded in Blogger.

So after some searching and quite a bit of tinkering, I managed to produce some charts that might give them some autonomy in creating graphs for their blog.

There were a few steps to making this chart:

After some googling I decided to use the set of tools provided by amCharts which embeds a Flash object using the protocol provided by SWFObject, into a blog post. It uses an XML file to control the flash object settings and to feed in the data. In the XML file, I needed to change only the line which points to the data (I was using online hosting, so to a hyperlink - see below). There are numerous other settings which I intend to play around with as I make more charts.

I uploaded the files associated with the project so that they can be hotlinked. This allowed me to try out the file uploading facility provided by Google Groups. I uploaded the data, the XML file, and the amCharts flash object there (you don't need to make your group public). The good part about this is that by changing dataset uploaded at Google Groups, the chart is automatically updated. It does not accept JavaScript however and to upload the .js file associated with SWFObject, I used yourjavascript.com, which, conveniently, does not need registration.

I chose to embed the chart in Blogger without using SWFObject as it was causing some problems. This works fine except that as this page mentions IE users will have to activate Flash when they view the site. In addition, you will need to have Flash 8 or higher installed.

The most important and incomplete part is the dynamic aspect, i.e., amCharts allows you to serve up the data to it dynamically everytime it loads. However, this will require some PHP programming which I could not get to work right off the bat. The final plan is to have a PHP script serve the data from the Google Finance API, replace the older .csv over at Google Groups (I guess this will require some encryption as well, because authentication will then be an issue), and the chart will update dynamically (there is an option in the amCharts controls to update at regular frequencies).

Since the chart is a little wide, I have embedded it in this other blog, which I tend to use as a sandbox for Blogger projects. I wish the fund, Aadiyatna, a lot of luck!

EDIT: Turns out you do need to make Google Group content public.

Hello (F#) World!

2010-02-25T01:26:00.007+05:30

My hobby: trying out new programming languages. Here is Microsoft's F# which is set to become standard in Visual Studio 2010 which is pretty cool (the number of F# jobs will skyrocket, for example). There are a whole host of functional programming languages out there from the rather pure (Haskell) to those with imperative support (OCaml), though the jury is still out on which is the best. I will try to learn it for rather more mundane tasks than it was intended, and hopefully post my progress here. Download the CLI here & command line options here.


printfn "Hello World!"

PS. Syntax highlighting via SyntaxHighlighter with C++ stripes.

Edit: Found an F# brush for SyntaxHighlighter here.

Summarise

2010-01-03T13:33:00.003+05:30

With G's permission, I am hi-jacking this blog for a little bit of statistical computing blogging. Mostly basic stuff, feel free to ask loads of questions. Ask for code!
G and I had a few discussions about data cleaning and pre-processing. Although I tend to do this mostly by intuition (read brute force), the first thing I always do (after labelling the data) is to look at summary statistics. In particular, I look at the so-called Five-number Summary - part of the overarching scheme of John Tukey's exploratory data analysis. These are the sample minimum, first quartile (25th %ile), the median, the third quartile (75th %ile) and the sample maximum; and I tend to throw in the mean and the sample size as well. This output has been surprisingly useful to me for capturing data recording errors and inconsistencies.
In R, this can be had by fivenum(x, na.rm = TRUE) and in Stata using John Gleason's -univar- (SSC).

Discussion Topic #1

2009-06-02T18:26:00.002+05:30

Assuming people are homogenous (racially, culturally etc. so that nationalist emotions are irrelevant), what is the most optimal way to divide the world into different countries?

Assume also that natural resources, population and economic development are all as they are today. I haven't defined what I meant by 'optimal', that should be a part of the solution.

Feeling blue?

2009-05-29T19:35:00.006+05:30

I've got the perfect solution

Step 1: Download

[You and I.mp3] - Queen

Step 2: Play

Enjoy! :D

Music is playing in the darkness
And a lantern goes swinging by
Shadows flickering my heart's jittering
Just you and I
Not tonight come tomorrow
When ev'rything's sunny and bright (sunny and bright)
No no no come tomorrow 'cause then
We'll be waiting for moonlight

We'll go walking in the moonlight
Walking in the moonlight
Laughter ringing in the darkness
People drinking for days gone by
Time don't mean a thing
When you're by my side
Please stay awhile

You know I never could forsee the future years
You know I never could see
Where life was leading me
But will we be together forever?
What will be my love?
Can't you see that I just don't know

No not tonight not tomorrow
Ev'rything's gonna be alright (sunny and bright)
Wait and see if tomorrow we'll be
As happy as we're feeling tonight
We'll go walking in the moonlight (we'll be happy)
Walking in the moonlight

I can hear the music in the darkness
Floating softly to where we lie
No more questions now
Let's enjoy tonight
(Just you and I) just you and I
Just you and I
Can't you see that we've gotta be together
Be together just you and I just you and I
No more questions just you and I

I'm watching

2009-05-29T00:24:00.005+05:30

The MSE years

2009-05-27T23:07:00.017+05:30

By the 15th of June 2007, things had really hit a low point and I was in pretty bad shape. Plagued by depression, fear and most of all the constant questioning of my own rationality, I needed something to happen to just keep things together.

The call from MSE came as a surprise. But it was an opportunity. A start that could turn everything around. The question facing me was: how do you do something that makes you sick to your stomach and that too for two years? Don't get me wrong. I loved Economics; its the exams and cramming that killed me.

~Digression~

I sometimes do a thought experiment where I choose some point in my history where I go back and replace the 'me' who made all those mistakes with all this new insight and knowledge. Stephen's is always the place I'd chose to start over. I don't know why. I did have fun and didn't screw up so badly over there. I guess Stephen's is the sort of place where a multitude of possible future paths may emerge. As the experiment progresses however, I begin to feel that regardless of how amazing a life I may be able to live this time round, it just isn't worth doing it all over again (I'm assuming my memory is intact when I replace myself)

My first month at MSE was very lonely. I was stuck up in the Guest flat away from the main hostel and none of my roommates had arrived. I was never good at just going out there and meeting new people unless I was forced to. So, initially, I spent a lot of my time in my room, horizontal. There was however internet access in the room and that changed things. Till then I only used the net at cyber cafe's and computer labs primarily for checking mail. This internet access blew my mind. I discovered blogs by several Stephanians with whom I had lost all contact. Initially it felt very wierd. When the surrounding environment doesn't change, at least for me, thoughts tend to be repetitive and I end up in my comfort zone. A sort of intellectual stagnation, if you will. I felt that I was somehow left behind. People were out there working and earning money, while others had started their PhDs. People had moved on, talking about new things, reading new books, listening to new genres of music while I had not changed a single bit from college and that was frightening.

The music I was listening to had not changed since my first year at Stephen's. The blogosphere was abuzzing with all sorts of new sounds and bands. I was soundly of the opinion, at that time, that rock was well and truly dead. I could not have been more wrong. I spent the first few months getting acquainted with likes of the Arctic Monkeys, The Killers, The Libertines, The Strokes, Vampire Weekend, (rediscovering) Oasis and so on (all thanks to zonuts.wordpress.com and millenniumhand.wordpress.com). This really fired me up. It was so different from what I was used to listening to (the usual classic rock stuff) and it was a rollercoaster ride of discovery, every day.

Classes were really easy. So all I did was read and watch videos online and nothing else for the first semester. I read 1984, Shalimar the Clown, The Glass Bead Game and 70% of The Idiot. 1984 fit right into things. I identified with Winston and his paranoia. I took pleasure in the way the sentences seemed to from themselves with Rusdie (I like the story of this book a lot too). Joseph Knetch of Castalia was a constant reminder to slow things down and to just take pleasure in knowledge. This really helped rejuvinate me thoroughly.

I wasn't all that anti-social either. I did end up meeting people and hanging out. I would take any opportunity to go to the beach. I just loved standing there in front of the massive ocean and feeling so insignificant.

~Digression~

MSE as an institution was rather peculiar. The people, teachers included, were extremely friendly. Something I was really not used to, within an academic setting. What's more was that the faculty seemed to have a different motivation for their classes i.e. for the first time since school, the emphasis was on the fact that everybody understands whats going on. Given that this is a master's level course, that may seem a little like spoon-feeding but this is what lead to the unique environment there. Intense competition was not prevelant. Collaboration was more the norm. Hands-on rather than internal visualization was the preferred way to understand things. There were free riders. I can't conclude that this was indeed a better approach. Students were being served all this knowledge on a silver platter. I found the reciprocal dishonesty, in terms of cheating and reckless nonchalance, loathsome.

I couldn't understand the motivation the teachers had for this. Their primary output was their research. There were teacher evaluations by students which were apparently taken seriously. Although I doubt if that is sufficient cause for them to put in such extra effort. There was also genuine humility among the teacher. The better the teacher the more humble they seemed to be (again, extremely uncharacteristic, if you've come from Delhi). That being said, humility was a virtue some teachers posessed and for others, it was forced upon them by their incompetence. The important thing I guess was that they cared. It mattered to them what we thought or felt.

~Digression~

If you've lost your faith in love and music
Oh the end won't be long
Because if it's gone for you then I too may lose it
And that would be wrong

I've tried so hard to keep myself from falling
Back into my bad old ways
And it chars my heart to always hear you calling
Calling for the good old days
Because there were no good old days
These are the good old days

- The Good old Days by The Libertines

Then there were after hours. I've had some of the most stimulating conversations there. The usual formula revolved around getting slightly wasted, either on the terrace or the COE steps and posing a hypothetical. The scientific method would be fully applied: assumptions would be clarified and step by step deductions drawn, till the generalized theory was agreed upon and stated. I've wanted to post those discussions here, but it was always seemed like too much work. Let me see if I can remember that stuff and post it on this blog.

Initially it was great meeting all my new classmates. Every evening people would be walking outside the hostel making chit-chat. The comments were usually guarded and the conversations had a sharp edginess to it, as they normally do when you have them with someone for the first time. As time progressed, people started forming small groups (I guess a minimization of said edginess over the set of all new people choosing n, number of people in the group, or the dual problem of maximising compatibility :-) ) of their own and those massive hanging out sessions stopped. This left me where I started: back in my room. I didn't mind it much. Being such a small college though, unlike Stephen's, if you weren't hanging out with people, you were alone. I was never alone in Stephen's: there was always somebody in my room or I was in someone else's. It was great though, getting to spend so much time by myself. The internet was the main provider of companionship.

The average day was pretty simple. Classes in the morning. Lunch. Nap for a bit. Evenings were tricky, since there simply was nothing to do (most of the time I'd go out in search of a snack). This was later replaced by football in the last semester and our own little league was an eagerly anticipated and regular feature. The late night sessions got rarer and towards the end, it was very much like the beginning. People had become like strangers with their own dreams and plans.

This blog is not dead, yet

2009-05-26T19:10:00.004+05:30

Following an amazing turn around in the author's perspective on sloth and laziness, and that terrible feeling he gets from the unending battle between the willingness of the mind and absolute lethargy of the body, he has decided that a fresh wave of life will be infused into this blog.

The blog will be expanded to include such exciting topics as

The author's personal life
Corporate life and the accompanying emptiness (of head, soul and purpose)
Economics (I know!)
Introspective nonsense

Risk

2008-12-16T16:47:00.007+05:30

On another note

2008-09-29T00:17:00.003+05:30

Please make your way over here. This is another blogging venture specifically about financial markets and investment strategies, started by my good friend Ravi Saraogi and myself.

On Irony

2008-09-24T23:10:00.006+05:30

I'm taking a bit of a step away from economics for a while, waiting for inspiration to strike. So I decided to re-read a book (for the first time). The re-reading of 'Sophie's world' was interesting, because the things I found engaging on the first read hardly caught my attention and instead I found myself engaged in some parts of the book I hardly remember reading the first time. The part that gripped me this time was the whole sub-plot with Alberto and Sophie, especially towards the end when the whole story becomes ironic.

I was first exposed to irony in the short stories of O' Henry, particularly 'The Gift of the Magi'. The teacher explained it along the lines of God/destiny playing a practical joke. This has been my benchmark definition. But in Sophie's world, the philosopher Alberto mentions at a point in the book that the events that were unfolding was an example of irony which challenged my existing understanding of the idea.

So what really is irony? It's really hard to define. I think there are two ways of looking at it. Firstly, it can be viewed as some kind of information asymmetry, or mutual lack of information that proves critical (for example, the story mentioned above). This is especially true for dramatic irony or tragic irony. On the other hand, as I discovered on reading the book, it can be viewed as something which evokes awareness. This made me look at things which I didn't consider ironic earlier, a bit closer.

I'll use an example - A Dylan song: I shall be free No. 10. The song rambles on about random stuff about some person without much structure or point. Then at the end he sings

Now you're probably wondering by now

Just what this song is all about

What's probably got you baffled more

Is what this thing here is for.

(Referring to a particular kind of strumming he was using throughout the song)

It's nothing

It's something I learned over in England

So, while your listening to the song your going along with the lyrics, figuring out the story, taking everything else for granted. But the moment you hear the last part, you suddenly become aware. You become aware that your listening to a song, that the song has a peculiar strumming style etc. In other words, your attention gets diverted from the content to the medium.

The movie, Annie Hall is another expample. There are scenes when things are happening in the background but Woody Allen converses directly with the audience, breaking out of the story, making the audience aware that they are watching a movie. When the content breaks out or away, that is when the audience becomes aware. In art, any kind of absurdity would also evoke the same reaction and so even the magic realism of Rushdie or Marquez can be looked at as having used irony to highlight the theme of the story.

Douglas Hofstadter, in GEB, refers to this kind of irony as a strange-loop (or self-reference) and claims that once there is self-reference within a system of any kind (mathematical or otherwise), the system has developed its own consciousness. Godel's Incompleteness theorem follows the laws of a mathematical system in the proof and concludes something about the system itself. There is irony in the theorem. But does that make the system that generated the irony conscious? The author claims that a strange-loop is is an attempt by a system to percieve reality outside the confines of its role and hence it constitutes consciousness.

There really is no point to this post. Just a few observations. So I'll leave you with an Escher picture.

In the long run, we are all the Grateful Dead

2008-06-07T10:48:00.004+05:30

Paul Krugman articulates much better what I tried to say in the earlier post. Check it out here.

In other news, I'm done with exams and enjoying the brilliant city of Bangalore for a couple of months.

A certain somebody recently accused me of being a "neo-liberal fascist" on account of my endorsment of free markets. This accusation wasn't entirely baseless. When you break it down, all my arguments require well-functioning markets. Something which, I was made to understand, can't be taken for granted. Although markets evolve into well-funtioning one over time, the collateral damage in terms of poverty and economic hardship requires one to think of stop-gap measures.

So I thought I'd do some reading. I picked up 'Capitalism and Freedom' by Milton Friedman and what I've read so far is the best articulation of the capitalist ideology I've ever seen. Check it out here if you have the time.

P.S: recommendations for things to check out in Bangalore are very welcome :)

The "In Rainbows" model

2008-02-17T17:20:00.004+05:30

This is a response to Navin's comment in the last post. He asked whether Radiohead, providing it's latest album In Rainbows for download, for a voluntary contribution made any economic sense (zero is also an allowed contribution).

A seller, if he is greedy (in the sense that he prefers more to less and given a choice chooses more) typically tries to make as much money as he can by selling. The ideal situation for him is to sell his goods to each person charging them the maximum they are willing to pay. This is what you see in an ordinary auction: the buyer has to bid the maximum he is willing to pay to compete for the auction. In this sense, the seller is a profit-maximizer. In a market when he is selling to a multitude of people however, he does not have the liberty of engaging in any activity to discover the maximum willingness to pay (MWP) of each of his customers. He usually charges a price, aware of the fact that this price may be below some people's MWP and above some other's. So by definition, he caters only to the people who are willing to at least pay the price he quotes.

The seller estimates what the people are willing to pay and contructs a demand curve (relationship between price and how much people buy) and chooses a point such that he makes maximum profit. This is the tradeoff that he has to contend with. If his goods are priced low enough, it is more likely that a larger number of people will have a MWP greater than the price but on each unit he is making less profit.

So how does the radiohead model work? They have spent some money making their new album. By offering it on the internet for a voluntary contribution, the choice is left to the buyer to state his MWP and pay it. The only problem is that their is no mechanism or incentive for the buyer to actually state his MWP (except possibly his conscience). This is where they deviate from predicted seller behaviour. They don't seem to be greedy. This makes for an interesting new point of view. A west versus east kind of thing. What if I get into a business saying that I want to cover my costs and over and above that all I want to do is make x rupees of profit. Then I don't need to worry so much about getting the price just right. Now all I need to do is ensure that so many units of my good gets sold. Then I tell my customers to pay whatever they want to in excess of the cost of making it. Will anybody pay more? They certainly have no incentive to. The only thing that could motivate them is some sense of loyalty to the seller or some kind of personal touch. I would probably pay more at my local shop where I know the proprieter, but not at a McDonald's.

Radiohead however only had this offer on for a limited period, which leads me to believe that they did this more as a publicity gimmick than as some kind of revolutionary business model that's going to change the world. Would it have been a sustainable new model? I don't think so because it ultimately boils down to a quesiton of telling the truth and not many people tell the truth.

I think it's a step in the right direction because the entire music industry is going to change. The recording industry association of america says that the losses from music piracy already runs up to 12.5 billion dollars. That is a lot of people not paying for what they listen to. Fighting piracy is one way to go about it. It's definitely not the smart way though. Work with it. Accept that it has come to the phase where people expect to get music for free and work around it trying to make the most of it.

Think about what google does. Their product for their customers is internet search, and they charge nothing for it. On the other hand they've got their hand in the advertising pie and that's where they make their money. This is similar to the indian railways: charge your customers less and make up for it in another market (freight). I think this is a sustainable model for the music industry. Musicians could start putting their hand in some other pies too. Advertising is an obvious option or they could link up with the producers of CD's and music equipment and have some kind of tie-up there. There may be something more specific that could pay better. An innovation here could be the final answer.

A change of direction

2008-02-09T14:26:00.000+05:30

I've been thinking about some stuff for a while now and since the readership on this blog has grown by leaps and bounds, I think its the right time to try and incorporate some open-ended quesitons here.

Like I mentioned, I don't have the answers and most of the quesitons I post over here are mere thoughts and may seem very naive to the well trained (brainwashed) economist. So I would really appreciate some constructive comments by all of you who spend more than the average 1 minute on this blog to take some time and help me take these germs of ideas to some kind of logical conclusion.

The posts are going to deal with issues like this:

1. The impact of removing the perfect divisibility assumption in consumer theory. If I had to contend with the fact that I cannot buy .67 of a book, how would discrete math help me solve the optimization problem?

2. Considering that demand curves, when empirically estimated, obviously involve many data points that deviate from the mean, can I meaningfully incorporate all the data points by coming up with a theory that talks about a demand correspondence (and its associated features) instead of a precise funtional form?

3. Creating a Bayesian updation framework (after reading up about it!) for a) firm vaulation and b) option pricing.

4. Exploring the possibilities of intermarket relationships from a non-general equilibrium framework. Something a little more game-theoretic I suppose, so as to have a platform to analyse an entire industry with its input markets, its substitute and complement market.

5. Reviewing Econometric methodology. What is the most efficient algorithm to tackle an econometric problem and is there some way to rank the possible approaches in terms of some desirability criteria.

6. Using rational expectations to explain microeconomic dynamics and other movements in individual markets.

So lets see how this goes. I'll tackle each of theses thoughts in separate posts. Please do post comments and help me sort these issues out.

Excuse for a post

2008-02-09T14:22:00.000+05:30