Econometrics Beat: Dave Giles' Blog

Actually Computing the Sample Variance!

2013-05-23T13:31:00.000-07:00

I always enjoy the posts from John Cook on his The Endeavour blog. John's a knowledgable guy and there's a lot on his blog that's of interest to econometricians. Take a look for yourself!

Back in 2008, John had a post that's relevant to something I've been blogging about recently. It also reminded me of some important issues associated with computation - issues that we used to worry about a great deal in the bad old days of "hand calculations", and computers with short word-lengths and very limited memory

One thing that needs to be stressed to students is that the algebraic formulae that they learn about are not necessarily expressed in the form that's most appropriate computationally. By "appropriate", I'm referring to both computational accuracy and computational speed. There are actually lots and lots of examples that illustrate the point that I want to make. However, let's just consider the "simple problem" of computing the variance of a sample of data.

Different estimators of the population variance have been the topic of a couple of my recent posts (here and here). I'll be concentrating here on the usual sample variance formula,

s² = [1 / (n - 1)] Σ[(x_i - x*)²] , (1)

where "n" is the sample size, and x* = (1 / n)Σ(x_i) is the simple sample mean.

However, the focal point of what I'll be talking about is the term, Σ[(x_i - x*)²], so my comments apply in varying degrees to other related measures that we use to estimate the population variance, or to describe the variability of a sample.

So, what's the big deal here? The formula for s² looks straightforward enough. First, you compute the sample average, x*. Then, you take the difference between each sample observation and x*. You square each of these differences. You add up all of the squared difference; and finally, you divide the answer by a number. Nothing to it!

What about the number of steps that were required to do this? What about the possibility of "rounding errors" as you go through the various steps I've outlined?

Are these the steps that are actually followed when you are using a statistical package and you use a command such as:

scalar SAMPVAR = @vars(X) (in EViews)

SAMPVAR = var(X) (in R or in gretl) ?

Before we answer this question, notice that there's more than one way to write the expression for s². For example, a mathematically equivalent formula is:

s² = [1 / (n - 1)] [Σ(x_i²) - nx*²] . (2)

The formula in (1) is sometimes called "the direct method" for computing s², and the formula in (2) is sometimes called the "one pass method". In a world of perfect arithmetic, both methods would yield the same numerical result.

The "one pass method" used to be popular some years ago when we relied on calculators with very limited memory, but no decent statistical package would use it today. The reason is that it's prone to "catastrophic loss of precision" (to borrow a phrase from Braun and Murdoch, 2007, p.11). Those authors show that if applied to the sample {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}, coding the formulae (1) and (2) in R yields the correct answer of 11. However, if you add the value 1.e10 to each sample value, coding formula (2) in R yields the value 0 instead of 11. (Recall that adding a constant doesn't alter the variance, so 11 is still the correct result.)

Not surprisingly, the var( . ) function in R doesn't use the "one pass method"!

In fact, there are other mathematically equivalent, and useful, expressions for s² that you may not have encountered. For instance, we can write:

s² = [1 / (n(n -1))] [nΣ(x_i²) - (Σ(x_i))²] . (3)

John Cook calls this the "sum of squares method".

Then there's the "corrected two pass algorithm", suggested by Chan et al. (1983), and favoured by Press et al. in their various editions of Numerical Recipes:

s² = [1 / (n - 1)] {Σ(x_i -x*)² - (1 / n) [Σ(x_i - x*)]²} . (4)

Another algebraically equivalent formula is:

s² = [1 / (2n(n -1))] ΣΣ(x_i - x_j)² (5)

Now, let's get back to the issue of computational accuracy.

Using a simple simulation experiment, John Cook illustrated that the "sum of squares" formula is hopelessly unreliable. So, we can rule out the formulae in (2) and (3) when it comes to computation. John also found that the "direct method" is very reliable.

In fact, it's essentially as reliable as Welford's (1962) method for computing a variance. The latter method involves a simple algorithm, and is often recommended for the robustness of its computational accuracy (e.g., Knuth, 1997). The algorithm is described in another post by John Cook.

What about the formulae in (4) and (5)? How good are they from a computational perspective?

I'll leave the answer to this question to those of you who'd like to run a little simulation experiment.

So, the take-away message is very simple - just because it's convenient to write a formula in a particular way, algebraically, this doesn't necessarily mean that this representation is the best way to code the calculations. You have to consider computational speed, at least to some extent, and you always need to be thinking about computational accuracy.

References

Braun, W. J. and D. J. Murdoch, 2007. A First Course in Statistical Programming With R, Cambridge University Press, Cambridge.

Chan, T. F., G. H. Golub, and R. J. LeVeque, 1983. Algorithms for computing the sample variance: Analysis and recommendations. American Statistician, 37, 242-247.

Knuth, D. E., 1997. The Art of Computer Programming, Volume 2: Seminumerical Algorithms, 3rd ed., Addison Wesley, Reading, MA.

Press, W. H., S. A. Teukolsky, W. T. Vettering, and B. P. Flannery, 1992. Numerical Recipes in FORTRAN 77: The Art of Scientific Computing, 2nd ed., Cambridge University Press, Cambridge.

Welford, B. P., 1962. Note on a method for calculating corrected sums of squares and products. Technometrics, 4, 419-420.

Minimum MSE Estimation of a Regression Model

2013-05-22T12:23:00.001-07:00

Students of econometrics encounter the Gauss-Markhov Theorem (GMT) at a fairly early stage - even if they don't see a formal proof to begin with. This theorem deals with a particular property of the OLS estimator of the coefficient vector, β, in the following linear regression model:

y = Xβ + ε ; ε ~ [0 , σ²I_n] ,

where X is (n x k), non-random, and of rank k.

The GMT states that among all linear estimators of β that are also unbiased estimators, the OLS estimator of β is most efficient. That is, OLS is the BLU estimator for β.

A "linear" estimator is simply one that can be written as a linear function of the random sample data. Here, these are the observed values of the elements of y. We can write the OLS estimator of β as b = (X'X)^-1X'y = Ay; where A = (X'X)^-1X' is a non-random matrix. So, each element of b is a linear combination of the elements of y, with weights that aren't random. It's a linear estimator.

Notice that the GMT holds without having to assume that the errors in the model are Normally distributed. (If they do happen to be Normal, then the OLS estimator of β is "Best Unbiased" - that is, we don't have to restrict ourselves to the family of linear estimators.) Also, if ε has a non-scalar (but known) covariance matrix, then the GLS estimator of β is BLU.

When I introduce students to the GMT I usually emphasize that it's a result that's of only limited interest. There are lots of interesting and important estimators that aren't linear estimators. Moreover, why on earth would we want to constrain ourselves to considering only estimators that are unbiased? Econometricians use non-linear and biased estimators all of the time. For example, most Instrumental Variables and GMM estimators fall into this category.

Putting this important point aside for the moment, it's also interesting to ask students to derive the linear minimum MSE estimator of β in the above model. That is, think of the family of estimators of β that are of the form, b = Ay, and determine what choice of A leads to the estimator with the smallest MSE. Strictly speaking, as b and β are vectors, we need to think of the MSE matrices, defined as MSE = V + (Bias Bias'), where V is the covariance matrix of the estimator and Bias is the (k x 1) bias vector. This quantity can be made scalar by taking trace(MSE).

If you want to try and answer this question, you need to know a bit of matrix differential calculus - specifically, you need to know how to differentiate functions of matrices with respect to the elements of a matrix. There are plenty of books on this, but the free download from Steven Nydick is all you need for this particular problem.

However, to make life a little easier, but without altering the message, we can simplify the model to the case where k = 1:

y_i = βx_i + ε_i ; ε_i ~ i.i.d. [0 , σ²]

and consider the family of estimators,

β* = [a₁y₁ + a₂y₂ + ......... + a_ny_n] ,

where the a_iweights are non-random.

Immediately, E[β*] = Σ(a_iβx_i) = βΣ(a_ix_i).

So, Bias[β*] = E[β*] - β = β[Σ(a_ix_i) - 1].

In addition, given the i.i.d. assumption for the errors,

Var.[β*] = Σ(a_i²Var.(y_i)) = σ²Σ(a_i²) .
So, M = MSE[β*] = σ²Σ(a_i²) + β²[Σ(a_ix_i) - 1]².

If we differentiate M, partially, with respect to each of the a_js, and set these derivatives to zero, we get:

2σ²a_j + 2β²x_j[Σ(a_ix_i)] = 2β²x_j ; j = 1, 2, ....., n. (1)

Then, dividing by 2 and multiplying each side of each of these "n" equations, (1), by x_j, we get:

σ²a_jx_j + β²x_j²[Σ(a_ix_i)] = β²x_j^{2 ; j = 1, 2, ....., n .}

Now, sum over all j:

σ²Σ(a_jx_j) + β²Σ(x_j²)Σ(a_ix_i) = β²Σ(x_j²) . (2)

Similarly, dividing each equation in (1) by 2, multiplying each side by y_j, and summing over all j, we get:

σ²Σ(a_jy_j) + β²Σ(x_jy_j)Σ(a_ix_i) = β²Σ(x_jy_j). (3)

Notice that (3) can be re-written as:

σ²β* + β²Σ(x_jy_j)Σ(a_ix_i) = β²Σ(x_jy_j) . (4)

It's now a simple matter to solve equations (2) and (4) for Σ(a_ix_i) and β*. The solution for the latter is:

β* = {β² / [(σ² / Σ(x_i²)) + β²] } b ,

where b = [Σ(x_iy_i) / Σ(x_i²)] is the OLS estimator of β.

There are three important things to notice about β*:

It isn't really an "estimator" because it's a function of the unknown parameters, β and σ². It can't actually be used!
|β*| < |b| . This MMSE "estimator" of β "shrinks" the OLS estimator towards towards the origin.
The "estimator", β* is both non-linear and biased.

Non-linear, biased, shrinkage estimators - ones that are genuine estimators and don't involve the unknown parameters - are often used in regression analysis. Examples are the Stein, James-Stein, Ridge, and Bayes estimators. The last of these can be especially appealing, as Bayes estimators allow us to shrink the OLS estimator towards a point that reflects our prior beliefs - not necessarily towards the origin.

So, we can now see why trying to "free up" one of the conditions associated with the Gauss-Markhov Theorem, by considering estimators that are linear, but not necessarily unbiased, really doesn't lead us very far if we have in mind that we want to minimize MSE. This also serves as another example of a situation where the MMSE estimator isn't feasible (computable).

EViews Tutorials

2013-05-22T08:35:00.001-07:00

If you're a student who is just learning to use the EViews econometrics package, the tutorials that IHS (the supplier of EViews) has made available should be very helpful. You'll find them here.

There are 13 tutorials at this time, ranging from "EViews basics" to "Forecasting".

"The tutorials are split into self-contained sessions, although we recommend that new users of EViews work their way through the tutorials one by one.

Each tutorial is accompanied by data files so that you may follow the tutorials in your own copy of EViews. The data files are available in the Supporting Files side bar of each tutorial. Each tutorial is available in Microsoft Powerpoint® format, along with the data files, bundled together in a Zip file, in the Download Package area of of the side bar of each tutorial.

You should note that the tutorials are written based on EViews 8, however the vast majority of material covered in them is applicable to earlier versions of EViews too."

Certainly, these tutorial won't tell you everything you'll want to know, but they're a good start.

Variance Estimators That Minimize MSE

2013-05-21T16:08:00.002-07:00

In this post I'm going to look at alternative estimators for the variance of a population. The following discussion builds on a recent post, and once again it's really directed at students. Well, for the most part.

Actually, some of the results relating to populations that are non-Normal probably won't be familiar to a lot of readers. In fact, I can't think of a reference for where these results have been assembled in this way previously. So, I think there's some novelty here. But we'll get to that in due course.

I can just imagine you smacking your lips in anticipation!
To get things started, let's suppose that we're using simple random sampling to get our n data-points, and that this sample is being drawn from a population that's Normal, with a (finite) mean of μ and a (finite) variance of σ². Let x* denote the sample average: x* = (1 / n)Σx_i, where the range of summation here (and everywhere below) is from 1 to n.

Let's consider the family of estimators of σ²:

s_k² = (1 / k)Σ[(x_i - x*)²],

where "k" is a positive number. Now, to be very clear, I'm not suggesting that we should necessarily restrict our attention to estimators that happen to be in this family - especially when we move away from the case where the population is Normal. I'll come back to this point towards the end of this post.

Clearly, if k = (n - 1), we just have the usual unbiased estimator for σ², which for simplicity we'll call s². If k = n, we have the mean squared deviation of the sample, s_n², which is a downward-biased estimator of σ².

However, we all know that unbiasedness isn't everything!

Often, we look at our potential estimators and evaluate them in the context of some sort of loss function. If this loss function is quadratic, then the expected loss (or "risk") of an estimator is its Mean Squared Error (MSE). You'll recall that the MSE of an estimator is just the sum of its variance and the square of its bias.

Let's compare the unbiased estimator, s², and the biased estimator, s_n², in terms of MSE.

Because we're using simple random sampling from a Normal population, we know that the statistic c = [(n - 1)s² / σ²] follows a Chi-square distribution with (n - 1) degrees of freedom. So, s² = (σ² / (n - 1))χ²_{(n - 1)}. We also know that the mean of a Chi-square random variable equals its degrees of freedom; and its variance is twice those degrees of freedom. So, E[s²] = σ², and Var.(s²) = 2σ⁴ / (n - 1).

The first of these two results also holds if the population is non-Normal, but the second result doesn't hold, as I discussed in this earlier post.

Next, noting that s_n² = (n - 1)s² / n, it follows that;

E[s_n²] = [(n - 1) / n]σ² ;
Bias[s_n²] = E[s_n²] - σ² = - (σ²/ n)

and
Var.[s_n²] = 2σ⁴(n - 1) / n².

So, the MSE of s_n² is given by the expression,

MSE(s_n²) = Var.[s_n²] + (Bias[s_n²])² = σ⁴ (2n - 1) / n².

Because s² is unbiased, its MSE is just its variance, so MSE(s²) = 2σ⁴ / (n - 1). Noting that MSE(s_n²) = [(n - 1) / n] MSE(s²) - (σ⁴ / n²), we see immediately that MSE(s_n²) < MSE(s²), for any finite sample size, n. This can be seen in the following chart, drawn for σ²= 1. (Of course, the two estimators, and their MSEs coincide when the sample size is infinitely large.)

Although s_n² dominates s², in terms of having smaller MSE for all possible sample values, and all (finite) sample sizes, it's still not the best we can do within the family of estimators we're considering! Let's go back to this class of estimators and ask, "what value of k will lead to the estimator with the smallest possible MSE for all members of this class?"

We can answer this question by using a bit of simple calculus. We'll write out the expression for the MSE of s_k², and it will be some function of "k". Then we'll differentiate this function with respect to "k", set the derivative to zero, and then solve for the value of k (say k*). We should then check the sign of the second derivative to make sure that k* actually minimizes the MSE, rather than maximizes it!

So, here goes ........

Notice that we can write a typical member of our family of estimators as

s_k² = (1 / k)Σ[(x_i - x*)²] = [(n - 1) / k]s² .

So, using the results that E[s²] = σ², and Var.(s²) = 2σ⁴ / (n - 1), we get:

E[s_k²] = [(n - 1) / k]σ² ;
Bias[s_k²] = E[s_k²] - σ² = [(n - 1 - k) / k]σ²;
and
Var.[s_k²] = 2σ⁴(n - 1) / k².

The MSE of s_k² is given by the expression,

M = MSE(s_k²) = Var.[s_k²] + (Bias[s_k²])² = (σ⁴ /k²)[2(n - 1) + (n - 1 - k)²].

Differentiating M with respect to "k", and setting this derivative to zero, yields the solution, k* = (n + 1). You can easily check that k* minimizes (not maximizes) M.

So, within this family that we've been considering, the minimum MSE (MMSE) estimator of σ² is the estimator,

s_n+1² = (1 / (n + 1))Σ[(x_i - x*)²] .

This is certainly a well-known result. It also extends naturally to the situation where we're estimating the variance of the (Normally distributed) error term in a linear regression model. In that case the MMSE of this variance is (1 / (n - p + 2))Σe_i², where e_i is the i^th OLS residual, and p is the number of coefficients in the model.

So far, so good!

Now, let's connect with the earlier post that I mentioned above, and see how all of this works out if we have a population that's non-Normal. We'll retain the simple random sampling, though. As was discussed in that post, in general the variance of s² is given by:

Var.[s²] = (1 / n)[μ₄ - (n - 3)μ₂² / (n - 1)] = V, say.

Here, μ₂ and μ₄ are the second and fourth central moments of the population distribution. Recall that μ₂ is the population variance, and for the result immediately above to hold the first four moments of the distribution must exist.

Let's extend this variance expression to members of the family, s_k². Then we'll work out the expression for the MSE of such estimators for a non-normal population. Finally, this will allow us to derive the MMSE estimator in this family for any population distribution - not just for the Normal population that we dealt with earlier in this post.

Once again, we'll begin by using the fact that we can write:

s_k² = (1 / k)Σ[(x_i - x*)²] = [(n - 1) / k]s² .

The estimator, s², is still unbiased for σ² even in the non-Normal case, so we still have the results:

E[s_k²] = [(n - 1) / k]σ² ; and Bias[s_k²] = [(n - 1 - k) / k]σ²= [(n - 1 - k) / k]μ₂

as before.

Also,

Var.[s_k²] = [(n - 1) / k]² Var.[s²] = [(n - 1) / k]²(1 / n)[μ₄ - (n - 3)μ₂² / (n - 1)] ,

and so the MSE of s_k² is given by:

MSE(s_k²) = [(n - 1 - k) / k]²μ₂² + V[(n - 1) / k]²,

where V = Var.[s²] is defined above.

Going through the calculus once again, it's easy to show (I used to hate that statement in textbooks) that the value of "k" for which the MSE is minimized is:

k** = (n - 1)(V + μ₂²) / μ₂²,

where V = (1 / n)[μ₄ - (n - 3)μ₂² / (n - 1)].

Check: For the Normal distribution, μ₄ = 3μ₂², and so k** = (n + 1) = k*, as before.

Having gone to all of this effort, let's finish up by illustrating the optimal k** values for a small selection of other population distributions:

Uniform, continuous on [a , b]

μ₂ = (b - a)²/ 12 ; μ₄ = (9 / 5)
k** = (n - 2) + (3 / n) +1296(n - 1) / [5n(b - a)⁴]

Standard Student's-t, with v degrees of freedom

μ₂ = v / (v - 2) ; μ₄ = 3v² / [(v - 2)(v - 4)]
k** = (n - 2) + (3 / n) +3(n - 1)(v - 2) /[n(v - 4)] ; for v > 4

χ², with v degrees of freedom

μ₂ = 2v ; μ₄ = 3(v + 4) / v
k** = (n - 2) + (3 / n) +3(n - 1)(v + 4)] / (4nv³)

Exponential, with mean θ

μ₂ = θ² ; μ₄ = 9θ⁴
k** = (n² + 7n - 6) / n

Poisson, with parameter λ

μ₂ = λ ; μ₄ = λ(3λ + 1)
k** = (n - 2) + (3 / n) +(n - 1)(3λ + 1) / (nλ³)

Now, a final word of caution.

Yes, setting k = k** in the case of each of these non-Normal populations, and then estimating the variance by using the statistic, s_k² = (1 / k)Σ[(x_i - x*)²], will ensure that your "estimator" is MMSE within this particular family of estimators. However, this doesn't mean to say that it's the "best", or even a feasible, estimator to use.

For instance, consider the last example where the population is Poisson. In that case, the population mean and variance are both λ. The MLE for λ is the sample average, x*. Therefore, x* is also the MLE for the population variance. Actually, x* is the "minimum variance unbiased" (MVUE) estimator of λ. The statistic s² is also an unbiased estimator of λ, but it is inefficient relative to x*. So x* dominates s² in terms of MSE. See here for a nice discussion.

If we were to try and implement our MMSE estimator of the variance in this case, we'd be trying to estimate λ. However, k** is a function of λ. We'd need to know the population variance in order to obtain the MMSE estimator of that parameter! You can see that the same issue applies to the Student's-t and χ² examples given above but it's not an issue with the other two examples.

Sometimes, MMSE estimators simply aren't "feasible". They're functions of the unknown parameters we're trying to estimate. They're not really estimators, at all!

Camp(s) Econometrics

2013-05-19T14:09:00.003-07:00

The New York Camp Econometrics VIII was held in Bolton Landing, NY, last month. I recall Badi Baltagi (one of the Camp Econometrics organisers) telling me about this great annual event a few years ago. The Texas Econometrics 2013 was held in Lost Pines back in February. This was the 18^th Camp for the group in Texas.

I also seem to recall that there used to be another regular Camp Econometrics in Southern California some years ago. If my neurons are still firing in the right order, I believe that Denis Aigner was one of the leaders of that venture.

Back to the NY Camp:

"This event is a gathering of econometricians and empirical economists whose successful goal is to: (1) Bring together a group of econometricians/empirical economists and guests of host universities to discuss issues in econometrics, both applied and theoretical; (2) Present papers for comments by participants; (3) Stimulate student interest in econometrics; (4) Help students develop their technical presentation skills by encouraging the students of host universities to participate in the meetings and present papers."

Events like the Camp(s) Econometrics, and The Econometric Game, in the Netherlands, really are great ventures!

Cookbook Econometrics - Reprise

2013-05-18T20:28:00.001-07:00

A few days ago I was looking at my copy of Econometric Foundations, written by Ron Mittelhammer, George Judge, and Doug Miller. It's an excellent book, by the way.

I noticed, for the first time, that on p.xxviii of the Preface they have the following to say, under the heading of "A Comment":

"Many view econometrics as a potpourri or bag of tricks, and the cookbook metaphor for econometrics textbooks has become commonplace. Unfortunately, this philosophy can produce analysts who know a list of econometric recipes but who have insufficient understanding of which techniques to apply in a given situation or how to interpret the results of an application properly. As the inventory of econometric procedures has grown, the importance of understanding when it is appropriate to apply each econometric procedure, as well as knowing the appropriate interpretation of the results, has grown more than proportionately. The number of reference works that describe the growing inventory has expanded pari passu. These reference works will be accessible to the well-trained analyst who has mastered the basic philosophy and principles on which econometrics is founded. However, analysts who have done little more than memorize the recipes in conceptual econometric cookbooks will find the growing literature on new econometric methods impenetrable. Our goal is for you to be able to determine or create the econometric procedures that are applicable to your problem and then be able to apply them empirically and interpret the results appropriately. This is what this book is about, and this is what we think modern graduate econometrics instruction should be about."

I don't know Doug Miller personally, but I do know and respect Ron and George, and so I wasn't surprised to find myself punching the air and ~~shouting~~ saying, "YES!"

I had similar things to say in a post couple of years ago, and I won't repeat them all here.

Yes, I enjoy helping readers by providing "how-to-do-it" posts (such as this one on testing for Granger causality). Judging by the feedback, these posts appear to be widely appreciated - thanks! However, I'm just as committed to fostering a clear understanding of:

The assumptions that are needed to get our standard econometric results
The connections between many of the "tools" that we use
How to approach problems we haven't met before - in a sensible way
Why some econometric tools work better than others
The history and intuition that provide the foundation for much of what we do in econometrics
~~Motherhood and apple pie~~
The crucial importance of data quality

O.K. - enough harping on for now!

I Know What You Did Last Summer!

2013-05-18T07:50:00.002-07:00

O.K., I know that I stole that title! It was absolutely blatant.

The other day, some colleagues and I were discussing the issue of students (including our own offspring), and their summer jobs - or no jobs, as the case may be. I'm not passing judgement here in what follows, by the way.

When I was a student I had to earn enough money over the summer to live off for the rest of the year. That's just the way it was. Period! My parents were great - I could live at home over the summer at no cost to me. But that was it. They lived in a very small rural town in New Zealand, a long way from where I attended university. The upside of this was that, being a rural area (in the mid/late 1960's through to the early 1970's), there were some seriously good work opportunities.

So, what did I do each summer?

Well, I did lots of different things, actually. They ranged from career-related, to "I don't care how tough this is - I just need the bucks!"

Let's look at some of these jobs. They were interesting, to say the least:

Summers of '66, '67 & '68: Sheep-shearing gang & harvesting hay and grain
Summer of '69: Actuarial office of a major life insurance company
Summer of '70: Research Section of the N.Z. central bank - the Reserve Bank of New Zealand

In retrospect, I was very lucky. I got to experience some work situations that were way outside what most students get to see.

Didn't get to Woodstock - not because I was making hay, but because I was in class - remember, I was in the Southern Hemisphere. The "Summer of Love" was my "Winter of Woe".
Decided that an actuarial life wasn't for me - financially, a bad decision!
Got to be one of 3 people working on the construction of the first version of the Reserve Bank's macroeconometric model. You'd be right if you guessed that I found that really exciting!

That last summer job led to a scholarship for my Masters-level studies, and ultimately to a very rewarding period at the RBNZ before I opted for an academic life in Australia.

You just never know where that summer job is going to lead to!

What's the Variance of a Sample Variance?

2013-05-17T09:44:00.001-07:00

This post is really pitched at students who are taking a course or two in introductory economic statistics. It relates to a couple of estimators of the variance of a population that we all meet in such courses - plus another one that you might not have met. In addition, I'll be emphasising the fact that some "standard" results depend crucially on certain assumptions. Not surprisingly - but not always made clear by instructors and text books.

To begin with, let's consider a standard problem. We have a population that is Normal, with a mean of μ and a variance of σ². We take a sample of size n, using simple random sampling. Then we form the simple arithmetic mean of the sample values: x* = Σx_i , where the range of summation (here and everywhere below) is from 1 to n.

Under my assumptions, we know that the sampling distribution of x* is N[μ , (σ² / n)]. The normality of the sampling distribution follows from the Normality of the population, and the fact that x* is a linear function of the data. The variance of the sampling distribution stated above is correct only because simple random sampling has been used.

Now, let's get to what I'm really interested in here - estimating σ². We all learn that the mean squared deviation of the sample, σ^*2 = (1 / n)Σ[(x_i - x*)²], is a (downward-) biased estimator of σ². If we allow for the fact that we've actually lost one degree of freedom by estimating μ using x*, then an unbiased estimator of σ² is s² = (1 / (n - 1))Σ[(x_i - x*)²].

O.K., now what does the sampling distribution of s² look like?

Well, under the assumptions I've made, including the Normality of the population, s² has a sampling distribution that is proportional to a Chi-square distribution. More specifically, the statistic, c = [(n - 1)s² / σ²] is Chi-square with (n - 1) degrees of freedom.

[As an aside, s² and x* are independently distributed if and only if the population is Normal. The "only if part" of the latter statement is due to the Irish statistician, Geary - see here.]

So, we now know something, indirectly, about the sampling distribution of s², and we know that E[s²] = σ². What is the variance of σ²?

Because we're assuming a Normal population, implying that the statistic I've called "c" follows a Chi-square distribution, we can use the result that the variance of a Chi-square random variable equals twice its degrees of freedom.

Re-arranging the formula for "c", we can write: s² = cσ² / (n - 1).

Then, Var.(s²) = {[σ² / (n - 1)]²Var.(c)} = {[σ⁴ / (n - 1)²]2(n - 1)} = 2σ⁴ / (n -1).

[As another aside, the mean of a Chi-square random variable equals its degrees of freedom, so applying this result to "c" and re-arranging, we immediately get the result that E[s²] = σ². However, we know this already, and this result holds even if the data are non-Normal.]

Now, this is as far as things usually go in an introductory economic statistics course. To sum up:

E[s²] = σ²
c = [(n - 1)s² / σ²] ~ χ²_{(n - 1)}
Var.[s²] = 2σ⁴ / (n -1)

Notice that Var.(s²) vanishes when n grows very large. This, together with the first above, implies that s² is a (mean-square) consistent estimator of σ².

Unfortunately, students often don't realize that the second and third of these results rely on both simple random sampling and the Normality of the population.

A thoughtful student will notice that the first result holds even if the data are non-Normal, and will ask, "what's the variance of s² if the population isn't Normal?" That's a good question!

To answer it, let's introduce an important concept - the "moments" of a probability distribution. Let X be a random variable. Then E[X^k] is called the k^th "raw moment" (or, moment about zero) of the distribution of X. (Here, "k" is a positive integer, but more generally we can allow k to be negative, or a fraction.) Let's denote the k^th such moment by μ'_k. So, the first raw moment is just the population mean. That is, μ'₁ = μ.

Then consider the quantities, μ_k = E[(X - μ)^k], for k = 1, 2, 3,.......... We call these the "centered moments" of the distribution of X. You'll notice that μ₂ is just the population variance. The third and fourth centered moments are used (together with μ₂) to construct measures of skewness and kurtosis, but that's another story.

By the way, there's an important detail. The expectations involved in the construction of the moments require forming an integral. If that integral diverges, the corresponding moment isn't defined. or instance, the k^th moment for a Student's-t distribution with v degrees of freedom exists only if v > k. In the case of the Cauchy distribution (which is just a Student's-t distribution with v = 1), none of the moments exist!

Alright - back to the question in hand! What is the variance of s² if the population is non-Normal? The answer, in the case of simple random sampling, is:

Var.(s²) = (1 / n)[μ₄ - μ₂²(n - 3) / (n -1)] .

If the population is Normal, then μ₄ = 3σ⁴, and μ₂² = σ⁴. So, we get Var.(s²) = 2σ⁴ / (n - 1), in this case.

Notice that this more general expression for Var.(s²) also vanishes as n grows. So, a pair of sufficient conditions for the mean-square consistency of s² (as an estimator of σ²) is:

The data are obtained using simple random sampling; &
At least the first 4 moments of the population distribution exist.

We can easily work out the expressions for Var.(s²) in the case where the population follows some other distributions that you may have heard about. Here are just a few illustrative results:

Uniform, continuous on [a , b]

μ₂ = (b - a)²/ 12 ; μ₄ = (9 / 5)
Var.(s²) = [1296(n - 1) - 5(n - 3)(b - a)⁴] / [720n(n - 1)]

Standard Student's-t, with v degrees of freedom

μ₂ = v / (v - 2) ; μ₄ = 3v² / [(v - 2)(v - 4)]
Var.(s²) = [2v²(2v - 5)] / [n(v - 2)²(v - 4)] ; for v > 4

χ², with v degrees of freedom

μ₂ = 2v ; μ₄ = 3(v + 4) / v
Var.(s²) = [3(n - 1)(v + 4) - 4(n - 3)v³] / [n(n - 1)v]

Exponential, with mean θ

μ₂ = θ² ; μ₄ = 9θ⁴
Var.(s²) = [2(4n - 3)θ] / [n(n - 1)]

Poisson, with parameter λ

μ₂ = λ ; μ₄ = λ(3λ + 1)
Var.(s²) = 2λ² / (n - 1) + (λ / n)

Keep in mind that in each of the cases, the sampling distribution of c = [(n - 1)s² / σ²] will no longer be a χ² distribution! Given our assumption of simple random sampling, you should be able to convince yourself that the asymptotic sampling distribution of "c" will be Normal.

References

Cho, E. & M. J. Cho, 2008. Variance of sample variance. Proceedings of the 2008 Joint Statistical Meetings, Section on Survey Research Methods, American Statistical Association, Washington DC,1291-1293.

Geary, R. C. (1936). The distribution of the Student's ratio for the non-normal samples. Supplement to the Journal of the Royal Statistical Society, 3, 178-184.

Top Economics Blogs

2013-05-15T19:41:00.001-07:00

Nice to be on the list - thanks, INOMICS!

What's Your Favourite Estimator?

2013-05-12T14:57:00.000-07:00

It's interesting to dwell on the popularity of different estimators that econometricians use. Some estimators are "in vogue" for a period, and then give way to others as new developments come along. Different topics have captured the attention of theoreticians and practitioners alike at different times in history.

Here's a Google Ngram showing the extent to which some familiar estimators for simultaneous equations models have been mentioned in books since 1960:

Not too surprisingly, good old OLS just goes on and on:

I was going to include the GMM estimator in these plots, but this acronym has meanings other than the obvious one that comes to mind. So, the results would have been misleading. To be safe, let's use the full phrase Generalized Method of Moments and allow for case sensitivity:

Interestingly, the phrase appeared in some books before the publication of Hansen's classic 1982 paper.

Flowers for Mom - From Quandl

2013-05-12T09:58:00.000-07:00

Today being Mothers' Day in many parts of the world, I thought that flowers would be appropriate. Well, a price index for (Gardens, Plants, and) Flowers. Specifically, a harmonized price index for these goods for 27 European Union countries.

I retrieved the monthly data for the period January 2006 to March 2013 from Quandl.com - a really nice resource that I posted about recently. As well as downloading the data in various formats, reading the data from R, etc., you can also embed an interactive chart of the data directly into a document such as this one, and make the data visible to viewers. (Just click on the graph.) For instance:

As this price is not seasonally adjusted, but obviously has a pronounced seasonal pattern, I thought I'd use this post to illustrate the basics of seasonal adjustment. What I'm going to do is take you through the steps associated with a bare-bones version of the ratio-to-moving-average method. This is something that I show students if I'm teaching an introductory descriptive economic statistics course.

Then, I'll show you how close the results are to the ones that you get if you seasonally adjust the data using the full-blown Census X-13 method that is employed by most statistical agencies world-wide.

Let the price index at time 't' be denoted by P_t, and let's assume that this series is made up of the product of trend (T_t), cycle (C_t), and irregular(I_t) components. (In fact, when I use the X13 method and allow it to decide whether the series is multiplicative or additive in its components, it selects the multiplicative model I'm using here.)

So, taking natural logarithms, we have:

ln(P_t) = ln(T_t) + ln(C_t) + ln(I_t) .

The rudimentary ratio-to-moving-average method involves the following steps. The column labels refer to the associated Excel workbook and csv file that are available on the data page of this blog:

First, we take an unweighted 12-month arithmetic moving average of the ln(P_t) data. (Column D.) Averaging the data smooths the series. A 12-month average should smooth out the seasonal movements, as well as any irregular movements.
There's a slight problem - it's to do with the "timing" of the observations in Column D. It arises because there's an even number (12) of months in a year. Conceptually, the first figures in Column D ( 4.48255) should be located half way between the June and July months if it's to be at the middle of the year. Right now, it's half a month out of alignment. This problem also arises if we're seasonally adjusting quarterly time-series data, because 4 is an even number too. (There's also an even number of weeks in a year - but I digress!)
To re-align the data, we now take an unweighted 2-period arithmetic moving average of the numbers in Column D. The results appear in Column E. Averaging two "out-of-alignment" numbers shifts them by half a month, and bingo, they're now lined up with the appropriate dates! We call the resulting series the "Centered Moving Average".
If we've done things properly there should be 6 values "missing" at the start of the series in Column E, and 6 missing at the end. (Two and two, if we had quarterly data.)
What have we achieved by this? Well, the data in Column E represent what is left of the ln(P_t) series after we've smoothed away the Seasonal and Irregular components. That is, they represent the combined ln(Trend) and ln(Cycle) components.
Next, we subtract the Centered Moving Average series (Column E) from the ln(P_t) data in Column C. This gives us (in Column F) the ln(Seasonal) and ln(Irregular) components.
Then, we take the arithmetic mean of all of the July month values in Column F. This gives us a single, common "seasonal factor" for that month. We then do the same with all of the August month values in that column, and so on. The results appear in Column G. Notice that we're implicitly assuming that the seasonal factors are going to be stable over time.
We're nearly there! If we add up the 12 seasonal factors they should sum to zero over the full year. Seasonality, by definition, is an intra-year phenomenon. Let's CHECK if we have this result. You can see in the workbook that they actually sum to 0.00178098. Not bad, but not good enough!
We apportion this discrepancy across the 12 seasonal factors by subtracting (0.00178098/12) from each of the numbers in Column G. The results, which are the final seasonal factors, appear in Column H. Notice that these factors are repeated, year after year.
We can now seasonally adjust the ln(P_t) series. We subtract these seasonal factors from the data in Column C. The results appear in Column I.
Taking the exponential of the Column I series gives us the seasonally adjusted series for P_t itself, as in Column J.

geometric averages

arithmetic

averages

dividing

subtracting

Ratio

moving

average

Next, I used Eviews to apply the X-13 seasonal adjustment method to P_t. Once you are viewing the series, you select "Proc", and then "Seasonal Adjustment", and go from there. Here's the original price index and its seasonally adjusted counterpart:

The EViews workfile that I used is on the code page for this blog. In that file there are actually three seasonally adjusted versions of the price index. The series called "X13" will be self-explanatory; the series called "Manual" was obtained using the basic steps outlined above, as shown in the Excel workbook; and the series called "RMA" was obtained using the "moving average" option under the "seasonal adjustment" procedure in EViews. The latter series should be almost identical to "Manual", and indeed it is. More on this below.

Now, how similar is my rudimentary seasonally adjusted series, "Manual", to the one obtained using X-13? Here's a scatter-plot of the two series - it's virtually a 45-degree line.

This is confirmed by the following OLS regression, and simple correlations:

You can see that there is almost a perfect correlation between my seasonally adjusted series, and both "RMA" and "X13":

The rudimentary ratio-to-moving average seasonal adjustment procedure that I went through above isn't always going to provide results that are this close to the ones that you get when you use X-13. There are several reasons for this:

X-13 can allow for outliers in the data. Our basic method ignores this possibility.
X-13 can allow for a seasonal pattern that "evolves" over the cycle, or over time. Our basic method assumes a stable seasonal pattern.
X-13 can allow for the fact that different months have different numbers of "trading days", and for "holiday effects", such as the moving dates for Easter. These effects are ignored in our basic method.
X-13 can deal with "end-point" effects that arise at the beginning and end of the sample, where values can't be computed for the centered moving averages. Our basic method doesn't take this into account.

However, in many cases, very similar seasonally adjusted series are obtained. This is very comforting for those of us who teach this material. You can take students through the rudimentary steps that I've outlined, and they can generate very convincing seasonally adjusted time-series.

Oh yes - don't forget those flowers for Mothers' Day!

New Paper Published

2013-05-10T14:12:00.000-07:00

A paper of mine appears in the latest issue of the Chilean Journal of Statistics. The paper is titled, "Exact asymptotic goodness-of-fit testing for discrete circular data with applications.

I've posted previously about this general topic, here, here and here.

R is His Friend

2013-05-09T10:14:00.000-07:00

Marcus Beck has a nice (& relatively new) blog called R is My Friend. You can guess that his posts relate to the use of R.

I particularly liked his piece on the use of the XML package in R to mine data from the internet; and his post on using the integrate function in R, even when the anti derivative has no closed-form solution.

Grad. student readers will also like his post, How Long is the Average Dissertation.

My own take on a related question can be found here.

Robust Standard Errors for Nonlinear Models

2013-05-08T11:52:00.002-07:00

André Richter wrote to me from Germany, commenting on the reporting of robust standard errors in the context of nonlinear models such as Logit and Probit. He said he 'd been led to believe that this doesn't make much sense. I told him that I agree, and that this is another of my "pet peeves"!

Yes, I do get grumpy about some of the things I see so-called "applied econometricians" doing all of the time. For instance, see my Gripe of the Day post back in 2011. Sometimes I feel as if I could produce a post with title almost every day!

Anyway, let's get back to André's point.

The following facts are widely known (e.g., check any recent edition of Greene' text) and it's hard to believe that anyone could get through a grad. level course in econometrics and not be aware of them:

In the case of a linear regression model, heteroskedastic errors render the OLS estimator, b, of the coefficient vector, beta, inefficient. However, this estimator is still unbiased and weakly consistent.
In this same linear model, and still using OLS, the usual estimator of the covariance matrix of b is an inconsistent estimators of the true covariance matrix of b. Consequently, if the standard errors of the elements of b are computed in the usual way, they will inconsistent estimators of the true standard deviations of the elements of b.
For this reason,we often use White's "heteroskedasticity consistent" estimator for the covariance matrix of b, if the presence of heteroskedastic errors is suspected.
This covariance estimator is still consistent, even if the errors are actually homoskedastic.
In the case of the linear regression model, this makes sense. Whether the errors are homoskedastic or heteroskedastic, both the OLS coefficient estimators and White's standard errors are consistent.

Moreover, in the case of a model that is nonlinear in the parameters:

The MLE of the parameter vector is biased and inconsistent if the errors are heteroskedastic (unless the likelihood function is modified to correctly take into account the precise form of heteroskedasticity).
This stands in stark contrast to the situation above, for the linear model.
The MLE of the asymptotic covariance matrix of the MLE of the parameter vector is also inconsistent, as in the case of the linear model.
Obvious examples of this are Logit and Probit models, which are nonlinear in the parameters, and are usually estimated by MLE.

I've made this point in at least one previous post. The results relating to nonlinear models are really well-known, and this is why it's extremely important to test for model mis-specification (such as heteroskedasticity) when estimating models such as Logit, Probit, Tobit, etc. Then, if need be, the model can be modified to take the heteroskedasticity into account before we estimate the parameters. For more information on such tests, and the associated references, see this page on my professional website.

Unfortunately, it's unusual to see "applied econometricians" pay any attention to this! They tend to just do one of two things. They either

use Logit or Probit, but report the "heteroskedasticity-consistent" standard errors that their favourite econometrics package conveniently (but misleading) computes for them. This involves a covariance estimator along the lines of White's "sandwich estimator". Or, they
estimate a "linear probability model" (i.e., just use OLS, even though the dependent variable is a binary dummy variable, and report the "het.-consistent standard errors".

If they follow approach 2, these folks defend themselves by saying that "you get essentially the same estimated marginal effects if you use OLS as opposed to Probit or Logit." I've said my piece about this attitude previously (here, here, here, and here), and I won't go over it again here.

My concern right now is with approach 1 above.

The "robust" standard errors are being reported to cover the possibility that the model's errors may be heteroskedastic. But if that's the case, the parameter estimates are inconsistent. What use is a consistent standard error when the point estimate is inconsistent? Not much!!

This point is laid out pretty clearly in Greene (2012, pp. 692-693), for example. Here's what he has to say:

"...the probit (Q-) maximum likelihood estimator is not consistent in the presence of any form of heteroscedasticity, unmeasured heterogeneity, omitted variables (even if they are orthogonal to the included ones), nonlinearity of the form of the index, or an error in the distributional assumption [ with some narrow exceptions as described by Ruud (198)]. Thus, in almost any case, the sandwich estimator provides an appropriate asymptotic covariance matrix for an estimator that is biased in an unknown direction." (My underlining; DG.) "White raises this issue explicitly, although it seems to receive very little attention in the literature.".........."His very useful result is that if the QMLE converges to a probability limit, then the sandwich estimator can, under certain circumstances, be used to estimate the asymptotic covariance matrix of that estimator. But there is no guarantee the the QMLE will converge to anything interesting or useful. Simply computing a robust covariance matrix for an otherwise inconsistent estimator does not give it redemption. Consequently, the virtue of a robust covariance matrix in this setting is unclear."

Back on July 2006, on the R Help feed, Robert Duval had this to say:

"This discussion leads to another point which is more subtle, but more important...

You can always get Huber-White (a.k.a robust) estimators of the standard errors even in non-linear models like the logistic regression. However, if you believe your errors do not satisfy the standard assumptions of the model, then you should not be running that model as this might lead to biased parameter estimates.

For instance, in the linear regression model you have consistent parameter estimates independently of whether the errors are heteroskedastic or not. However, in the case of non-linear models it is usually the case that heteroskedasticity will lead to biased parameter estimates (unless you fix it explicitly somehow).

Stata is famous for providing Huber-White std. errors in most of their regression estimates, whether linear or non-linear. But this is nonsensical in the non-linear models since in these cases you would be consistently estimating the standard errors of inconsistent parameters.

This point and potential solutions to this problem is nicely discussed in Wooldrige's Econometric Analysis of Cross Section and Panel Data."

Amen to that!

Regrettably, it's not just STATA that encourages questionable practices in this respect. These same options are also available in EViews, for example.

Reference

Greene, W. H., 2012. Econometric Analysis. Prentice Hall, Upper Saddle River, NJ.

Turn on the Economy

2013-05-07T23:22:00.000-07:00

"Turn on the economy". That's one of the invitations issued to (web) visitors to the museum of New Zealand's central bank - The Reserve Bank of New Zealand. Accepting this invitation will allow you to see a virtual version of Bill Phillips' famous MONIAC computer at work, and to "play" with the economy yourself.

Doesn't that appeal to you?

Bill Phillips - "the Indiana Jones of Economics" - gave us "the Phillips Curve", of course. However, the MONIAC computer was a revolutionary device that Bill used to demonstrate economic stabilization policy.

If you happen to be visiting New Zealand's capital city - Wellington - you can visit the Bank's (physical) museum, and see the MONIAC "in the flesh".

The Indiana Jones of Economics

2013-05-07T20:59:00.002-07:00

All students of economics have heard about The Phillips Curve in one of its forms or another. The Phillips Curve is named after A. W. H. (Bill) Phillips, a remarkable New Zealander who made a number of fundamental contributions. His work, undertaken largely at the London School of Economics, dealt with stabilization policy, and modelling in continuous time, to name just two topics.

Bill Phillips was quite a character, and his varied life has been amply documented in various places. His entry in Wikipedia is a useful starting point, and the memorial piece written in 1978 by one of my former teachers, Brian Easton, is also a "must read" item.

Some time ago, I wrote about Bill in a post titled, "A Moniacal Economist", in reference to his famous MONIAC machines. These were hydraulic analogue computers that could be used to demonstrate the workings of the macro-economy.

In February of this year, the BBC Radio aired a piece about Bill Phillips, titled "The India Jones of Economics". In this 14-minute broadcast, Tim Hartford provides an interesting commentary of Bill's life and contributions to our discipline. You can download the broadcast - it's Episode 4, 6 February 2013 - from here.

More about Bill Phillips at a later date..............

My Recent Reading

2013-05-06T16:10:00.000-07:00

Here are some of the papers that I have been reading in the past few days:

Majid M. Al-Sadoon, 2013. Geometric and long run aspects of Granger causality. Discussion Paper, Barcelona Graduate School of Economics.
David Ardia & Lennart Hoogerheide, 2013. GARCH models for daily stock returns: Impact of estimation frequency on value-at-risk and expected shortfall forecasts. Tinbergen Institute Discussion Paper.
Otilia Boldia & Alastair R. Hall, 2013. Estimation and inference in unstable nonlinear least squares models. Journal of Econometrics, 172, 158-167.
Kazuhito Higa, 2013. Estimating upward bias in the Japanese CPI using Engel's law. Working Paper, Hitotsubashi University.
Anna Mikusheva, 2013. Survey on statistical inferences in weakly identified instrumental variables models. Applied Econometrics, 29, 117-131.

Econometrics Lectures on YouTube

2013-05-06T15:42:00.005-07:00

I'm always keeping my eyes open for new or different resources that I can integrate into my Economic Statistics and Econometrics courses. For example, as I've mentioned before in previous posts (here and here), I've been really pleased with what I've been able to achieve with Wolfram's cdf files.

In my undergrad. Statistical Inference course I also refer the students to some of the excellent mini-lectures by Keith Bower. I find his presentations to be clear and (very importantly) accurate.

If you check out YouTube you'll find a number of video presentations relating to the teaching of econometrics. To be honest, many of them don't particularly impress me. Maybe I;m just hard to please!

There are some exceptions to this, though, including David Hendry's 20111 lecture on Teaching Undergraduate Economics at Oxford, and the great series of videos of Mark Thoma in action in the classroom.

Burgernomics

2013-05-06T14:32:00.001-07:00

Looking through the papers that are "in press" at Economics Letters today, I came across a paper by Anthony Landry, titled "Borders and Big Macs". (The link is to the working paper version.) Here's the abstract:

"I provide new estimates of border frictions for 14 countries using local, national, and international Big Mac prices. I ﬁnd that borders generally introduce only small price wedges, far smaller than those observed across New York City neighboring locations."

This led me to wonder just how many academic papers have been written using the well-known "Big Mac Index" (BMI) that's published annually by The Economist magazine. I don't know the exact answer, but there are 39 listed on RePEc's IDEAS site.

That's a lot of burgers!

A Visual Proof That OLS is BLU

2013-05-06T09:48:00.000-07:00

Back in the day (as they say), we had monochrome monitors on our P.C.'s. Do you remember the ghastly green or weird amber colours? Then, one bright day everything became multi-coloured! This is not just me reminiscing - this is leading up to an innovative proof of the Gauss-Markhov Theorem. Honestly!

In a post yesterday, I mentioned Ken White's sense of humour - that's Ken White, "The SHAZAM Man", as my kids used to affectionately call him. On one of his many visits in the late 1980's, Ken offered to give a talk to a group of students about using the SHAZAM econometrics package. (We had no money for software at the time, but thanks to Ken's outstanding generosity we always had the latest version of his package for everyone to use.)

There were (and probably still are) some hidden tricks in SHAZAM. For instance, if you entered the command "USER KEN", then all sorts of extra things were available to you. For instance, the subsequent commands became case-sensitive. Another command that wasn't widely known was "SET COLOR", which could be rescinded with "SET NOCOLOR". With this option toggled on, the background colour on the monitor changed as certain SHAZAM commands were executed.

I must admit, this got a little hard on the eyes after a while, which might account for my declining vision! However, you have to remember that we didn't have the internet to amuse us, and it really didn't take too much to get us excited - technologically speaking! So, if you issued the command, "READ X", to indicate that a series of data was to be read, the background colour on the monitor turned red.

So, Ken got started on his promo-talk about SHAZAM. The students got right into it. Then he said, "You know, the great thing about SHAZAM is that unlike the competing packages, you can actually use it to prove some important theorems." Now the class got visibly excited!

"For instance", said Ken. "Let's prove the Gauss-Markhov Theorem". He keyed in the command OLS Y X. As the results began to appear on the screen, the background colour of the monitor turned very bright blue. in response to the use of the OLS command. The colour was so bad that you could hardly see the results.

Dead-pan, Ken said, "And that, my friends, is how you prove that the OLS estimator is BLU!"

The Frequent Regressor Club

2013-05-05T12:07:00.002-07:00

My friend, Ken White, developed the SHAZAM econometrics package in 1977. Ken's a funny guy - that's to say, he has a great sense of humour.

On one of his many visits to Christchurch, New Zealand (when I was living there, many years ago) he gave me a wooden die that he'd had an artisan carve at the local Arts Centre in Chistchurch. On each of the six faces he'd had the guy put the name of an econometrics/statistics package - TSP, LIMDEP, GAUSS, RATS, and SHAZAM. Yes, I know that's only 5 names. The thing was, SHAZAM appeared on two of the faces! The idea was to roll the die to decide which package to use in your lab. class. I still have the die - much to the occasional bemusement of students who see it on my desk.

Ken also kindly gave me a very early edition of the Captain Marvel comic - the one where the meaning of the acronym, SHAZAM, is revealed to the readers. That comic is still in my office too. There's going to be a heck of a garage sale in the department when I retire!

Some of you will recall the manuals that went with earlier versions of SHAZAM. There was an "Appendix" at the back, titled Frequent Regressor Club. At that time, Ken travelled A LOT. He was an avid collector of frequent flyer points with United Airlines. He had so many that he hardly knew what to do with them. This was the inspiration for the Frequent Regressor Club.

Readers of the manual were told that they could collect points for every OLS regression that they ran. They just had to mail a copy of the printout to Ken. (This was pre-PC days.) Nonlinear regressions earned extra points; and step-wise regressions weren't allowed!

If you earned enough points you could apparently get a free upgrade of SHAZAM. It was just a spoof, but Ken was constantly stunned by the number of people who sent him their printouts!

Those were the days!

Granger Causality Testing Done Properly

2013-05-04T08:31:00.002-07:00

I enjoy following David Stern's Stochastic Trend blog. David is Research Director at the Crawford School of Public Policy at the Australian National University. He's an energy and environmental economist who does some really interesting work - not my field at all, but I always enjoy reading what he has to say.

In his latest blog post, David links to a recent paper that he's co-authored with Robert Kaufman, from Boston University. The paper is titled, "Robust Granger Causality Testing of the Effect of Natural and Anthropogenic Radiative Forcings on Global Temperature".

As I said, this isn't my field. However, if you want to see an example of Granger causality testing done well, you should take a look at this well-written paper.

Nice one!

When Will the Adjusted R-Squared Increase?

2013-05-03T20:37:00.000-07:00

The coefficient of determination (R²) and t-statistics have been the subjects of two of my posts in recent days (here and here). There's another related result that a lot of students don't seem to get taught. This one is to do with the behaviour of the "adjusted" R² when variables are added to or deleted from an OLS regression model.

We all know, and it's trivial to prove, that the addition of any variable to such a regression model cannot decrease the R² value. In fact, R² will increase with such an addition to the model in general, and it will stay the same if the added variable is totally uncorrelated with the other variables in the model. Conversely, deleting any regressor from an OLS regression model will reduce the value of R².

Indeed, this is precisely why various "adjusted" R² measures have been suggested over the years. You can boost the goodness-of-fit of the model by throwing anything into the regression, whether it makes economic sense or not.

The adjusted R² that we typically use involves "correcting" both the numerator and denominator sums of squares (in the usual R² formula) for the appropriate degrees of freedom. If the sample size is "n", and the model includes "k" regressors (including the intercept) this adjusted R² can be expressed as:

R_A² = 1 - [(n - 1) / (n - k)][1 - R²] ,

and it can be shown that R_A² ≤ R² ≤ 1. The adjusted R² can take negative values, and this will occur if and only if R² ≤ [(k - 1) / (n - 1)].

Now, what can we say about the behaviour of R_A² if we add regressors or delete them from the model? The adjusted R² may increase or decrease (or stay the same) when we do this, and there are some simple conditions that determine which will occur.

The first result is that adding a regressor will increase (decrease) R_A² depending on whether the absolute value of the t-statistic associated with that regressor is greater (less) than one in value. R_A² is unchanged if that absolute t-statistic is exactly equal to one.

If you drop a regressor from the model, the converse of the above result applies. Dropping a regressor amounts to imposing a (zero) restriction on its coefficient. If you square the t-statistic, you get an F-statistic, and it's exactly the F-statistic for testing if the single linear restriction is valid. Not surprisingly, then, there's a more general result than the one given above - one that applies to a situation where several regressors are simultaneously added to or dropped from the model.

Adding a group of regressors to the model will increase (decrease) R_A² depending on whether the F-statistic for testing that there coefficients are all zero is greater (less) than one in value. R_A² is unchanged if that F-statistic is exactly equal to one.

So, you can increase the adjusted coefficient of determination by adding regressors that are statistically insignifcant, but the situation isn't quite as bad as with the usual (uadjusted) R².

Finally, the second result given above generalizes to the case where we are considering any set of linear restrictions on the regression coefficients - not just zero restrictions.

In summary, and not too surprisingly, the behaviour of the adjusted coefficient of determination as we add or delete regressors is quite systematic. If you're a student who's hoping that deleting a regressor with a t-statistic of 1.1 will increase the value of R_A², think again!

Mark Thoma on "Replication"

2013-05-03T10:25:00.000-07:00

Yesterday, in his Economist's View blog, Mark Thoma discussed the importance of replicating results in empirical economics. He's absolutely right, of course.

I'll leave you to read what had to say, but I especially liked his closing passage:

"One place where replication occurs regularly is assignments in graduate classes. I routinely ask students to replicate papers as part of their coursework. Even if they don't find explicit errors (and most of the time they don't), it almost always raises good questions about the research (why this choice, this model, what if you relax this assumption, there's a better way to do this,here's the next question to ask, etc., etc.). So replication does occur routinely in economics, and it is very valuable, but it is not a formal part of the profession the way it should be, and much of the replication is done by people (students) who generally assume that if they can't replicate something, it is probably their error. We have a lot of work to do on the replication front, and I want to encourage efforts like this."

At least one of my colleagues also assigns replication exercises in this way, and I really should do the same. Fortunately, more journals are either recommending or requiring that data-sets be made available as a condition of publication. The Journal of Applied Econometrics is one such journal, and we've recently been pushing in that direction with the Journal of International Trade & Economic Development.

This should become part of our culture.

When Can Regression Coefficients Change Sign?

2013-05-03T09:51:00.003-07:00

Let's suppose that you've been running regressions happily all morning. It's sunny day, but what could be better than enjoying some honest-to-goodness econometrics? Suddenly, you notice that one of the estimated coefficients in your model has a sign that's the opposite to what you were expecting (from your vast knowledge of the underlying economics). Shock! Horror!

Well. it's really good that you're on the look-out for that sort of thing. Congratulations! However, something has to be done about this problem.

Being young, with good eyesight, you also happen to spot something else that's interesting. One of the other estimated coefficients has a very low t-statistic. You have a brilliant idea! If you delete the variable associated with the very small t-value, maybe the "wrong" sign on the first coefficient will be reversed. Is this possible?

Sadly, no, it's not going to be that simple.

Let's make sure that what I'm talking about is quite clear. Here are the results for a hypothetical regression, estimated by OLS, and with t-statistics in parentheses:

y = 0.43 + 1.45X₁ - 0.89X₂ + residual ; n = 34 ; R² = 0.89
(1.45) (0.66) (-1.83)

You were expecting a positive coefficient for X₂. If you drop X₁ from the regression, and re-estimate the model by OLS, could the sign of the coefficient of X₂ become positive? No - that's impossible.

Leamer (1975) proved that such a change in sign cannot occur if the absolute value of the t-statistic for the variable you're deleting is less than the absolute value of the t-statistic for the variable whose sign you're interested in. That's the situation we have in the example above.

On the other hand, if we were to delete X₂ from the above model, its possible for the sign of the coefficient for X₁ to change.

A number of extensions/generalizations of this result are also available, including:

Leamer's necessary condition, stated above, was extended to include a sufficient condition by Visco (1978).
Leamer's result was re-stated in a somewhat simpler form by Oksanen (1987).
Leamer's result was generalized by McAleer et al. (1986) to apply to cases where the deleted variables are combined in arbitrary linear combinations. See, also, Visco (1988).
All of the above results were shown by Gikles (1989) to hold if the model is estimated by any Instrumental Variables estimator, rather than by OLS.

This last extension is based on the algebraic relationship between the OLS and (generalized) I.V. estimators, and the results on restricted I.V. estimation given by Giles (1982).

By the way - HT to a former student of mine, Darren Gibbs, who asked some interesting questions that led me to write the 1989 paper referenced below. I never teach a course without learning something new!

References

Giles, D. E. A., 1982. Instrumental variables estimation with linear restrictions. Sankhya: The Indian Journal of Statistics, B, 44, 343-350.

Giles, D.E.A., 1989. Coefficient sign changes when restricting regression models under instrumental variables Estimation. Oxford Bulletin of Economics and Statistics, 51, 465-467.

Leamer, E. E., 1975. A result on the sign of restricted least-squares estimates. Journal of Econometrics, 3, 387-390.

McAleer, M., A. Pagan, & I. Visco, 1986. A further result on the sign of restricted least-squares estimates. Journal of Econometrics, 32, 287-290.

Oksanen, E. H., 1987. On sign changes upon deletion of a variable in linear regression analysis. Oxford Bulletin of Economics and Statistics, 49, 227-229.

Visco, I., 1978. On obtaining the right sign of a coefficient estimate by omitting a variable from the regression. Journal of Econometrics, 7, 115-117.

Visco, I., 1988. Again on sign changes upon deletion of a variable from a linear regression. Oxford Bulletin of Economics and Statistics, 50, 225-227.