My pet-peeve almost made the list at number 39.

Lack of statistical significance indicates that there is not enough evidence to reject the null hypothesis of no effect, so inferences based on the direction of nonstatistically significant coefficients should be offered cautiously or not at all.

I would elaborate the point and say that researchers should not interpret insignificant coefficients as "no effect" either. I've talked about that plenty, so I'll just point non-regular readers to this post and this paper.

Point 17 seems a little weird.

Hypotheses should be numbered consecutively, such as H₁, H₂, and H₃, instead of being named with number-letter combinations, such as H_1a, H_1b, and H₂, because number-letter combinations foster confusion: H₂ is the third hypothesis in the aforementioned example.

I can't imagine that the number-letter combinations always foster more confusion that than simple numbering. (Though I did once read a paper with an H₁, H_1a, and H_2, but no  H_1b. That strategy left me scratching my head.) The reader won't remember numbers or number-letter combinations, so it seems like we should just get rid of numbering hypotheses altogether. If a hypothesis isn't important enough to have a descriptive name, perhaps it's not worth the reader's time.

Point 57 really stepped on my toes.

The manuscript should not contain errors in grammar, spelling, or punctuation; these errors indicate that the writing of the manuscript was not conducted carefully and suggests that the reported research might not have been conducted carefully, either.

I find typos in papers that I've edited dozens of times. It never ends.

In particular, my friend had a hypothesis that the variance of the latent outcome (commonly called "y-star") should increase with an explanatory variable of interest. He was using the heteroskedastic probit model, which looks something like $Pr(y_i = 1) = \Phi(X_i\beta, e^{Z_i\gamma})$, where $\Phi()$ is the cumulative normal with mean $X_i\beta$ and $e^{Z_i\gamma}$.

He wanted to argue that his explanatory variable increased both the mean function ($X\beta$) and the variance function ($e^{Z\gamma}$). To do this, he included his variable in both the $X$ and $Z$ matrices and tested the statistical significance of the associated coefficients. He found that they were both significant. It would seem that his variance increases the mean and the variance of the latent outcome. He wanted to know if this was good evidence for his theory.

I replied that I did not think so, because a binary outcome variable doesn't contain any direct information about a non-constant variance. Indeed, the variance of a Bernoulli random variable is tied directly to the probability of success. This implies that any inference about changes in $e^{Z\gamma}$ must come from observed changes in the probability of a success (i.e. changes in the mean function). Because we've assumed a specific (i.e. linear) functional form for the mean function, deviations from this will be attributed to the variance function. Because of this structure, the results are driven totally by our assumption of the linearity of the mean function. Indeed, it would not be hard to find a plausible non-linear mean function (e.g. quadratic specification) that makes the $\gamma$ parameter no longer significant.

Example One

I thought a good way to illustrate this claim would be to show that for a large but plausible sample size of one million, the heteroskedastic probit will suggest a non-constant variance when the relationship is simply a logit.

To see an illustration of this, start by simulating data from a simple logit model. Then estimate a regular probit and a heteroskedastic probit.

n <- 10^6
x <- runif(n)
y <- rbinom(n, 1, plogis(-4 + 3*x))

r1 <- glm(y~x,family=binomial(link="probit"))

library(glmx)
h1 <- hetglm(y ~ x)

Now if the coefficient for x is significant in the model of the scale, then we should conclude there is heteroskedasticity, right? No, because we already know that the latent variance is constant. However, we've barely mis-specified the link function (we're using a probit, the true model is logit). This slight mis-specification causes the results to point toward non-constant variance.

Coefficients (binomial model with probit link):
             Estimate Std. Error z value Pr(>|z|)
(Intercept) -2.090026   0.007269  -287.5   <2e-16 ***
x            1.589261   0.007418   214.2   <2e-16 ***

Latent scale model coefficients (with log link):
  Estimate Std. Error z value Pr(>|z|)
x -0.20780    0.01475  -14.09   <2e-16 ***

Here is a plot of the predicted probabilities from the true, probit, and heteroskedastic probit models. Notice that in the range of the data, the heteroskedastic probit does a great job of representing the relationship. However, that's not because the variance is non-constant as the heteroskedastic probit would suggest. It's because the link function is slightly mis-specified.

Example Two

I think the logit example makes the point powerfully, but let's look at a second example just for kicks. This time, let's say that we believe there's heteroskedasticity that can be accounted for by x, so we estimate a heteroskedastic probit and include x in the mean and variance function. However, again we're wrong. Actually the true model has a constant variance, but a non-linear mean function ($\beta_0 + \beta_1x^2$).

If we simulate the data and estimate the model, we see again that our mis-specified mean function leads us to conclude that the variance is non-constant. In this case though, the mis-specification is severe enough that you'll find significant results with much smaller sample. If we made conclusions about the non-constant variance from the statistical significance of coefficients in the model of the variance, then we would be led astray.

Coefficients (binomial model with probit link):
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  -3.6122     0.3726  -9.694   <2e-16 ***
x             3.1493     0.2809  11.210   <2e-16 ***

Latent scale model coefficients (with log link):
  Estimate Std. Error z value Pr(>|z|)
x  -0.8194     0.2055  -3.988 6.66e-05 ***

Again, the plot shows that the heteroskedastic probit does a good job at adjusting for the mis-specified mean function (working much like a non-parametric model).

So What Should You Do?

I think that researchers who have a theory that allows them to speculate about the mean and variance of a latent variable should go ahead and estimate a statistical model that maps cleanly onto their theory (like the heteroskedastic probit). However, these researchers should realize that this model does not allow them to distinguish between non-constant variance and a mis-specified mean function.

The formatting is similar to the old site, but I've gotten rid of the trashier elements, like social media buttons and comments. The site had become bulky and difficult to navigate, but I think it's a little nicer now.

You'll need to know a little bit about WordPress to make it work for your site, but if I can get it working, I'm sure you can too.

My reaction best belongs in the comment section of his blog, but I want to use some equations and I know that I can make them work here. [In hindsight, this was a bad idea. I wrote this post on the day Justin published his, but just got LaTeX back working on my blog, through a redesign of the whole site.] So first, hop over and read his nuanced discussion and then come back and read my coarse reaction and questions.

The claim I find most interesting is his conclusion from the simulation study.

So, what can we conclude? First, a small magnitude but statistically significant result contains virtually no important information. I think lots of political scientists sort-of intuitively recognize this fact, but seeing it in black and white really underscores that these sorts of results aren’t (by themselves) all that scientifically meaningful. Second, even a large magnitude, statistically significant result is not especially convincing on its own. To be blunt, even though such a result moves our posterior probabilities a lot, if we’re starting from a basis of skepticism no single result is going to be adequate to convince us otherwise.

The key question is how to update our posterior belief about the effect being zero or non-zero in light of statistical significance. This is an interesting idea to me, because I recently defended statistical significance as a useful way to quickly summarize empirical results (compared to a confidence interval).

Justin's Idea

Here's Justin's idea. What is the posterior probability that the effect is zero, given that it is statistically significant? Of course this assumes that we put some prior mass on exactly no effect. Justin suggests that a skeptical social scientist might believe that the null hypothesis (of exactly no effect) is true with probability 0.9. (I don't care for mass priors at zero and I don't think Justin does either, it just happens to be convenient here to make a point.) Then the posterior probability that the null is true is given by the equation below.

$Pr(\beta = 0 | Sig.) = \dfrac{Pr(Sig. | \beta = 0)Pr(\beta = 0)}{Pr(Sig. | \beta = 0)Pr(\beta = 0) + Pr(Sig. | \beta \neq 0)Pr(\beta \neq 0)}$

As I noted above, Justin views 0.9 as a useful prior probability for the null. For simplicity, we can consider the limiting case--a sample so large (or an effect so big) that we always find significance when the effect is not zero. In this limiting case, we know that the probability of significance given the null is false is one. We also know, by construction, that the probability of significance, given the null is true, is 0.05. We can just plug this information into the equation above.

$Pr(\beta = 0 | Sig.) = \dfrac{0.05 \times 0.9}{0.05 \times 0.9 + 1 \times 0.1} \approx 0.31$

That is, even with huge sample size and statistical significance, we only think there is a 31% chance that the null is false. That is a counter-intuitive result.

But why is this? When I first saw Justin's simulations, I was a little puzzled and had to run the code myself and work through the analytic stuff to believe the result.

Like other tricky Bayes' rule problems (e.g. Monty Hall problem, boy/girl problem), you have to be careful about the information contained in the likelihood. Here, the likelihood doesn't contain any information about the relative likelihood of the observed data under the null except that it is relatively unlikely. This means that extremely large estimates are treated the same as large estimates. To see this, note that once we're rejecting almost always, it doesn't matter how large the effect is.

Okay, I understand why the posterior probability doesn't go to zero. But why 31%? Shouldn't it settle a little closer to zero? It settles well away from zero because the amount of information in the likelihood is capped. The best thing we can observe is statistical significance. Because this happens fairly often when the null is true (5% of the time), and we strongly believe the null is true (90% of the time), statistical significance doesn't help us much.

Now that I've worked through that bit in my own mind, I see that Justin is making a powerful point--statistical significance doesn't contain that much information.

Remaining Questions

The main question I still have is about the prior. I don't believe that the probability that the null is true is 0.9. I think it is zero. So then why is this prior useful?

Instead of placing a mass at zero, my skeptical prior places a strongly informative prior about zero, say a normal distribution with mean zero and standard deviation 0.1. Then I would hypothesize about the sign of the coefficient. How would this change the argument?

I think this would be more inline with Justin's simulation when the prior probability of the null is 0.5. The math I think is quite similar for a continuous prior centered at zero and the discrete prior that placed 0.5 at zero. Working through the math would require some complicated integration or simulation. Nonetheless, I'll fearlessly intuit the answer. First, when one uses a continuous prior centered at zero, the prior used matters much less than when a discrete prior is used. Second, the resulting posterior allows us to be much more skeptical about the null. In short, I don't think Justin's results would hold for my skeptical, but continuous prior centered about zero.

P.S. While I was fixing up my website to display the equations, Justin posted this, which gets at my questions, but doesn't leave me completely satisfied. I'll think more about it and discuss it in a future post.

You will never be dumber than you are right now. You will also never have more time than you do right now. Thus, you have a relative abundance of time and a relative dearth of knowledge. How do we strike a balance between these resources to optimally leverage them for learning?

He argues that, in some instances, we pick up new skills at the point of need. Other times, we learn skills well beforehand in anticipation of need. Matt thinks that just-in-time learning is undervalued.

To answer the question we started with, I think that we need to place more value on just-in-time learning and less on just-in-case learning.

I tend to agree, but the two are not independent. For example, just-in-case training in probability theory allows more efficient just-in-time learning of specific statistical models.

I think this has some application to how we train graduate students in political methodology.

I was taught the details of various statistical models well before I needed them. For example, I learned a lot about duration models, although I've never applied beyond the practice sets. I've also watched Gary King's lectures and he structures his class similarly.

I think political scientists should consider an alternative strategy for educating graduate students. Rather than teach students the details of many statistical models, we could focus more on probability theory. Not probability theory at the expense of specific models, but probability theory in addition to specific models. With a stronger background in probability theory, students can better understand models they already know, explain their inferences to others, and efficiency learn or derive new statistical models just-in-time.

When I took "maximum likelihood," for example, the class spent one day studying MLE and then moved on to specific applications (e.g. logit and probit). That balance doesn't seem quite right to me.

I think political scientists could give up a little breadth in the number of models that students learn and gain some depth in the foundations of probability theory. This sacrifices some just-in-case learning of statistical models for just-in-case learning of probability theory. Which would help students more?

Today, I'd like to briefly argue that these hypotheses are indeed common.

When first learning statistics as both an undergraduate and graduate student (and even in some of my advanced classes in the statistics department!), I was taught that "null results" were a bad thing. As political scientists, we were supposed to look for effects. Much to my surprise, the very first academic presentation I ever attended, the speaker presented a hypothesis of no effect. This was not some hypothesis off to the side--it was central to the empirical argument.

I've been noticing researchers  hypothesizing "no effect" ever since, even in good journals, but I wanted to do a more rigorous study. I gathered all the empirical research articles published in 2011 and 2012 in both the American Political Science Review and the American Journal of Political Science that present explicit hypotheses. Of these articles in the best journals, 30% (18/61) presented a hypothesis of no meaningful effect!

Unfortunately, the authors do not make strong empirical arguments for their hypotheses. Each uses the lack of statistical significance (i.e. insignificance) as evidence for their claim. However, the absence of evidence for an effect does not imply no effect and it does not imply the effect is not meaningful.

These hypotheses are both important and common. Political scientists should start thinking carefully about how we should evaluate these claims.

This recruitment season, though, I have a concrete recommendation: David Allen's Getting Things Done (GTD).

If you want to learn a little more before jumping in, you might find this or this collection of resources helpful.

I'm not usually a big fan of self-help books, but this book is fantastic and totally changed the way I handle everything from student e-mails to writing my dissertation (and I'm not alone). It helped me become more productive, efficient, and creative. All hopeful academics should read this book.

A panelist could not get his carefully prepared PowerPoint presentation to work. He had no other notes. There were a lot of people in the audience. At first, he was clearly frazzled by the idea of not having slides or notes. But he just started talking to the audience about his research. It was one of the more engaging talks that I've seen at a conference.

Even though he had prepared graphs for the audience, the presentation went really well without graphs. I love and expect graphs, but they were not necessary.

I think the lesson for presenters is this. Talk to your audience. If you want to design slides that supplement what you're saying, that's fine. But don't let talking to the audience become reading slides.

1. Each presenter talks. No questions or discussion. The begins audience to lose interest.
2. The discussant talks for a minute or two about the general themes of the panel, trying in vain to connect the unrelated papers. The audience is not aroused.
3. The discussant directs a series of technical suggestions to each presenter, boring the audience to sleep.
4. Now the bored, sleeping audience is asked to discuss the papers.

It doesn't have to be that way. A recent "series of unfortunate events" imposed a different structure on a panel I attended and it worked much better.

I rolled out of bed early on Friday morning in Orlando to attend a panel out of my field because my friends Jacob Ausderan and Nick Nicoletti were presenting. I expected their presentations to be good and I expected to be able to offer some valuable feedback.

By a series of seeming accidents, the panel was quite a bit different than planned.

• At least one person missed the panel, leaving only three presenters.
• The discussant had an emergency and couldn't make it, so he provided written comments rather than verbal.
• I accidentally instituted a norm of discussing each paper after the immediately presentation. (I just wanted to see a graph before Jacob exited his slideshow.)

At a consequence, the panel had the following form:

1. Each presenter gave a presentation, then controlled a brief discussion session.
2. There was no feeble attempt to connect the papers, except when it came up naturally in the discussion.
3. There were no technical suggestions given needlessly to the audience. These were e-mailed to the presenters instead.

This had a very positive impact on the quality of panel and the discussion.

• Each presenter got thirty minutes for presenting and discussing. The longer they presented, the shorter we discussed, giving an incentive to keep things brief.
• The presenters controlled the discussion. It helps the flow to have a person standing at the front of the room to direct the question toward.
• The presenters got lots of questions. Somehow questions that seem relevant at the end of a talk seem stale after other presentations and discussants' comments. The discussions still had plenty of momentum when they had to be cut off for time's sake.
• There were no discussant comments. While these are helpful to the presenters, they are usually boring to the audience. Saving this time for discussion made the panel much more pleasant.

In defense of hard to learn statistical tools, i.e. #rstats prsm.tc/gyTBRK <- pretty funny 'who uses what software' at the end.

— JD Long (@CMastication) January 3, 2013

@prisonrodeo oh how I'd love to be done with "I USE R I'M BETTER THAN U" writing/thinking

— Brenton Kenkel (@brentonk) January 5, 2013

While I don't think this type of post is particularly useful, it is fun (especially the John Myles White line), so I'm writing up my thoughts on the issue.

For better or worse, I think the software one uses certainly sends a signal.

I've heard others apply the same arguments to typesetting programs. LaTeX and Beamer, for example, are said to send a "technically competent" signal compared to Word and PowerPoint. For better or worse, I think I am vulnerable to these signals, although I don't use Beamer.

R and Stata are the software packages that I run into most often in political science, and I certainly have stereotypes of their users, but it is a matter of style rather than competence. (These are just my stereotypes.)

• R users are more likely to be interested in graphics and simulation (i.e. Gelman and Hill). Also, R users are more likely to care about statistical programming. This is how I first became an R user. The first paper that I wrote as a graduate student required a lot of simulation and a few custom graphs, and I needed to do a little programming to get these right. I think programming is a powerful tool and that graphics and simulation are really important in communicating results. Because I think R users are more likely to emphasize these things, I update upward slightly on R users. That said, there are plenty of terrible methodologists that use R.
• Stata users are much more interested in estimating fancier econometric models (i.e. Cameron and Trivedi). These users put less emphasis on model checking methods such as cross-validation and value complicated models (e.g. bivariate probit with partial observability) more than R users. Since I like model checking and think complicated models are over-used (or at least over trusted), I tend to update downward slightly on Stata users. That said, there are plenty of great methodologists that rely on Stata.

I don't run into users of other software much in political science, but I do in the statistics department. (Again, these are just my stereotypes.)

• Matlab users work on more theoretical problems. By that I mean building and evaluating new estimators and methods, not proving theorems.
• SAS users care about analyzing data. They work on real-world problems, probably for a drug company.

I use both R and Stata.

I rely mostly on R in my research. I occasionally use Stata for two purposes.

1. Recoding data. Whenever I work with huge chucks of (especially survey) data, Stata offers a really useful set of commands for cleaning up the data.
2. Maximizing a difficult likelihood. Sometimes I'll have a custom model and regular optimization algorithms (e.g BFGS) fail. In this situation, I use a little magic that is found in Stata's ", difficult" option. I don't quite understand why it works so well, but it is relentless. It is the single best feature of Stata.

I don't update much on users' competence.

While I do update on the methodological style of software users, I don't think I update much (if at all) on their competence. Here are some statements from Taylor's post that I disagree with.

• "When you don’t have to code your own estimators, you probably won’t understand what you’re doing." I think that many people don't code their on estimators (and couldn't easily start), but understand what they are doing. I also think that plenty of people who do code their own estimators have no clue.
  
• "When operating software doesn't require a lot of training, users of that software are likely to be poorly trained." I'm sure that researchers who don't want to learn statistics are much less likely to want to learn software beyond point-and-click, but I think that most people who are using any software and writing about it to the public are not "poorly trained."
• "Researchers who care about statistics enough should have gravitated toward R at some point." I spent three years in the statistics department at Florida State. People over there care about statistics and most use something other than R. I've also met plenty of political scientists who care about statistics and use Stata exclusively. I do think that those who care about certain styles of analysis (e.g. graphs, simulations, and programming) are likely to be drawn to R, but I don't think it's universal.