Toban Wiebe

Quantifying uncertainty in probability predictions

2019-02-24T13:00:00+00:00

Suppose you’re interested in knowing the chances of an event $X$ occuring (e.g., $X =$ “a nuclear strike over any populated area in the year 2019”). When making predictions about events with binary outcomes (either the event happens or it doesn’t), people generally report a single probability (e.g., a 2% chance of $X$ occurring). But, you may wonder, why not report a confidence interval (e.g., 2% $\pm$ 0.5%), or a distribution of probabilities (e.g., a Beta distribution) to reflect uncertainty?

For example, this question comes up with prediction markets, where the market price can be interpreted as the best estimate of the probability of the event $X$ occuring. But there are no confidence intervals on this market price. Or consider models for classification, such as logistic regression or other machine learning algorithms, which produce predicted probabilities for each possible class (e.g., $X_i =$ “transaction $i$ is fraudulent”). In both of these cases, we face the same issue with representing uncertainty — how does the market/model express confidence in its predicted probabilities?

In this post, I’ll explain why this question stems from a fundamental confusion: it’s a misconception to think that a predicted probability is a point estimate that doesn’t convey any uncertainty. Below, I’ll show that there are two distinct sources of uncertainty that are being conflated here, and that one or both can be used to express uncertainty.

Two types of uncertainty

The key distinction here is between:

Uncertainty over the outcome, $X$ vs $\neg X$
- Also known as aleatoric uncertainty
- (FYI: the symbol “$\neg$” is the negation operator and can be read as “not”)
Uncertainty over model parameters which are used to generate a prediction for the outcome
- Also known as epistemic uncertainty

Let’s unpack each case in depth.

Uncertainty over outcomes

When we aren’t working with a model, we only have the first source of uncertainty to deal with. But it isn’t obvious where the uncertainty lies: if we say $Pr[X] = 0.02$, it may appear that we’ve just given a point estimate. But recall that this is a binary outcome space, i.e., the only possible outcomes are $X$ or $\neg X$. So the full probability distribution (over the two possible outcomes) can be summarized by one probability, $p := Pr[X]$ (which implies $1-p = Pr[\neg X]$). As we’ve provided a full probability distribution over the outcome space, it’s not possible to say anything more — any uncertainty must be embedded in this distribution.

Intuitively, probabilities near 0 or 1 reflect a high degree of certainty. A prediction without any uncertainty at all would just be a yes or no answer, i.e., a predicted probability of 0 or 1. It would just state which outcome will occur, with no notion of uncertainty or hedging.

More precisely, confidence in a probability prediction is reflected by how extreme it is relative to a baseline or prior belief. To see this, suppose that there is an event $X$ that is very likely to occur, and that a prediction market has given $X$ a predicted probability of 0.97. If you are maximally uncertain/ignorant about $X$, what probability do you assign? Intuitively, you hedge your bets and stick to 0.97. Here, 0.97 is the baseline, which you can treat as your prior probability. Given this prior information, a prediction of 0.97 reflects maximal uncertainty. (If you didn’t have any prior information whatsoever, you would go with 0.5.)

Then, if you have some new information about $X$, you can update your prior to get a posterior. If your information provides strong evidence in favor of $X$, then your posterior probability might jump up to, say, 0.997. On the other hand, if your information strongly supports $\neg X$, then your posterior might drop to, say, 0.78. Thus, your degree of confidence is revealed by the degree to which your probability moves away from the baseline and toward 0 or 1.

You can use Bayes’ Theorem to play with some numbers yourself. Denote your prior by $p := Pr[X]$, and assume you’ve used your information $D$ to compute the likelihoods $q(X) := Pr[D \mid X]$ and $q(\neg X) := Pr[D \mid \neg X]$. Denote the likelihood ratio by $\lambda := q(X) / q(\neg X)$.

Then compute the posterior and rearrange in terms of the likelihood ratio:

\[\begin{aligned} Pr[X \mid D] &= \frac{p \cdot q(X)}{p \cdot q(X) + (1-p)q(\neg X))}\\ &= \frac{p \cdot q(X)/q(\neg X)}{p \cdot q(X)/q(\neg X) + (1-p)}\\ &= \frac{p \cdot \lambda}{p \cdot \lambda + (1-p)} \end{aligned}\]

Note that the posterior can be expressed purely in terms of the prior and the likelihood ratio (i.e., it doesn’t depend on the individual likelihoods). This means that the magnitudes of the likelihoods don’t matter; all that matters is their ratio, which indicates how much the information $D$ favors $X$ relative to $\neg X$.

If you play with this formula, you’ll get a sense of how the information in the likelihoods updates the prior to a posterior probability. Notice that when $\lambda = 1$, the posterior reduces to $p$, the prior. In other words, when $D$ is uninformative about $X$, it leaves your prior belief unchanged. Furthermore, for any prior belief $p$, if $\lambda > 1$, then your posterior will be pushed upward from your prior (and vice versa for $\lambda < 1$). That is, any information in favor of $X$ will increase your confidence in $X$ — even if $p=0.999$!

Judging the quality of probability predictions is simple: just check that they’re calibrated. For example, predictions made with, 80% confidence should be correct 80% of the time. With enough completed predictions, you can plot a reliability diagram to assess the calibration of the predictions.

Uncertainty over models

When we are working with a model, we also have a second source of uncertainty — that of the model. This uncertainty is reflected in the posterior distribution over the parameters (at least for Bayesians — frequentists would use the sampling distribution of the parameter estimator to derive confidence intervals / standard errors). Because these parameters are used to produce predictions, their uncertainty propagates through to produce additional uncertainty over the outcome.

This is meta-uncertainty: uncertainty over the model which produces the uncertain prediction of the outcome. (In fact, you can have higher levels of meta-uncertainty by including uncertainty over any hyperparameters of the model.)

For example, here’s a specification for a Bayesian logistic regression model, where I’ve put an informative prior on the model coefficients:

\[\begin{aligned} y_i &\sim Bernoulli(p_i) \\ \log\left(\frac{p_i}{1 - p_i}\right) &= \beta_0 + x_{i1} \beta_1 + x_{i2} \beta_2 + \ldots + x_{iK} \beta_K \\ \beta_k &\sim \mathcal{N}(0,1.5), \; k = 1,2,\ldots,K \end{aligned}\]

You can see how the uncertainty from the model’s prior (Normal distribution) propagates through, adding to the uncertainty in the likelihood (Bernoulli distribution). As a result, we get a full density for $p_i$ over the interval $(0,1)$. The spread of this density reflects model uncertainty, whereas the distance from the baseline probability reflects the degree of confidence in the prediction of the outcome. The key thing to realize is that these two sources of uncertainty are orthogonal.

For example, you could simultaneously have a very confident prediction of the outcome but with a lot of model uncertainty — a widely spread posterior distribution that is far away from the prior probability (where the prior is marked by the small X):

  Confident prediction, uncertain model
⠀⢸⠀⠀⠀⠀⠀⠀⠀⢀⠤⠤⣄⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⠀⠀⠀⡔⠁⠀⠀⠀⠘⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⠀⠀⡸⠀⠀⠀⠀⠀⠀⠀⠱⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⠀⢰⠁⠀⠀⠀⠀⠀⠀⠀⠀⠘⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⢀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⡸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⡄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⢰⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠘⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⡎⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⢀⠇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢣⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⡎⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢢⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠣⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⢰⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⠢⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⡎⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠈⢆⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠑⢄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠁⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠑⠢⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠀⢸⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠉⠒⠤⢄⣀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸
⠒⢺⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠛⠛⠛⠒⠒⠒⠒⠒⠒⠒X⠒⠒⠒⠒⠒⠒⠒⢺

Or you could have a very unconfident prediction of the outcome with very little model uncertainty — a tightly distributed posterior distribution centered on the prior probability:

  Uncertain prediction, confident model
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⡀⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⡇⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⡇⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⡇⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⡇⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⢱⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⢸⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢠⠃⢸⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⢸⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⢸⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⢸⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⢸⠀⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠘⡄⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡎⠀⠀⡇⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⡇⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡇⠀⠀⡇⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢀⠇⠀⠀⡇⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⢸⠀⠀⠀⢣⠀⠀⠀⠀⠀⡇
⠀⡇⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⡜⠀⠀⠀⢸⠀⠀⠀⠀⠀⡇
⠒⡗⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠓⠒X⠒⠚⠓⠒⠒⠒⠒⡗

Summing up

The argument I’ve made here can be summed up as:

A probability prediction is a full probability distribution and so it inherently quantifies uncertainty — it’s a misconception to think that you need a confidence interval to express uncertainty
Model uncertainty propagates through to produce an additional (but orthogonal) layer of uncertainty over the outcome

This may seem obvious in retrospect, but it’s always good to gain clarity on the fundamentals, where confused intuitions may lurk unnoticed. Here, intuitions such as “a scalar prediction must be a point estimate” and “you need a confidence interval to express uncertainty” are highly misleading.

In this case, understanding the distinct sources of uncertainty has resolved some confusion I had about prediction markets and machine learning model predictions. The upshot is that (if I’m comfortable ignoring model uncertainty), I need only be concerned that the probability predictions are calibrated. This is straightforward to check: for prediction markets, you just need some historical data on outcomes; for classifier models, you can check on a holdout dataset. Then you can be comfortable that the predictions are accurately quantifying uncertainty.

Find Your Community: Increasing user engagement at Reddit

2017-06-22T20:00:00+00:00

Toban Wiebe is an Insight Data Science Fellow in Silicon Valley. For his fellowship project, he performed an analysis of Reddit user engagement to provide actionable insights for the business.

Reddit is a social media platform where people can browse or join communities called subreddits to submit posts as well as vote and comment on submissions. Any user can create a subreddit — as a result, there are now over 1 million subreddits spanning almost any topic imaginable. Subreddits are based around topics such as r/cats or r/datascience.

When someone first visits reddit.com, they are shown the reddit frontpage, a post feed comprised of the currently most popular content from a curated selection of 50 default subreddits. (Note: this has recently changed to showing popular posts from any subreddit, with some exceptions. However, this only affects new accounts — existing users still see the default frontpage subreddits). Logged-in users (also called redditors) can customize their feed by subscribing to other subreddits; otherwise they see the default frontpage content.

However, while frontpage subreddits are prominently featured, the web interface doesn’t make it very obvious how to find other subreddits. Reddit has a very useful subreddit discovery feature, but it’s only accessible through the small “More” link in the subreddit navigation bar at the top of the page. This made me wonder: would reddit benefit from making subreddit discovery more prominent in the user interface? To answer this question, I analyzed user engagement across subreddits to look for evidence of a causal relationship.

Data

Data on all reddit comments is publicly available on Google BigQuery thanks to redditor fhoffa. The data is very large and is divided into monthly tables — the table for May 2017 alone contains 80 million rows and takes up 20GB on disk. As such, it was very important to use BigQuery for performant queries.

To access the data, I used the Google Cloud Datalab, a Jupyter notebook environment running on a VM instance which connects to BigQuery. By using SQL queries to aggegate the data down to a manageable size, I could load the results into pandas dataframes for my analyses.

Analysis of user engagement

As reddit content is entirely user-generated, I decided to focus on commenting as a proxy for adding value to the reddit experience. Comments are an integral part of reddit — as many a redditor knows, discussions in the comments often contain some of the most valuable content. For each user, I calculated the number of monthly comments as a metric for user engagement.

Frontpage subreddits vs the rest

The distinction between the frontpage and other subreddits is a key divide in the spectrum of reddit usage styles. On one end of the spectrum are the “frontpage-focused” users; at the other end are the users who participate solely in non-frontpage subreddits. To measure this, I calculated a participation metric: the percentage of comments outside of frontpage subreddits for each user in each month.

The frontpage surfaces excellent content with broad appeal and is extremely popular. However, the frontpage default subreddits only produce 17% of all reddit comments, despite their outsized popularity. Other subreddits vary widely, but many thriving communities are based around a specific interest, such as r/coffee.

Based on my experience as a redditor, I hypothesized that people would get the most value out of reddit by going beyond the frontpage and finding subreddits related to their specific interests. As a result, they would become more engaged in those communities and leave more comments overall. However, given the popularity of the frontpage, it wasn’t obvious that this would be true.

Initial cohort-level analysis

I started by looking at the relationship between engagement and participation in other subreddits at the cohort level in order to avoid selection bias from tenure. For example, people who have been on reddit longer are likely to be both more engaged as well as participate more broadly in other subreddits.

I took four monthly cohorts (from July to October of 2016) and measured their commenting activity in their 6th month of tenure. As I suspected, there is a strong relationship between engagement and participation in other subreddits. The chart below shows that “frontpage-focused” users (0-20% of comments outside of frontpage subreddits) have the lowest engagement with an average of 22.8 comments per month. Engagement increases by 22.7% to an average of 27.8 comments per month for users with the majority of their comments outside of frontpage subreddits. The difference between the first two groups is 2.55 with a 99% confidence interval of (1.69, 3.41).

(Note: This chart is limited to users with 5-150 comments per month, to exclude outliers and low comment counts, which could skew the results. I also excluded the 80-100% group, which is almost entirely composed of users who never comment on frontpage subreddits. This group is large and behaves very differently — likely because they joined reddit to participate in a specific subreddit, with no interest in the frontpage. As my focus is on increasing engagement for frontpage-focused users, I restricted my analysis to users with at least some frontpage engagement.)

Furthermore, there are many users in this frontpage-focused group, so there are potentially large gains to be made from helping them to discover other subreddits. The chart below adds bars for the number of users in each group (the average number per monthly cohort).

Panel data analysis

Though my cohort analysis controlled for the selection bias from tenure, I was worried that it was being confounded by another selection effect: what if the more highly engaged users within each cohort are more likely to seek out other subreddits? To control for this, I constructed a panel dataset to quantify the effect at the user level. This way, I could directly see how each user’s engagement changes over time as their participation shifts to other subreddits.

Formally, I modeled this with user fixed-effects in a linear model:

\[y_{it} = \alpha_i + \beta^{\prime} x_{it} + \varepsilon_{it}.\]

Here, $y_{it}$ is the engagement of user $i$ in month $t$. $\alpha_i$ is the user fixed-effect. $x_{it}$ is a vector which includes the participation metric (% of comments outside of frontpage subreddits) as well as user tenure. As I don’t care about recovering the fixed-effects, I differenced them out to get the “within” estimator:

\[y_{it} - \bar y_{i} = \beta^{\prime} (x_{it} - \bar x_{i}) + (\varepsilon_{it} - \bar \varepsilon_{i}).\]

I ran this linear regression on all users who joined reddit within the last two years, which gave me 14 million user-month observations. This also yielded a positive effect, with an estimated 0.38 more monthly comments per user for every 10 percentage point increase in participation outside of the frontpage (with a 95% confidence interval of (0.35, 0.41)).

However, this is only about half of the effect size suggested by the above cohort-based analysis. This means that selection on engagement was confounding the previous analysis. Still, the estimated effect is quite substantial: if a frontpage-focused user shifted their participation to 70% outside of frontpage subreddits, that would induce an average increase of 11.6% in engagement.

For comparison, an extra year of tenure on reddit is associated with a 7% increase in engagement.

Proposed feature and A/B test

Based on this evidence, I conclude that reddit should do more to help frontpage-focused users discover other communities. Specifically, I propose adding a subreddit discovery feature into the frontpage sidebar. This way, subreddit discovery would be much more evident to frontpage visitors.

To evaluate the impact of this proposed sidebar feature (and to establish whether my finding is indeed a causal relationship), I designed an A/B test that could be implemented by reddit. The goal of the experiment is to determine if this feature increases the commenting activity of frontpage-focused users.

Target group

The experiment targets existing frontpage-focused users: registered users with some commenting activity, but who haven’t subscribed to any other subreddits (and hence still see the default frontpage). I also limited the experiment to users that joined in 2016 to avoid potential bias from older cohorts, and also because the newest cohorts are by far the largest. (Though it is typical to target newly registered users, reddit is already running an A/B test for subreddit discovery during the onboarding flow. As such, my proposed test focuses on existing users.)

Duration and metrics

The test would run for 5 weeks in total. The first week would be an adaptation period for users to try out the feature and subscribe to other subreddits. Then, the following 4 weeks would track the number of comments for each user.

An important secondary metric to track is the number of subscriptions to other subreddits. This tells us how effective the feature is at driving users to find other communities.

Power analysis

Unlike most other social media platforms, reddit is very conservative about changing the user experience (and primarily runs A/B tests to learn more about its users). As such, I tailored the design of the test to reflect this cautious approach.

I set the test significance at $\alpha = 0.01$ to be very stringent with respect to false positives. I set the power at $1 - \beta = 0.9$ to reflect less concern about false negatives. This way, a positive result would be very strong grounds for launching the feature.

The minimum detectable effect size that would be worthwhile should be based on the tradeoff between the value of additional comments and the opportunity cost of sidebar space (which would otherwise be ad space). As I don’t have this information, I chose an intuitively plausible effect of 5%, or about 1 extra comment per user per month, on average.

To reduce the number of users exposed to the new feature, I used an unbalanced design with a much larger control group. I started from a total sample size of 27,000 users because that is roughly the number of frontpage-focused users among all of the 2016 cohorts. This is more than enough for a balanced design, so I minimized the treatment group size subject to the other constraints. This gave me a sample size of 7,000 users in the treatment group and 20,000 users in the control group.

Impact

There are roughly 300,000 actively commenting users who only comment on the frontpage, so a 5% increase in their commenting would result in an overall increase of roughly 0.5% in commenting across all of reddit! This effect could also grow over time as these users continue to explore other subreddits. Moreover, this is only considering the impact from users who are commenting at all — presumably there are far more users who never comment at all.

Conclusion

Reddit’s mission is “to help people discover places where they can be their true selves”. This fits nicely with my recommendation of making community discovery more prominent in the user interface. Furthermore, helping people to find other communities has a direct financial impact as reddit sells targeted subreddit ads. All told, my analysis suggests that reddit could benefit substantially from improving the subreddit discovery process, and it would be well worth running an experiment to precisely quantify this impact.

Dealing with Anaconda Python in Linux

2016-09-30T15:00:00+00:00

Anaconda is a very handy Python distribution that bundles a slew of scientific packages along with the handy conda package manager, allowing you to easily update your packages to the latest versions. However, it doesn’t seamlessly integrate with Linux: the installer will simply add a line to your shell configuration file (e.g., ~/.bashrc) prepending the Anaconda bin directory to the PATH environment variable. This means that Anaconda’s python and other packages will override Linux’s system python and packages (to see this, run which python). For the most part, this shouldn’t be an issue, but in some cases it can cause trouble.

For example, if you use the yaourt package manager on Arch Linux, you’ll run into two problems with Anaconda Python. First, Anaconda’s curl will break yaourt. Second, yaourt will blindly install Python packages into the Anaconda Python package directory. This sometimes causes conflicts with Python packages installed through pacman (which are properly installed into the system Python package directory). I ran into this problem when installing Reddit Terminal Viewer.

If you just want to use Anaconda but not have it take over like this, there’s an easy fix: instead of prepending the Anaconda bin directory to PATH (which gives it priority), simply append it so that system Python (/usr/bin/python) and packages take precedence. In your shell configuration (e.g., ~/.bashrc), change the line added by the Anaconda installer:

# Default configuration of Anaconda installer
#export PATH="/path/to/anaconda3/bin:$PATH"

# Append Anaconda so that it doesn't override system Python
export PATH="$PATH:/path/to/anaconda3/bin"

Now, python (and curl, for example) will still be the system version, but Anaconda applications will also be available (jupyter, ipython, conda, etc). My Anaconda workflow is to work in a Jupyter notebook along with a text editor. Since jupyter is tied to the Anaconda python, it continues to work seamlessly as before.

If you want to temporarily use Anaconda’s python at the shell, just use conda’s environment manager to activate/deactivate it:

source activate <env>
source deactivate

The root environment uses the standard Anaconda python installation, so to activate the Anaconda’s python, just do source activate root. (Note: this only affects the current shell session).

Bonus: Python linting and auto-completion in Vim

Since Vim (and Neovim) are compiled against the system python, they won’t work with your Anaconda packages by default. For example, running pylint as a syntax checker in Syntastic or Neomake will use the system python and give errors when you import Anaconda packages. The solution is very simple: just include your Anaconda package directory in the PYTHONPATH environment variable. This way, the system python will be able to import the Anaconda packages.

In your shell configuration (e.g., ~/.bashrc), append your Anaconda site-packages directory as such:

export PYTHONPATH="$PYTHONPATH:/path/to/anaconda3/lib/python3.6/site-packages"

Also, the (Neovim-only) python completions package deoplete-jedi allows you to specify which python interpreter to use for the completion server. It seems that setting PYTHONPATH isn’t sufficient (for example, I couldn’t get pandas completions to work). Add the following line to your Vim config (with the correct path):

let g:deoplete#sources#jedi#python_path = '/path/to/anaconda3/bin/python'

Make your own Julia packages

2016-08-20T20:30:00+00:00

[ Update: with the release of Julia 1.0, the package manager, Pkg, was completely rewritten, though it remains quite similar. Check out the Pkg documentation to see what’s new. ]

Julia is a fantastic language for scientific computing and as a result is gaining traction among researchers. In research projects, it often happens that you need to write code which could be generalized and reused. For example, in a recent project, I coded up a marriage market model as a component of a larger model. The best way to make such code reusable is to create a package (most languages provide a packaging system).

Julia provides a convenient way to create a new package. As explained in the manual, Pkg.generate("NewPackage", "MIT") initializes a git repo containing the package structure for a package named NewPackage with an MIT license. If you configure your GitHub username in git (git config --global github.user "USERNAME"), it will even configure the remote repository (which you’ll still need to create in GitHub).

Since the Julia package directory isn’t a very convenient location for developing your package (e.g., ~/.julia/v0.4/), I recommend moving your package (cut and paste) to your desired location and then symlinking it to the Julia package directory (ln -s your/directory/NewPackage ~/.julia/v0.4/NewPackage).

Converting your code into a package takes a few simple steps. src/NewPackage.jl contains the actual module that users will import. The manual explains how to create a module. You simply define functions and types and use the export statement to choose which ones to make available externally. A good way to organize your code is to put it in the src directory and then include() it in the module file.

Next, you should specify which packages are required as dependencies in the REQUIRE file. The file already includes the current stable version of Julia as a requirement.

While optional, it’s a good idea to include some tests of your code! Put them in test/runtests.jl and run Pkg.test("NewPackage") to run your tests. If you enable Travis CI in GitHub, it will automatically run your tests when you git push.

Finally, don’t forget to fill out the README.md!

To share your package with others, push it to GitHub and then others can install it with Pkg.clone("git@github.com:username/NewPackage.jl.git"). The manual also has instructions for registering your package as an official Julia package that can be installed with Pkg.install().

Look at my package for a simple example. The QuantEcon package is a more comprehensive example that I drew on in making my package.

MathJax with Kramdown

2016-02-10T10:30:00+00:00

Last week, GitHub Pages upgraded to Jekyll 3.0. One major consequence of the upgrade is that, for Jekyll sites hosted on GitHub Pages, only the kramdown engine is supported.

This broke my existing MathJax setup – I was using the redcarpet Markdown engine before, which understood math with the delimiters \$ \LaTeX \$ and \\[ \LaTeX \\] for inline and displayed math, respectively.

kramdown only recognizes $$ \LaTeX $$ for both inline and displayed math. This means that it will automatically infer whether you want inline or displayed math.

Here’s the live demo with inline $\LaTeX$ and displayed

\[\LaTeX\]

This is an easy fix unless you have a lot of old posts with LaTeX to convert, in which case you should probably write a small converter script to switch your math delimiters to $$.

Simple keyboard shortcuts for any website

2015-11-28T16:30:00+00:00

I’m a huge fan of Gmail’s vim-inspired keyboard shortcuts. It’s been nice to see the adoption of keyboard shortcuts in other sites, but for most of the web, keyboard access remains completely ignored. I always just figured that it must be prohibitively difficult to implement keyboard shortcuts in a website, and so only the big players could do it. It turns out, however, that implementing keyboard shortcuts is embarassingly easy.

With no knowledge of javascript, I was able to add some simple keyboard navigation to this website (source repo). Though this website is built with Jekyll, this method would work on any website as it only uses some simple client-side javascript. I used the excellent Mousetrap library to handle keyboard input, and wrote some simple navigation commands in javascript.

I implemented two kinds of navigation in the spirit of Gmail: jumping and list navigation. (Note: I’m using the bind dictionary extension to bind multiple keys at once):

Jumping

To get <g then h> style jumping, all you need is:

Mousetrap.bind({
	'g h': function() {	window.location.href = "/"; },
	'g b': function() {	window.location.href = "/blog"; },
	'g r': function() {	window.location.href = "/research"; },
})

That’s it: incredibly simple.

Adding j/k navigation was a little more complicated. I wrote a function blogNav which takes a key press and does the following:

gets a list of all the post links (using their unique class, post-link)
gets the currently focused element
moves the link focus accordingly (‘j/k’), or launches the focused link (‘o’)

I also tweaked the CSS to change the color of focused links. To bind it in Mousetrap:

Mousetrap.bind({
	'j': function() { blogNav('j'); },
	'k': function() { blogNav('k'); },
	'o': function() { blogNav('o'); }
})

Conclusion

Adding keyboard shortcuts to a website is surprisingly easy. It’s unfortunate that most web development is entirely focused on mouse input, relegating keyboard users to hacky extensions like vimium and pentadactyl. Hopefully Mousetrap can help more websites become keyboard friendly.

Economic growth via effective regulation

2015-11-07T08:30:00+00:00

I somehow ended up reading this long essay by economist John Cochrane on growth-oriented policy. I really enjoyed it because it takes a hard-headed approach to topics where reason is too often crowded out by moral instinct. (Other examples of this approach include Effective Altruism, or the Eco-Modernist Manifesto.)

It’s an idealistic vision – it ignores the political economy that sustains the gigantic inefficiencies he decries – yet it’s an overwhelmingly sensible one. Perhaps I’m biased towards the views of pro-growth economists, but this essay was packed with an uncommon amount of common sense (which is a pretty low bar when it comes to the topic of policy).

The essay makes a two-part argument: that growth should be a very high policy priority, and that there’s a lot of inefficient regulation that hinders growth. I’ll quote a bunch of highlights below if you don’t want to read the whole thing.

In the long run, nothing but growth matters. Small differences in growth rates lead to dramatically different outcomes, because of compounding.

If the US economy had grown at 2% rather than 3.5% since 1950, income per person by 2000 would have been $23,000 not $50,000. That’s a huge difference. Nowhere in economic policy are we even talking about events that will double, or halve, the average American’s living standards in the next generation.

The primacy of economic growth should be a really obvious point, but it’s easy to lose sight of the big picture. If you think about it, economic growth is the only thing that has lifted nations out of poverty.

Nothing other than productivity matters in the long run. A factor of three increase in income in 50 years, and the much larger rise in income and health since the dawn of the industrial age, dwarfs what unions bargaining for better wages, progressive taxes or redistribution, monetary, fiscal or other stimulus programs, minimum wage laws or other Federal regulation of labor markets, price caps and supports, subsidies, or much of anything else the government can do.

And the primacy of growth should be non-controversial:

38% more income — or 26% less income — drives just about any agenda one could wish for, from strong defense, to environmental protection, to the affordability of social programs, to the welfare of any segment of the population, to public investments, health, and fundamental research.

Dumb regulation hinders growth. To increase growth, there’s a lot of obviously wasteful regulation that could be made a lot smarter. Unfortunately, that regulation is there because of problems of political economy, and so it’s not as easy to fix as it looks.

When the average person (voter) expresses concern over inequality, what they really mean is that they are concerned that average people are not getting ahead economically. If the average person were getting ahead, whether some big shot CEOs fly on private jets or not would make little difference. Conversely, the average voter, if not the average left-wing pundit, does not support equality of misery. If the average person continues to do poorly, it would bring them little solace for the government to tax away the lifestyles of the rich and famous.

On the problems of health insurance regulation:

The central problem of preexisting conditions was an artifact of regulation. In the ideal form of health insurance, you buy cheap catastrophic insurance when young, but the insurance policy can follow you as you age, change jobs, and move from state to state, and does not radically increase premiums if you get sick. […] We need to allow simple, portable, largely catastrophic, lifelong, guaranteed-renewable health insurance to emerge. Right now it’s illegal.

On energy policy:

The poster child for inefficiency may well be the mandate for gasoline producers to use ethanol. Corn ethanol, it turns out, does nothing to help the environment: It takes nearly as much petroleum energy to produce it as it contains, in the form of fertilizer, transport fuel and so on; it uses up valuable land, which directly emits greenhouse gases, and contributes to erosion and runoff; it drives up the price of food.

If you are serious about carbon, let the words “nuclear power” pass your lips. We have sitting before us a technology that can easily supply our electricity and many transport needs, with zero carbon or methane emissions. New designs, if only they could pass the immense regulatory hurdle, would be much safer than the 1950s Soviet technology that failed at Chernobyl or the 1960s technology that failed at Fukushima. We are now operating antiques. And even with this rate of accident, nuclear power has caused orders of magnitude less human or environmental suffering than any other fuel.

Similarly, the most environmentally friendly way for people to live is in tightly packed cities, fed by genetically modified foods which yield more per acre of farmland and require fewer fertilizers and pesticides, from laser-leveled fields run efficiently by large corporations in the highest productivity locations. Federal policies to the contrary are not just anti-growth, they’re anti-environment too. When Federal policy can say these things in public, it will have a bit more standing to invoke the name of “science.”

On taxation:

Often, however, tax reform proposals sacrifice too quickly the principles of what a good tax system should be with perceived political accommodations to powerful interest groups. Economists should not play politician. We should always start with “in a perfect world, here is what the tax code should look like,” and accommodate political constraints only when asked to. Political constraints change quickly. Economic fundamentals do not.

The right corporate tax rate is zero. Corporations never pay taxes. Every dollar of taxes that a corporation pays comes from higher prices of their products, lower wages to their workers, or lower returns to their owners. […] For all these reasons, eliminating the corporate tax is as likely to be more rather than less progressive. The higher prices a corporation charges hurt everyone. The lower wages corporations pay hurt workers. The income it passes along to its owners is subject to our highly progressive tax system.

When we say broaden the base by removing deductions and credits, we should be serious about that. Thus, even the holy trinity of mortgage interest deduction, charitable donation deduction, and employer provided health insurance deduction should be scrapped. The extra revenue could finance a large reduction in marginal rates. Why? Consider the mortgage interest deduction. Imagine that in the absence of the deduction, Congress proposes to send a check to each homeowner, in proportion to the interest he or she pays on money borrowed against the value of the house. Furthermore, rich people, people who buy more expensive houses, people who borrow lots of money, and people who refinance often to take cash out get bigger checks than poor people, people who buy smaller houses, people who save up and pay cash, or people who pay down their mortgages. A rich person buying a huge house in Palo Alto, who pays 40% marginal income tax rate, gets a check for 40% of his huge mortgage. A poor person buying a small house in Fresno, who pays a 10% income tax, gets a check for 10% of his much smaller mortgage. There would be riots in the streets before this bill would pass. Yet this is exactly what the mortgage interest deduction accomplishes.

On labor markets:

Start, of course, with taxes: income taxes and payroll taxes are primarily taxes on employment. But the regulatory burdens of employment are larger still, as anyone who has tried to get a nanny legal will attest. Minimum wages, occupational licensing, anti-discrimination laws, laws regulating hours people can work, benefits they must receive, leave they must be given, fear of lawsuits if you fire someone, and so forth all impede the labor market.

The usual argument is that workers need protection of all these laws. Well, the supposed protections do cost economic growth, and they do reduce employment. How much do they actually protect workers? The strongest force for worker protection is a vibrant labor market — if you don’t like this job, go take another. The tightly regulated labor market makes it much harder to get a new job, and thus, paradoxically, lowers your bargaining power in the old one.

An exceptional section on immigration. Had a hard time not quoting the whole thing:

Immigrants contribute to economic growth. Even if income per capita is unchanged, imagine how much better off our social security system, our medicare system, our unfunded pension promises, and our looming deficits and debt would be, if America could attract a steady flow of young, hard-working people who want to come and pay taxes. Aha, we can attract them! They’re beating the doors down to come. But then we keep them out.

Allowing free migration is, by many estimates the single policy change that would raise world GDP the most. If you believe in free trade in goods, and free investment, then you have to believe that free movement of people has the same benefits.

On education:

The culprit is easy to find: awful public schools run by and for the benefit of politically powerful teachers’ and administrators’ unions. (Don’t forget the latter! Teachers account for only half of typical public school expenses.) Education poses a particularly large tradeoff between profits to incumbents and economic growth, since education lies at the foundation of higher productivity. In addition, the costs of awful schools fall primarily on lower-income people who cannot afford to get out of the system. It is one of the major contributors to inequality. The solution is simple as well: widespread financing by vouchers and charter schools. As with health care, a vibrant market demands that people control their spending, and can move it to where they get better results. As with health care, the government does not have to directly provide a service in order to help people to pay for that service. But as with health care, a healthy market also demands supply competition, that new schools be allowed to start and compete for students.

As exciting as this vision is, I’m not very optimistic about any of these prescriptions being implemented, at least not in the near term. But it’s good to know that we have a lot of simple solutions if we ever get desperate. And I hope that some of these proposals (like immigration) can gain enough momentum to be implemented in the medium term.

Note taking like a hacker

2015-10-06T10:30:00+00:00

In the spirit of blogging like a hacker, I am now taking notes like a hacker, thanks to a system I pieced together using Markdown and Dropbox. I have used Evernote and Google Docs for my notes in the past, but I was never satisfied.

I liked the idea of Markdown, and tried some online editors like StackEdit and Laverna. This was a nice experience, but I was lured by the tantalizing prospect of editing the Markdown files in my text editor (vim), rather than the web editor. (StackEdit can sync with Dropbox, allowing local editing of the plaintext files, but unfortunately, each file has to be synced manually, so you can’t just sync a directory.)

My note taking system had to have these features:

Markdown based
Accessible across devices
Edit notes in any text editor
MathJax enabled for writing math with $\LaTeX$

I don’t really care about embedding media or features like Evernote’s clipping functionality. As a PhD student, I mostly just want to save academic things like research ideas and paper summaries.

My system is minimal, but effective. Notes are just Markdown files in my Dropbox. On my desktop, I can edit them in any text editor. With vim, I use the vim-preview plugin to render an html preview.

On a mobile device, I can of course access the plaintext through the Dropbox app. But there are also Markdown apps like Jotterpad which can connect to Dropbox.

And that’s it! Very simple, but now my notes are in a very portable format. (Speaking of which, I wrote a little script to export notes from Laverna.)

New Jekyll site

2015-09-05T08:30:00+00:00

So I decided to move from Wordpress to Jekyll. Not that I had any complaints with Wordpress, it’s an absolutely fantastic platform. I just wanted to start blogging like a hacker. With Jekyll I can work from a text editor, use git for the entire site, and publish with a simple git push. Furthermore, I can host the site on Github Pages, which automatically rebuilds the site whenever a change is made.

The comments were a cesspool back on Wordpress (and I’m talking about legitimate, non-spam comments here), so I’m nuking all the past comments and starting here with no commenting system for the time being.

I’ve chosen a Creative Commons 4.0 license to encourage sharing. Feel free to copy code from the github repository that this site is generated from.

Double standard: caffeine vs nicotine

2015-01-16T14:20:48+00:00

[...] there is a kind of puritanical view that everything relating to nicotine is bad and harmful and should be stamped on.

-Richard West

I'm no expert on this topic, but it seems to me that the harms of tobacco are from smoking (and chewing) and not from the nicotine itself (apart from the addictiveness). So nicotine administered with gum/patch/vaping would be somewhat more like caffeine in harmfulness (loads of people are addicted to caffeine).

Nicotine also seems to have similar useful effects. From Wikipedia: "Nicotine appears to have significant performance enhancing effects, particularly in fine motor skills, attention, and memory." A Martian scientist comparing the harms and benefits (for humans) of drinking coffee vs vaping nicotine might think that both drugs are safe and have productivity-boosting effects, and wouldn't see a reason for only one of them to be socially acceptable.

So why is nicotine held to a much higher standard? Well, the public health effort to end smoking became a moral crusade. As Jonathan Haidt puts it, morality binds and blinds. The anti-smoking crusade gained enough strength that it created a hated enemy. Smokers in North America are now a low-status out-group from the perspective of most non-smokers. As usual, when activists morally charge their case by painting something as sacred or evil, they commit themselves to not giving up an inch. Smoking is pure evil, and nicotine is central to smoking, so nicotine must be evil too. Any admission that nicotine might not be horrible is helping the enemy.

Caffeine, on the other hand, did not get loaded with negative affect because it had a safe delivery system. Imagine if nicotine had historically been consumed in drink form instead of smoking -- in such a world, I imagine that it would just be another stimulant drink like coffee or tea.

It's a real shame that e-cigarettes (nicotine) are being morally condemned along with cigarettes (tobacco). If smokers can switch to vaping, they can avoid all the harm and still get their nicotine. From a public health perspective, getting smokers to switch is a huge win. Doctors should be prescribing e-cigs to smokers, government should promote switching. Unfortunately, since nicotine has been painted evil by association, it's probably not going to happen.

A thought on methodology

2014-08-26T01:30:50+00:00

I've never felt comfortable with the logical positivists' "science is prediction" characterization, primarily because it neglects what I intuitively think of as the heart of science: explanation. For example, Darwin's theory of evolution – perhaps the greatest scientific discovery ever – is big on explanation, but not nearly as big on prediction. (Because evolution happens on such long timescales, although microbial evolution can be observed on very short timescales.) Or consider the 'selfish gene' paradigm, the evolutionary paradigm of viewing the gene as fundamental unit of selection, and the organism as a mere tool fabricated by the genes for the purpose of propagating themselves into the future. Dawkins' discusses (I think it was in The Extended Phenotype) criticisms of the idea as not providing any new predictions. My initial reaction to these criticisms is always: So what? They're wonderful explanations of the world. They make sense of the world. Isn't that pretty remarkable?

The point I want to make here is that the emphasis on prediction is just a convenient special case of a more general principle: that a theory should correspond to reality, just as a map corresponds to the territory. Reality can be observed past, present, and future. It is just as well to vet a theory against past observations as against future observations. The reason the scientific method favors prediction is that it prevents the scientist from concocting a 'just so' story that too neatly fits the existing facts ("overfitting"). An idealized honest scientist can test a theory against any empirical evidence.

Darwin's theory is so amazing because it makes sense of so many existing facts. (And you can still make predictions about historical facts ("retrodictions"), such as the famous quip that evolution would be falsified by finding fossil rabbits in the precambrian.) Of course, if you have a beautiful theory that has no connection to reality, then you're not doing science. Science is concerned with explaining reality, and so scientific theories must say things about reality – things which can (in principle) be empirically checked. It ultimately doesn't matter where that evidence is temporally located.

Toban Wiebe

Quantifying uncertainty in probability predictions

Two types of uncertainty

Uncertainty over outcomes

Uncertainty over models

Summing up

Find Your Community: Increasing user engagement at Reddit

Data

Analysis of user engagement

Frontpage subreddits vs the rest

Initial cohort-level analysis

Panel data analysis

Proposed feature and A/B test

Target group

Duration and metrics

Power analysis

Impact

Conclusion

Dealing with Anaconda Python in Linux

Bonus: Python linting and auto-completion in Vim

Make your own Julia packages

MathJax with Kramdown

Simple keyboard shortcuts for any website

Jumping

Blog post navigation

Conclusion

Economic growth via effective regulation

Note taking like a hacker

New Jekyll site

Double standard: caffeine vs nicotine

A thought on methodology