There are some things you may need to learn not for the content itself but for the confidence boost. Maybe you need to learn them so you can confidently say you didn’t need to. Also, some things you need to learn before you can see uses for them. (More on that theme here.)

I’ve learned several things backward in the sense of learning the advanced material before the elementary. For example, I studied PDEs in graduate school before having mastered the typical undergraduate differential equation curriculum. That nagged at me. I kept thinking I might find some use for the undergrad tricks. When I had a chance to teach the undergrad course a couple times, I increased my confidence. I also convinced myself that I didn’t need that material after all.

My experience with statistics was similar. I was writing research articles in statistics before I learned some of the introductory material. Once again the opportunity to teach the introductory material increased my confidence. The material wasn’t particularly useful, but the experience of having taught it was.

**Related post**: Psychological encapsulation

[1] See Yeats’ poem The Second Coming:

…

The best lack all conviction, while the worst

Are full of passionate intensity.

…

]]>

Do we even need probability approximations anymore? They’re not as necessary for numerical computation as they once were, but they remain vital for understanding the behavior of probability distributions and for theoretical calculations.

Textbooks often leave out details such as quantifying the error when discussion approximations. The following pages are notes I wrote to fill in some of these details when I was teaching.

- Error in the normal approximation to the binomial distribution
- Error in the normal approximation to the gamma distribution
- Error in the normal approximation to the Poisson distribution
- Error in the normal approximation to the t distribution
- Error in the Poisson approximation to the binomial distribution
- Error in the normal approximation to the beta distribution
- Camp-Paulson normal approximation to the binomial distribution
- Diagram of probability distribution relationships
- Relative error in normal approximations

See also blog posts tagged Probability and statistics and the Twitter account ProbFact.

**Last week**: Numerical computing resources

**Next week**: Miscellaneous math notes

There are three ways Bayesian posterior probability calculations can degrade with more data:

- Polynomial approximation
- Missing the spike
- Underflow

Elementary numerical integration algorithms, such as Gaussian quadrature, are based on polynomial approximations. The method aims to exactly integrate a polynomial that approximates the integrand. But likelihood functions are not approximately polynomial, and they become less like polynomials when they contain more data. They become more like a normal density, asymptotically flat in the tails, something no polynomial can do. With better integration techniques, the integration accuracy will *improve* with more data rather than degrade.

With more data, the posterior distribution becomes more concentrated. This means that a naive approach to integration might entirely miss the part of the integrand where nearly all the mass is concentrated. You need to make sure your integration method is putting its effort where the action is. Fortunately, it’s easy to estimate where the mode should be.

The third problem is that software calculating the likelihood function can underflow with even a moderate amount of data. The usual solution is to work with the logarithm of the likelihood function, but with numerical integration the solution isn’t quite that simple. You need to integrate the likelihood function itself, not its logarithm. I describe how to deal with this situation in Avoiding underflow in Bayesian computations.

***

If you’d like help with statistical computation, let’s talk.

]]>**Update **(27 Jan 2015): You may want to skip the post below and use ace-window. I plan to use it rather than my solution below. Also, see Sacha Chua’s video on window management, including using ace-window.

You can use `C-x o`

to move the cursor to the “other” window. That works great when you only have two windows: `C-x o`

toggles between them.

When you have more windows, `C-x o`

moves to the next window in top-to-bottom, left-to-right order. So if you have five windows, for example, you have to go to the “other” window up to four times to move to each of the windows.

The cursor is in the A window. Repeatedly executing `C-x o`

will move the cursor to C, B, and D.

There are more direct commands:

`windmove-up`

`windmove-down`

`windmove-left`

`windmove-right`

You might map these to Control or Alt plus the arrow keys to have a memorable key sequence to navigate windows. However, such keybindings may conflict with existing keybindings, particularly in `org-mode`

.

You can move windows around with the buffer-move. (Technically the windows stay fixed and the buffer contents moves around.)

Since the various Control arrow keys and Alt arrow keys are spoken for in my configuration, I thought I’d create Super and Hyper keys so that Super-up would map to `windmove-up`

etc.

Several articles recommend mapping the Windows key to Super and the Menu key to Hyper on Windows. The former is problematic because so many of the Windows key combinations are already used by the OS. I did get the latter to work by adding the following to the branch of my `init.el`

that runs on Windows.

(setq w32-pass-apps-to-system nil w32-apps-modifier 'hyper) (global-set-key (kbd "<H-up>") 'windmove-up) (global-set-key (kbd "<H-down>") 'windmove-down) (global-set-key (kbd "<H-left>") 'windmove-left) (global-set-key (kbd "<H-right>") 'windmove-right)

I haven’t figured out yet how to make Super or Hyper keys work on Linux. Suggestions for setting up Super and Hyper on Linux or Windows would be much appreciated.

]]>I’ve done consulting on the side throughout my career, and I planned to ramp up my consulting to the point that I could gradually transition into doing it full time. That never happened. I had to quit my day job before I had the **time** and **credibility** to find projects.

When you have a job and you tell someone you can work 10 hours a week on their project, working evenings and weekends, you sound like an amateur. But when you have your own business and tell someone you can only allocate 10 hours a week to their project, you sound like you’re in high demand and they’re glad you could squeeze them in.

When I left MD Anderson, I had one small consulting project lined up. I had been working on a second project, but it ended sooner than expected. (I was an expert witness on a case that settled out of court.) The result was that I started my consulting career with little work, and I imagine that’s common.

As soon as I announced on my blog that I was going out on my own, someone from a pharmaceutical company contacted me saying he was hoping I’d quit my job because he had a project for me, helping improve his company’s statistical software development. First day, first lead. This is going to be easy! Even though he was eager to get started, it was months before the project started and months more before I got paid.

In general, the larger a client is, the longer it takes to get projects started, and the longer they sit on invoices. (Google is an exception to the latter; they pay invoices fairly quickly.) Small clients negotiate projects quickly and pay quickly. They can help with your cash flow while you’re waiting on bigger clients.

This is a corollary to the discussion above. You might not have much income for the first few months, so you need several month’s living expenses in the bank before you start.

If you’re thinking about going out on your own and would like to talk about it, give me a call or send me an email. My contact information is listed here.

]]>

- Stand-alone code for numerical computing
- Accurately computing running variance
- IEEE floating-point exceptions in C++
- Double exponential integration
- Math.h in POSIX, ISO, and Visual Studio

See also the Twitter account SciPyTip and numerical programming articles I’ve written for Code Project.

**Last week**: Regular expressions

**Next week**: Probability approximations

In an all too familiar trade-off,

the result of striving for ultimate simplicity is intolerable complexity; to eliminatetoo-long proofswe find ourselves “hopelessly lost” among thetoo-long definitions. [emphasis added]

It’s as if there’s some sort of conservation of complexity, but not quite in the sense of a physical conservation law. Conservation of momentum, for example, means that if one part of a system loses 5 units of momentum, other parts of the system have to absorb exactly 5 units of momentum. But perceived complexity is psychological, not physical, and the accounting is not the same. By moving complexity around we might increase or decrease the overall complexity.

The opening quote suggests that complexity is an optimization problem, not an accounting problem. It also suggests that driving the complexity of one part of a system to its minimum may disproportionately increase the complexity of another part. Striving for the simplest possible proofs, for example, could make the definitions much harder to digest. There’s a similar dynamic in programming languages and programs.

Larry Wall said that Scheme is a beautiful programming language, but every Scheme program is ugly. Perl, on the other hand, is ugly, but it lets you write beautiful programs. Scheme can be simple because it requires libraries and applications to implement functionality that is part of more complex languages. I had similar thoughts about COM. It was an elegant object system that lead to hideous programs.

Scheme is a minimalist programming language, and COM is a minimalist object framework. By and large the software development community prefers complex languages and frameworks in hopes of writing smaller programs. Additional complexity in languages and frameworks isn’t felt as strongly as additional complexity in application code. (Until something breaks. Then you might have to explore parts of the language or framework that you had blissfully ignored before.)

The opening quote deals specifically with the complexity of theorems and proofs. In context, the author was saying that the price of Grothendieck’s elegant proofs was a daunting edifice of definitions. (More on that here.) Grothendieck may have taken this to extremes, but many mathematicians agree with the general approach of pushing complexity out of theorems and into definitions. Michael Spivak defends this approach in the preface to his book Calculus on Manifolds.

… the proof of [Stokes’] theorem is, in the mathematician’s sense, an utter triviality — a straight-forward calculation. On the other hand, even the statement of this triviality cannot be understood without a horde of definitions …

There are good reasons why the theorems should all be easy and the definitions hard. As the evolution of Stokes’ theorem revealed, a single simple principle can masquerade as several difficult results; the proofs of many theorems involve merely stripping away the disguise. The definitions, on the other hand, serve a twofold purpose: they are rigorous replacements for vague notions, and machinery for elegant proofs. [emphasis added]

Mathematicians like to push complexity into definitions like software developers like to push complexity into languages and frameworks. Both strategies can make life easier on professionals while making it harder on beginners.

**Related post**: A little simplicity goes a long way

- Regular expressions in PowerShell and Perl
- Regular expressions in Python
- Regular expressions in R
- Regular expressions in Mathematica
- C++ TR1 regular expressions

See also blog posts tagged regular expressions and the RegexTip Twitter account.

**Last week**: Probability resources

**Next week**: Numerical computing resources

It’s conventional to classify mathematicians as “problem solvers” or “theory builders,” depending on temperament. My experiences and the sources I consulted in writing this book convince me that curiosity about problems guides the growth of theories, rather than the other way around. Alexander Grothendieck and Robert Langlands … count among the most ambitious of all builders of mathematical theories, but everything they built was addressed to specific problems with ancient roots.

**Related post**: Examples bring a subject to life

It looks like the story is a matter of fraud rather than sloppiness. This is unfortunate because sloppiness is much more pervasive than fraud, and this could have made a great case study of bad analysis. However, one could look at it as a case study in how *good* analysis (by the folks at MD Anderson) can uncover fraud.

Now there’s a new development in the Potti saga. The latest issue of The Cancer Letter contains letters by whistle-blower Bradford Perez who warned officials at Duke about problems with Potti’s research.

]]>This blog is seven years old today. I’ve written 2,273 posts so far, a little less than one per day.

Over the holidays I combed through older posts looking for typos, broken links, etc. I fixed a lot of little things, but I’m sure I missed a few. If you find any problems, please let me know.

Here are some of the most popular posts for the last three years.

In 2011 I split my most popular post list into three parts:

In 2010 I split the list into two parts:

I didn’t post a list of most popular posts for 2009, but the most popular post that year was Why programmers are not paid in proportion to their productivity.

Finally, my most popular posts for 2008.

Here are a few other places you can find things I write:

]]>- Diagram of probability distribution relationships
- Central limit theorems
- Counting selections with replacement
- Distributions in R, Mathematica, Excel, SciPy
- Cauchy distribution parameter estimation
- Adult heights and mixture distributions

See also posts tagged probability and statistics and the Twitter account ProbFact.

**Last week**: Python resources

**Next week**: Regular expression resources

Bicycles on the Google campus in Mountainview, California:

Sunrise at Isle Vista, California:

View from University of California Santa Barbara:

Reflection of the Space Needle in the EMP museum in Seattle, Washington:

Paradise Falls, Thousand Oaks, California:

Bed and breakfast in Geldermalsen, The Netherlands:

Amsterdam, The Netherlands:

Chinatown in San Francisco, California

Heidelberg, Germany

“The bottom line is this. And it is hard for me to believe. They are going to use Magog to bring someone back from the past.”

Jack did not blink or move. His heart was beating very quickly now, but he merely said wryly, “Yes? Who are they going to bring back? Tell me it is John Coltrane.”

“You are never really serious are you?”

“I’m very serious about my music. Why are bad guys always bringing back people that everyone was glad to see go the first time? We could use more Coltrane …”

**Related post**: Nunc dimittis

Eroom’s law — that’s Moore’s law backward — observes that the number of new drugs approved per billion dollars spent on R&D has halved every nine years since 1950.

**Update**: Here’s an article from Nature that gives more details. The trend is pretty flat on a log scale, i.e. exponentially declining efficiency.

- Python counterparts for C math functions
- Bessel functions in SciPy
- Gamma and related functions in SciPy
- Distributions in SciPy

See also blog posts tagged Python, SciPy, and SymPy and the Twitter account SciPyTip.

**Last week**: Special functions

**Next week**: Probability resources

- How medieval astronomers made trig tables
- Monads are hard because …
- Time exchange rate
- Hello-world is the hard part
- What would Donald Knuth do?

Be sure to read the comments on the last post.

]]>- Diagram of relations between special functions
- Identities for gamma and related functions
- Relations between Bessel functions
- Bessel functions in SciPy
- Gamma and related functions in SciPy

See also blog posts tagged special functions.

I tweet about special functions occasionally on AnalysisFact

**Last week**: HTML, LaTeX, and Unicode

**Next week**: Python resources

`"101"*2`

. The string 11001101, on the other hand, is primitive. (It contains substrings that are not primitive, but the string as a whole cannot be factored into multiple copies of a single string.)
For a given *n*, let’s count how many primitive bit strings there are of length *n*. Call this *f*(*n*). There are 2^{n} bit strings of length *n*, and *f*(*n*) of these are primitive. For example, there are *f*(12) primitive bit strings of length 12. The strings that are not primitive are made of copies of primitive strings: two copies of a primitive string of length 6, three copies of a primitive string of length 4, etc. This says

and in general

Here the sum is over all positive integers *d* that divide *n*.

Unfortunately this formula is backward. It gives is a formula for something well known, 2^{n}, as a sum of things we’re trying to calculate. The Möbius inversion formula is just what we need to turn this formula around so that the new thing is on the left and sums of old things are on the right. It tells us that

where μ is the Möbius function.

We could compute *f*(*n*) with Python as follows:

from sympy.ntheory import mobius, divisors def num_primitive(n): return sum( [mobius(n/d)*2**d for d in divisors(n)] )

The latest version of SymPy, version 0.7.6, comes with a function `mobius`

for computing the Möbius function. If you’re using an earlier version of SymPy, you can roll your own `mobius`

function:

from sympy.ntheory import factorint def mobius(n): exponents = factorint(n).values() lenexp = len(exponents) m = 0 if lenexp == 0 else max(exponents) return 0 if m > 1 else (-1)**lenexp

The version of `mobius`

that comes with SymPy 0.7.6 may be more efficient. It could, for example, stop the factorization process early if it discovers a square factor.

Ever wonder what the rules were for when to use thou, thee, ye, or you in Shakespeare or the King James Bible?

For example, the inscription on front of the Main Building at The University of Texas says

Ye shall know the truth and the truth shall make you free.

Why *ye* at the beginning and *you* at the end?

The latest episode of The History of English Podcast explains what the rules were and how they came to be. Regarding the UT inscription, *ye* was the subject form of the second person plural and *you* was the object form. Eventually *you* became used for subject and object, singular and plural.

The singular subject form was *thou* and the singular object form was *thee*. For example, the opening lines of Shakespeare’s Sonnet 18:

Shall I compare thee to a summer’s day?

Thou art more lovely and more temperate.

Originally the singular forms were intimate and the plural forms were formal. Only later did *thee* and *thou* take on an air of reverence or formality.

- Common Math Symbols in HTML, XML, TeX, and Unicode
- Accented letters in HTML, TeX, and Microsoft Word
- Greek letters in HTML, XML, TeX, and Unicode
- Unicode resources

See also blog posts tagged LaTeX, HTML, and Unicode and the Twitter account TeXtip.

**Last week**: C++ resources

**Next week**: Special functions

Unicode often counts the same symbol (glyph) as two or more different characters. For example, Ω is U+03A9 when it represents the Greek letter omega and U+2126 when it represents Ohms, the unit of electrical resistance. Similarly, M is U+004D when it’s used as a Latin letter but U+216F when it’s used as the Roman numeral for 1,000.

The purpose of such distinctions is to capture semantic differences. One example of how this could be useful is increased accessibility. A text-to-speech reader should pronounce things the same way people do. When such software sees “a 25 Ω resistor” it should say “a twenty five Ohm resistor” and not “a twenty five uppercase omega resistor,” just as a person would. [1]

Making text more accessible to the blind helps everyone else as well. For example, it makes the text more accessible to search engines as well. As Elliotte Rusty Harold points out in Refactoring HTML:

Wheelchair ramps are far more commonly used by parents with strollers, students with bicycles, and delivery people with hand trucks than they are by people in wheelchairs. When properly done, increasing accessibility for the disabled increases accessibility for everyone.

However, there are practical limits to how many semantic distinctions Unicode can make without becoming impossibly large, and so the standard is full of compromises. It can be quite difficult to decide when two uses of the same glyph should correspond to separate characters, and no standard could satisfy everyone.

***

[1] Someone may discover that when I wrote “a 25 Ω resistor” above, I actually used an Omega (Ω, U+03A9) rather than an Ohm character (Ω, U+2126). That’s because font support for Unicode is disappointing. If I had used the technically correct Ohm character, some people would not be able to see it. Ironically, this would make the text *less* accessible.

On my Android phone, I can see Ω (Ohm) but I cannot see Ⅿ (Roman numeral M) because the installed fonts have a glyph for the former but not the latter.

***

This post first appeared on Symbolism, a blog that I’ve now shut down.

]]>- IEEE floating-point exceptions in C++
- Unraveling Strings in Visual C++
- C++ TR1 regular expressions
- Random number generation in C++

See also posts tagged C++

**Last week**: R resources

**Next week**: HTML, TeX, and Unicode

If you see any problems with a post, please let me know. You could send me an email, or leave a comment on the post. (For a while I had comments automatically turn off on older posts, but I’ve disabled that. Now you can comment on any post.)

For the first couple years, this blog didn’t have many readers, and so not many people pointed out my errors. Now that there are more readers, I find out about errors more quickly. But I’ve found some egregious errors in some of the older posts.

Thanks for your contribution to this blog. I’ve been writing here for almost seven years, and I’ve benefited greatly from your input.

]]>- R language for programmers
- Default arguments and lazy evaluation in R
- Distributions in R
- Moving data between R and Excel via the clipboard
- Sweave: First steps toward reproducible analyses
- Troubleshooting Sweave
- Regular expressions in R

See also posts tagged Rstats.

I started the Twitter account RLangTip and handed it over the folks at Revolution Analytics.

**Last week**: Emacs resources

**Next week**: C++ resources

They will follow a Poisson distribution with an average of two per day. (Times are truncated to multiples of 5 minutes because my scheduling software requires that.)

]]>

- Heads
- Tails
- Equal probability of heads or tails.

Each is reasonable in its own context. The last answer is correct assuming the flips are independent and heads and tails are equally likely.

But as I argued here, if you see nothing but heads, you have reason to question the assumption that the coin is fair. So there’s some justification for the first answer.

The reasoning behind the second answer is that tails are “due.” This isn’t true if you’re looking at independent flips of a fair coin, but it could reasonable in other settings, such as sampling without replacement.

Say there are a number of coins on a table, covered by a cloth. A fixed number are on the table heads up, and a fixed number tails up. You reach under the cloth and slide a coin out. Every head you pull out increases the chances that the next coin will be tails. If there were an equal number of heads and tails under the cloth to being with, then after pulling out 10 heads tails are indeed more likely next time.

**Related post**: Long runs

- Emacs kill (cut) commands
- Emacs point (cursor) movement
- Getting started with Emacs on Windows
- Notes on Unicode in Emacs

See also the Twitter account UnixToolTip and blog posts tagged Emacs.

**Last week**: Miscellaneous math notes

**Next week**: R resources

- A big part of being a statistician is knowing what to do when your assumptions aren’t met, because they’re never exactly met.
- A lot of statisticians think time series analysis is voodoo, and he was inclined to agree with them.