StreetEYE

“Cat Food” revisited – final thoughts – part 4

StreetEYE — Wed, 15 May 2013 16:05:51 +0000

Here is the long-awaited conclusion to the wonky 4-part discussion of safe retirement spending. We went pretty far down the rabbit hole, and I think the conclusions are useful.

Attempting to finance a stable retirement from a risky portfolio is inherently fraught with risk and tradeoffs.

A desirable strategy achieves

High initial and lifetime spending
Low frequency and severity of shortfalls
Low variability in spending

Increasing the level of spending today is desirable – but increases the risk of shortfall in the future. High variability in spending is bad – but upward adjustments let you increase spending to take advantage of better-than-expected investment performance, while downward adjustments reduce risk of even greater income shortfall when performance falls short. Finally, accepting spending variability in the future enables higher spending today, if you accept a thinner buffer against shortfall, and/or invest in higher-risk, higher-return portfolios.

The $64,000 question is, how risk-averse are you? How willing are you to trade current income for future income, variability and risk of shortfall?

If you’re risk-neutral, you just want maximum expected lifetime spending. This leads to a highly equity-oriented, volatile portfolio, and variable spending something like a consistent percentage of the portfolio, starting low and allowing growth in the portfolio and spending over time (our test added smoothing and mortality updating, but spending is still highly variable).
If you’re highly risk-averse, you want a highly diversified portfolio, and you’ll tend toward a traditional fixed 4% rule (which should probably be lower for conservative retirees in current market conditions).
If you’re between those extremes, your portfolio, optimal lifetime spending, variability, and shortfall risk will tend to lie between those two extremes. You might consider combining a fixed amount and variable amount to achieve the best balance between variability, shortfall risk, and high initial and lifetime spending.
What is more important than the specific initial rate is a strategy with an appropriate degree of spending flexibility, giving you the ability to take advantage of higher long-term equity returns without imposing intolerable variability, and a clear picture of the range of potential outcomes. To the extent possible, retirees should err on the side of moderate initial spending, embrace the volatility they can tolerate as the key to unlocking maximum lifetime spending, and accept that their retirement trajectory is ultimately dependent on how the timing of their retirement intersects long-term economic and market trends.

Here’s a slightly more technical discussion.

I got a little stuck trying to write a conclusion. First of all, I realized a constant spending parameter was needed for our model to make it more general, and felt compelled to run it again.

The updated model is

where:

s_i is spending in period i
P_i is the value of the portfolio for period i
L_i is the retiree’s remaining life expectancy in period i
K is the constant spending parameter
n is the exponential moving average smoothing parameter
h is the variable spending parameter
b is the mortality insensitivity parameter

Other than the addition of K, this is the model in our last post. If n is 1, b is large, and K is 0, this reduces to something like a fixed percentage spending model; if h is 0, this is a constant spending model like Bengen’s 4% rule.

I tested about 20 values for each parameter, and 21 allocations from 0% to 100% equity in 5% steps, ultimately about 3.7m combinations.

Mapping the initial spending ‘severity frontier’, we see that the highest initial spending with no shortfall is basically the Bengen solution: 3.9% spending, 50% stocks, mostly fixed spending.

Shortfall: the worst observed decline below initial spending. The ‘frontier’ is the lowest achievable shortfall for an initial spending rate.

As an aside, below is a plot of probability of shortfall at various levels. Reading across, you can see that at 6%, the best you can do is about a 20% chance of some shortfall and about 4% probability of 50% shortfall (not with identical portfolios). Also, you can see that for each 1% you increase initial spending, best achievable probability of shortfall at any cutoff goes up about 10%.

Mapping the lifetime spending ‘severity frontier’, we see that highest lifetime expected spending with no shortfall is 118.06% of the initial portfolio. This is notably higher than maximizing initial spending with no shortfall. But initial spending is low – 2.33% of the initial portfolio, which is all equity, like all the other solutions for maximum lifetime spending.

This exposes the essential dilemma::

If you want maximum initial spending with no shortfall, you have a low-risk portfolio, low lifetime spending. You leave a lot on the table.
If you want maximum lifetime spending with no shortfall, you have low initial spending, high equity allocation in a portfolio that grows over time and gives you back-loaded spending.

This doesn’t really help us pick an optimal point. We need a ‘Sabermetric’ to combine the ‘holy trinity’ of high spending, low variability, and low risk of shortfall.

There is one in the literature. It’s ‘certainty equivalent’ (CE) spending, which takes an income stream and applies a discount according to 1) its level of variability and risk, and 2) a risk aversion parameter. If risk aversion is 0, you’re risk-neutral, no variability discount is applied. The higher your risk aversion the higher the discount. The form we use is constant relative risk aversion (CRRA), which means a stream that varies between $1 and $2 gets the same discount as one with similar variation between $100 and $200.

We calculate certainty-equivalent lifetime spending with a risk aversion parameter of 8 – the lowest that gives us reasonably diversified portfolios. Here is what the ‘severity frontier’ looks like:

This gives us a higher initial income than maximizing lifetime spending, and a higher lifetime spending than maximizing initial income. For instance, the solution in bold has initial spending of 3.83%, 1.7% probability of shortfall > 10%, lifetime spending > 110% of initial portfolio. Compared to maximizing initial spending for 0 shortfall in Table 1 (3.9% initial spending, 70.7 lifetime spending), we achieve a > 50% increase in lifetime spending at a price of a < 2% decrease in initial spending and a worst-case decline of 12.7% from initial spending.

What we find is:

If we are completely risk neutral, our risk aversion parameter is 0 and we maximize lifetime spending…. we are all equities, low initial spending, high volatility.
If we are highly risk averse, we will aim for the highest sustainable fixed spending, similar to the Bengen ‘4% rule’.
If risk aversion is between these two extremes, we will seek a strategy with volatility and lifetime spending between these two extremes. As you increase risk aversion, the variable component of spending decreases and constant spending increases, the portfolio swings from all equities to a diversified mix, and initial spending generally decreases.

If you put full faith in this model and your estimate of risk aversion, you could just maximize certainty-equivalent spending. Or you might want to look up and down the ‘severity frontier’ and pick the best solution, one that combines high lifetime spending and CE spending with reasonable initial spending and shortfall risk.

In practice, maximizing certainty-equivalent spending depends on estimating risk aversion, which is not precisely knowable. We could query retirees on which potential outcomes and risk profiles they prefer, but that’s like describing two movies to them and asking which they would prefer. A portfolio outcome, like a movie, is an experience good, which can only be judged successful after experiencing the whole thing (although failures can often be discovered more quickly, or even from the description). Risk aversion may not even be consistent over time, but may increase abruptly at market extremes.

Nevertheless, certainty-equivalent spending has the advantage of being a consistent, objective measure that takes into account risk and variability, and it can identify a schedule of solutions at different levels of initial and lifetime spending and worst shortfall that are locally optimal, and you can pick the ones that look most desirable.

I wrote this up as a paper with all details, in case others want to delve into it and do more work on it, and will share code and data in this space shortly, as soon as I’ve cleaned it up a little.

I’ll throw out one other approach, which is to use a Kahneman-Tversky behavioral economics/prospect theory utility function, which is discontinuous and heavily penalizes losses from an initial frame of reference. That means it leads to inconsistent choices depending on the initial frame of reference and it’s path-dependent. Yet, it is highly predictive of human choices.

You could apply it to this problem by choosing the highest initial spending such that, if you set it as the initial frame of reference, future utility gains outweigh or equal losses. If you’re highly averse to losses, you’ll set initial spending low enough that the pain from small potential declines in income is offset by very large potential future gains. If you’re less loss averse, you’ll set initial spending a little higher, so moderate losses are offset by large future gains. We still have the thorny problem of estimating the degree of loss aversion. Maybe I’ll do a detailed exploration of that idea, but it might take a while, so I’ll just leave it out there, or if anyone wants to collaborate on a study like that, drop me a line.

‘Cat Food’ revisited – testing dynamic spending rules – Part 3

StreetEYE — Mon, 18 Feb 2013 16:51:58 +0000

In the last part of our look at dynamic rules for spending in retirement, we discussed how changing the allocation between stocks and bonds affects the maximum sustainable spending rate. We can summarize this relationship by plotting the highest feasible initial spending rate for any acceptable shortfall level.¹

Figure 15. Initial spending v. worst shortfall
equity=0-100%, s=0.05-1.2, smoothing=1.0-6.0)

Max shortfall	Init spend %	s	Equity %	Smoothing	Lifetime Spend Exp.
0.0%	1.7%	0.30	20%	5	58.0
4.3%	2.0%	0.35	40%	4	66.3
12.1%	2.3%	0.40	50%	3	77.4
16.9%	2.6%	0.45	60%	2.5	89.1
24.7%	2.9%	0.50	75%	2	105.9
32.2%	3.2%	0.55	80%	1.5	113.6
43.2%	3.5%	0.60	95%	1.5	132.5
50.4%	3.8%	0.65	95%	1	133.8
63.5%	4.1%	0.70	100%	1	141.0

In part 3 today, we’re going to discuss 1) adding a parameter to change the way we increase the spending rate as we get older and 2) reducing equity allocation as we get older. Finally, we’re going to do a brute-force search for the highest initial spending and best parameters for each level of shortfall risk (skip to conclusion).

Adjusting how fast spending rate increases with age

Suppose we are willing to accept the portfolio in bold and spending rate, which has never resulted in a decline greater than 24.7% below the initial spending amount. We use spending factor s=0.5, equity = 75%, smoothing factor = 2. We end up with a spending profile that looks like this:

Figure 16. Spending profile: s=0.5, equity=75%, smooth=2

Notice the ‘hump.’ In the average case, spending rises steadily, then drops. Ideally, we would like a flatter spending profile, preferring less variability in spending. If we could take the hump and spread part of it earlier, without increasing shortfall risk, it would increase the safe spending rate.

The spending rate at age 99 is s/life_expectancy=0.5/2.19 = 22.8%. Spending rates at advanced ages are too high, decimating the portfolio and causing the decline in spending. If we reduce them, maybe we don’t need to build as big a war chest in middle years to support such a high spending rate in late years.

There are several ways we could change the model to flatten the spending profile. The one we choose is to add a parameter we’ll call ‘life expectancy buffer’ to life expectancy, so we’ll spend at a rate of s/(life_expectancy+buffer). If the parameter is 2, then when our life expectancy is 3, instead of spending at a rate of s/3, we’ll spend at a rate of s/(3+2). If life expectancy buffer is 0 or low, the spending rate will rise relatively quickly as we age; if life expectancy buffer is large, the spending rate will change slowly as we age.

With a buffer of 2, we get a flatter spending profile later in life, without changing too much at the start of retirement.

Figure 17. Spending profile: s=0.5, equity=75%, smooth=2, lifespan buffer=2

Let’s compare efficient frontiers, with and without the buffer:

Figure 18. Efficient Frontier: Initial spending v. worst shortfall
Buffer=1 v. Buffer=0

We find in all cases, the buffer gives us a higher level of initial spending for any level of shortfall risk. However, increasing the buffer does not always increase lifetime spending and sometimes worsens the tradeoff.

Figure 19. Efficient Frontier: Lifetime spending v. worst shortfall
Buffer=1 v. Buffer=0

It’s a mixed picture. The buffer may in many cases allow us to spend safely at a higher initial rate, but doesn’t always improve lifetime spending. Sometimes it reduces portfolio growth and overall lifetime spending.

Third dynamic rule: Changing equity allocation over time.

Does it make sense to reduce the equity allocation as you get older? Let’s introduce a variable to gradually reduce the amount of equity. As long as your life expectancy is > 15 years, use the initial allocation. After that, we’ll reduce the equity percentage by stepdown percentage points per year.

Figure 20. Efficient frontiers: Initial spending v. worst shortfall
Equity stepdowns from 0 to 3% per year when LE<15 years.

Figure 21. Efficient frontiers: Lifetime spending v. worst shortfall
Equity stepdowns from 0 to 3% per year.

What I thought would happen: stepping down equity over time would let you start with more equity, relying on smoothing to get you over early drawdowns and strong long-run real returns to catch up. Then with lower equity in later years you wouldn’t be exposed to big drawdowns late in life.

What actually happened: stepping down equities always worsens the risk/reward tradeoff.

Apparently my preconception was faulty, or this is the wrong model, stepping down too early and killing returns, or there’s a mistake somewhere (although I don’t think so). Watch this space for possible future elaboration or ninja-edit. I’ll consider this a first draft and move on.

Discarding the stepdown, we now have 4 parameters: spending factor s, starting equity, smoothing factor, and lifetime buffer. We have discussed how each one impacts the spending profile and shortfall risk. Let’s put it all together, try running a full range of reasonable values, and come up with the best efficient frontier over all values.

Figure 22. Efficient frontiers – all parameter values
(s=.05-1.2; equity=0-100%; smooth=1-6; buffer=0-5)

Max shortfall	Init spend %	s	Equity %	Smoothing	Buffer	Lifetime Spend Exp.
0.0%	2.1%	0.40	20%	6.0	2	55.5
1.6%	2.3%	0.45	35%	6.0	2	64.4
4.8%	2.5%	0.50	40%	6.0	3	66.9
7.2%	2.6%	0.55	40%	6.0	4	67.18
7.4%	2.7%	0.55	55%	6.0	3	79.0
9.8%	2.8%	0.60	55%	6.0	4	78.7
11.8%	2.9%	0.65	55%	6.0	5	78.5
13.4%	3.1%	0.65	75%	6.0	4	95.3
15.4%	3.2%	0.70	75%	6.0	5	94.6
20.1%	3.4%	0.75	95%	6.0	5	114.1
27.3%	3.5%	0.70	90%	3.5	3	117.6
27.8%	3.5%	0.75	80%	3.5	4	106.7
28.4%	3.6%	0.80	85%	4.0	5	109.7
31.0%	3.7%	0.75	75%	2.5	3	106.2
32.3%	3.8%	0.80	85%	3.0	4	113.8
34.0%	3.8%	0.85	90%	3.5	5	117.1
35.1%	4.0%	0.80	95%	2.5	3	127.8
37.0%	4.0%	0.85	90%	2.5	4	120.8
38.6%	4.1%	0.90	90%	3.0	5	119.4
39.0%	4.2%	0.85	95%	2.0	3	129.3
40.7%	4.2%	0.90	95%	2.0	4	127.6
42.3%	4.3%	0.95	90%	2.5	5	121.2
44.0%	4.5%	0.95	100%	2.0	4	134.5
45.5%	4.5%	1.00	100%	2.0	5	132.9
48.1%	4.7%	1.05	95%	1.5	5	128.6
50.6%	5.0%	1.10	100%	1.5	5	134.8

Let’s look at the rule in bold. (s=0.70, equity=75%, smooth=6.0, buffer=5)

Figure 23. Average Spending, All Cohorts (s=0.7, eq=75%, smooth=6, buffer=5)

What have we learned?

When you retire matters. A lot.
The 4% rule doesn’t really work with a restrictive large cap/long term bond portfolio. The best solution is 95% equity and you still get a worst case 35% shortfall.
< 3.5% has generally had a tolerable worst-case, and on average lets your spending grow over time to > 4% of the initial portfolio.
Even at 3.2-3.5%, a dynamic rule gives you higher lifetime spending than a fixed 4% rule, and the failure scenarios are gradual, not sudden and catastrophic.
A large equity allocation is desirable for real long-term growth, and stepping down the amount of equity as you get older doesn’t seem to improve risk-reward (at least, not a fixed annual stepdown when life expectancy is < 15 years).
Taxes and investment expenses aren’t included and must be taken into account, reducing returns and the level of safe spending.
On the other hand, including higher-risk, higher-return investments, like small caps, international equities, higher-yielding bonds should improve risk-reward and allow higher returns and spending.
Of course, this analysis assumes the future will not be very different from the past.
Finally, I would add that life annuities, in theory, could be a free lunch. In the analysis above, you put aside 100% of the money you may need in the 25% chance you live to be 90; alternatively, you could join an annuity pool which only needs to put aside money for the survivors, which also has tax advantages. However, fixed annuities are subject to inflation and credit risk, and the market for insurance products for the elderly is a jungle teeming with predators.

These dynamic rules are chosen for analytical simplicity, to be easy to understand and calculate. There’s no reason to believe that they are the best, ie spending a constant/(life expectancy+buffer) is the ideal model to get the highest spending rate at each point in time for the lowest shortfall risk.

In the last part of this series, we’ll think about what an ideal model would look, and how to maximize spending while keeping shortfall risk at an acceptable level.

¹On further review, the table in Part 2 isn’t the best way to look at it. No one thinks about the best spending factor they can achieve, they think about the initial spending %. And people don’t think about ‘lowest probability of 10% shortfall.’ Maximum acceptable shortfall is simpler and more informative.

‘Cat Food’ revisited – testing dynamic spending rules – Part 2

StreetEYE — Sun, 03 Feb 2013 21:08:57 +0000

The last post discussed a framework for evaluating simple dynamic spending rules.

We defined a spending factor s as spending each year at a rate of s/(remaining life expectancy); and lifetime spending expectancy as the total amount you could expect to spend over your lifetime.
We showed how, as you increase the spending factor, lifetime spend expectancy initially increases rapidly, but the curve flattens out as spending rate increases.
We showed how, as you increase the spending factor, shortfall risk initially increases slowly, but the curve steepens as spending rate increases.

We found a simple dynamic spending rule could increase lifetime spending vs. a traditional fixed 4% rule, while keeping shortfall risk relatively low (arguably reducing risk by making the worst case more benign, at the cost of increased volatility, lowered starting spending, higher probability of modest shortfalls).

In this post we’ll look at how smoothing spending can improve outcomes, and how changing the equity/bond mix over time affects outcomes.

Second dynamic rule: Smoothing.

One oddity in Figure 5: even at ultra-low spending rates, you still have a measurable risk of a 25% shortfall from your starting income. At any spending rate, spending is going to vary more or less in line with the market¹. So years like 1973 and 1974 might yield big spending drops, even for a 60/40 portfolio at spending rates where you are not at much risk of running out of funds in your lifetime.

If you’re already spending at a low rate, based on a safe long run return expectation, it doesn’t make sense to vary spending exactly in line with a short term change in the markets. And it’s clearly a significant disadvantage over a 4% rule to have that much variation in spending, even when there’s no shortfall.

Let’s introduce smoothing. We’ll still spend at a rate of s/life_expectancy, but instead of multiplying the rate by the latest portfolio value to get spending, we’ll multiply the rate by an exponential moving average² of the value of previous years’ portfolios.

Figure 8. Lifetime spending expectancy (% of initial portfolio) v. spending factor (exponential MA, multiplier 1/3).

Figure 9. Probability of spending shortfall v. spending factor (exponential MA, multiplier 1/3)

We see a notable improvement – near 0 shortfall risk out to s of 0.5. As the next chart shows, at s of 0.5, we have slightly higher lifetime spending, smoother overall spending and a better-looking worst case. But, even though overall shortfall probability is much lower, in the worst case you can go broke, if you keep overall spending smooth in the face of a market shortfall very late in life.

Figure 10. Average Spending, All Cohorts (s=0.5, smoothed w/multiplier=1/3)

Efficient frontier³. To better visualize the tradeoff, we can combine these 2 graphs into one, which shows the best lifetime spend expectancy we can achieve for any level of shortfall risk:

Figure 11. Lifetime spend expectancy v. probability of 10% shortfall
(‘Efficient frontier’)

For low shortfall rates (the left side), the smoothed spending results in higher lifetime spending expectancy with lower shortfall risk. Smoothing improves the tradeoff, and represents a flexible middle ground between varying spending in line with the market, and setting a fixed spending amount at retirement like the 4% rule.

Portfolio composition

So far, we have looked only at a 60/40 portfolio. Let’s keep the smoothing multiplier at 1/3 and create efficient frontiers for a range of allocations from 0% to 100% stocks.

Figure 12. Lifetime spend expectancy v. probability of 10% shortfall
(‘Efficient frontier’), by % of equities, smoothing multiplier=1/3

We can do this as a big panel (easier to read, esp. the legend): Figure 13. What does this tell us? For this particular smoothing multiplier 1/3, the 50/50 portfolio dominates others (the black line), giving greater lifetime spending at low shortfall probabilities.

Let’s cut to the chase and find for each spending factor, the highest spending expectancy we can achieve with zero shortfall risk, or if zero risk is not feasible, the lowest feasible risk:

s	Life spend expect.	Prob 10% shortfall	Prob 25% shortfall	Prob 50% shortfall	Equity %	Smoothing
0.30	104.27	0.0%	0.0%	0.0%	100%	5.0
0.35	90.71	0.0%	0.0%	0.0%	75%	4.0
0.40	77.62	0.1%	0.1%	0.1%	50%	3.5
0.45	89.74	0.9%	0.9%	0.9%	60%	4.0
0.50	97.99	2.6%	2.6%	2.4%	65%	4.5
0.55	110.23	5.4%	3.3%	2.9%	75%	4.0
0.60	122.56	8.8%	8.4%	8.1%	85%	6.0
0.70	126.98	19.3%	13.5%	11.9%	85%	5.0
0.80	124.34	28.5%	23.9%	22.1%	80%	6.0
0.90	134.70	36.9%	32.6%	29.0%	90%	6.0
1.00	139.22	43.3%	39.6%	36.0%	95%	6.0

That’s pretty much all we have time for this week. But before we go, let’s look at the spending profiles for 4 solutions.

Figure 14. Spending profiles for selected spending factors

Spending factor 0.35; Shortfall risk 0%; Lifetime spend expectancy 90.7; Initial spend 2.0%

Spending factor 0.45; Shortfall risk 0.9%; Lifetime spend expectancy 89.7; Initial spend 2.6%

Spending factor 0.5; Shortfall risk 2.6%; Lifetime spend expectancy 98.0; Initial spend 2.9%

Spending factor 0.55; Shortfall risk 5.4%; Lifetime spend expectancy 110.2; Initial spend 3.2%

To be 100% safe, we need an initial spending rate below 4% – keeping in mind that the portfolio and spending usually rises over time at safe withdrawal rates. With this restrictive investment universe, the 4% rule is not 100% safe either, and the last chart is a less undesirable spending profile than the 4% fixed spending profile from the last post. On average, total spending is significantly higher, and the worst case is more desirable since it falls short later.

Next time, we’ll look at varying the equity percentage dynamically as you get older, other dynamic rules that might reduce late in life shortfall risk (and the undesirable hump in these curves), and we’ll talk about what the best possible optimal rule would look like.

¹modulo the increase in spending rate as you age, from our s/life_expectancy rule. The s spending rate stays the same, but the denominator decreases.

²The exponential moving average has a good combination of sensitivity to the underlying series, and smoothing. A simple n-period moving average, which weights all n periods equally, has undesirable properties. In particular, it’s possible for a simple moving average to move up, when the underlying series is moving below the moving average. Consider this series: 100, 110, 120, 109. A 3-period moving average is after the 3rd element is 110, after the fourth is 113, because the trailing element which left the series (100) is lower than the new element (109).

³This is not exactly the same as the CAPM efficient frontier. For one thing, instead of return, we plot lifetime spending in inflation-adjusted dollars. For another, instead of volatility, it’s the probability of a 25% shortfall from the starting spending. It’s a similar risk v. return plot, but can’t be interpreted quite the same way (eg adding risk-free cash to find additional feasible/optimal portfolios).

‘Cat Food’ revisited – testing dynamic spending rules – Part 1

StreetEYE — Sat, 26 Jan 2013 01:08:34 +0000

How much can you safely spend out of a portfolio in retirement? Spend conservatively and you may be unnecessarily curbing the lifestyle and aspirations of you and your loved ones. Overspend and risk a shortfall and painful adjustment – in the extreme, the (hopefully apocryphal) “cat food” diet.

A traditional rule of thumb is a fixed 4% per year of your starting portfolio, adjusted each year for inflation. A previous post discussed why this rule may not be safe:

Low bond yields – 1.8% for 10-year Treasurys and negative TIPS out to 10 years – mean historical bond returns are mathematically unobtainable.¹
2.2% real returns since 2000 on a 60/40 blended portfolio suggest that long-run return expectations need to be revisited. Low long-term interest rates are a forecast of low future returns, ie low growth and inflation expectations. To the extent equity risk premiums haven’t widened, they forecast lower than normal equity returns.
Taxes and investment expenses must be included. Work supporting 4% tends to ignore them.
US demographics are not very positive for growth, inflation, tax rates, and hence, real after-tax investment returns (which is reflected in the US fiscal position). The US dependency ratio is forecast to rise by 15 points over the next 20 years.

If the 4% rule hasn’t been decisively breached, forward-looking indicators are a bit worrisome. Could a more flexible rule not only be safer, but in favorable circumstances allow a higher level of spending? In this 3-part post, we test dynamic rules that vary withdrawal rates based on age and the size of the portfolio, and vary the composition of the portfolio over time.

Dynamic rule 1: Vary spending by age.

The first rule we’ll test is to spend a percentage of the actual portfolio each year (not a fixed percentage of the initial portfolio) and vary the percentage by age.

Suppose you are a 65 year old male. You have a life expectancy of 17.19 years per US government actuarial tables. It would make sense to vary spending as a percentage of your assets inversely with your life expectancy. As a base case you could spend 1/17.19=5.82% of your portfolio this year. Next year your life expectancy would be 16.48% You would spend 1/16.48=6.07%. This is higher than 4% for life expectancy < 25 years, but it’s an arbitrary base case – just a starting point to test a rule based on the size of the portfolio and life expectancy.

Let’s apply this rule to a 60/40 stock/bond portfolio for someone who retired in 1928 to and see what spending it would have supported.

Figure 1. Inflation-adjusted spending for a 65-year-old single male who retires in 1928, with dynamic spending rule of 1/life_expectancy (spending factor=1) v. survival rate

This retiree would have experienced volatility, but he would really have started to go broke after around 1947, aged 85, shortly after his original life expectancy at retirement. From the blue survival line (from current life expectancy tables), over 40% would still have been alive for that drop. If he lived to be 100 after 35 years in 1963, he would have been penniless.

Is this rule too spendthrift, or was 1928 a particularly bad year to start retirement? Let’s try the same rule in all available 35-year retirement cohorts 1928-1977, and plot their average spending.

Figure 2. Average spending, 60/40 portfolio, spending factor 1, retirement age 65, all cohorts

The middle blue line is the average income by retirement year. The green and red are the best and worst cases. The middle 2 lines represent the +/- 1 standard deviation confidence interval.

Even in the best case, you eventually go broke. Your life expectancy is 17.2 years at retirement, and on average your spending goes below the starting amount around year 19. This spending rule may not conservative enough.

Let’s call s spending factor, and try different spending rates s/life_expectancy. We can run spending factors between 0.05 and 1.2: Figure 3. Big chart panel (opens in new window). As we move toward the top of the page, we see safer profiles, and even the worst case scenarios start to seem fairly acceptable.

This exercise demonstrates the tradeoff between spending, and the risk of the average or worst-case income path exhibiting a major shortfall from the starting income.

To better visualize this big panel, we can estimate ‘lifetime spend expectancy’ and shortfall probability for each spending factor.

Figure 4. Lifetime spending expectancy (% of initial portfolio) v. spending factor

In Figure 4, for each year, we multiply the spending outcome by the probability of a cohort retiree surviving to that year, and sum up the years, to get the lifetime spend expectancy for a given spending factor. This summarizes each line in the Figure 3 panel as a single point, the expected lifetime spending as a percentage of starting portfolio.

Figure 5. Probability of spending shortfall v. spending factor.

In Figure 5, for each s, we calculate the percentage of retirees in all cohorts who survived to a year where the spending falls to 25% below the initial spending.

These last 2 charts illustrate that as you increase the spending factor past 0.6, increasing lifetime spend expectancy flattens out, and the probability of 25% shortfall accelerates sharply.

Finally, how does the fixed 4% rule compare? For a 65-year-old with a 60/40 portfolio, the 4% rule yields 70.4 expected lifetime spending with a 2% lifetime shortfall probability.

By comparison, a 0.5 spending factor, which starts at about 3% spending, yields expected spending of 91.2 with a 9.7% probability of a 25% drop from the initial spend amount, and a worst case drop of 37% (for 1973 retirees – they eventually recovered).

But you do start at a lower rate, and spending is variable.

Figure 6. Average spending by retirement year, all cohorts, spending factor 0.5.

Figure 7. Average spending by retirement year, all cohorts, fixed 4% rule

In the next post, we’ll test additional rules, to vary the composition of the portfolio by age, and to try smooth spending.

¹The objection has been made that today’s rates can go lower and bond returns can be higher in the short run. True, but a 10-year bond bought at a yield of 1.8% will return, best case, 1.8% nominal over its lifetime (less in the event of default). If the yield goes to zero this year, it will return 18% this year, and zero over the rest of its lifetime. It’s called “fixed” income for a reason. When interest rates are below inflation, thinking bonds can be a real total return instrument in the base case is setting up for disappointment. It hasn’t been true historically, and it’s not what the market is pricing in. Bonds still have an important role as a liquidity and deflation hedge.

^?View Code RSPLUS

ageframe <- function (lifetable, age, maxage)
{
    start = age+1
    end=maxage+1
    keep <- c("Age","Mmort","Mlives","MLE")
    retframe = lifetable[start:end,keep]
    retframe$survivepct=retframe$Mlives/retframe[1,"Mlives"]
    return(retframe)
}
 
cohortframe <- function (ageframe, portframe, startyear)
{
    retframe <-ageframe
    retireyears=nrow(ageframe)
    endyear = startyear+retireyears-1
    retframe$portreturn = portframe[startyear:endyear,"portreturn"]
    retframe$year = portframe[startyear:endyear,"Year"]
    return(retframe)
 
}
 
calcspending <- function(cohortframe, spendingfactor)
{
    retframe <- cohortframe
    retframe$spend=0
    retframe$startportval=0
    retframe$endportval=0
    portval=100
    for (index in 1: nrow(retframe)) {
        myrow = retframe[index,]
        myrow["startportval"]=portval
        spendrate = spendingfactor/myrow["MLE"]
        spend = spendrate * portval
        myrow["spend"]=spend
        portval = portval - spend
        portval = portval * (1 + myrow["portreturn"])
        myrow["endportval"]=portval
        retframe[index,]=myrow
    }
    return(retframe)
}
 
calcspendingfixed <- function(cohortframe, spendingfactor)
{
    retframe <- cohortframe
    retframe$spend=0
    retframe$startportval=0
    retframe$endportval=0
    portval=100
    for (index in 1: nrow(retframe)) {
        myrow = retframe[index,]
        myrow["startportval"]=portval
        if ( portval > 4) {
            spend = 4
        } else {
            spend <- portval
        }
        myrow["spend"] <- spend
 
        portval = portval - spend
        portval = portval * (1 + myrow["portreturn"])
        myrow["endportval"]=portval
        retframe[index,]=myrow
    }
    return(retframe)
}
 
testspendingfactor <- function(realreturns, spendingfactor, minage, maxage)
{
    tempageframe <- ageframe(lifetable, minage, maxage)
    ntrials <- nrow(realreturns)-nrow(tempageframe) +1
    nyears <- maxage-minage+1
    trials <- data.frame(1:nyears)
    colnames(trials) = "age"
    for (index in 1:ntrials) {
        tempframe1 = cohortframe(tempageframe,realreturns,index)
        tempframe2 = calcspending(tempframe1,spendingfactor)
        colname = paste("trial",index, sep="")
        trials[colname] <- tempframe2$spend
    }
 
    trials <- trials[2:ncol(trials)]
    return(trials)
 
}
 
testspendingfactorfixed <- function(realreturns, spendingfactor, minage, maxage)
{
    tempageframe <- ageframe(lifetable, minage, maxage)
    ntrials <- nrow(realreturns)-nrow(tempageframe) +1
    nyears <- maxage-minage+1
    trials <- data.frame(1:nyears)
    colnames(trials) = "age"
    for (index in 1:ntrials) {
        tempframe1 = cohortframe(tempageframe,realreturns,index)
        tempframe2 = calcspendingfixed(tempframe1,spendingfactor)
        colname = paste("trial",index, sep="")
        trials[colname] <- tempframe2$spend
    }
 
    trials <- trials[2:ncol(trials)]
    return(trials)
}
 
calctrialssummary <- function(trials)
{
    trialssummary <- data.frame(1:nrow(trials))
    colnames(trialssummary) = "year"
    trialssummary$mean=apply(trials,1,mean)
 
    trialssummary$max=apply(trials,1,max)
    trialssummary$min=apply(trials,1,min)
    trialssummary$sd=apply(trials,1,sd)
    trialssummary$plus1sd=trialssummary$mean + trialssummary$sd
    trialssummary$minus1sd=trialssummary$mean - trialssummary$sd
    keep <- c("year", "mean","min","max","minus1sd","plus1sd")
    trialssummary <- trialssummary[,keep]
 
    return(trialssummary)
}
 
 
calcexpectedspending<- function(trials, survival)
{
    trialssummary <- data.frame(1:nrow(trials))
    colnames(trialssummary) = "year"
 
    trialssummary$survivepct=survival$survivepct
    trialssummary$mean=apply(trials,1,mean)
    trialssummary$expmean= trialssummary$mean * trialssummary$survivepct
    trialssummary$max=apply(trials,1,max)
    trialssummary$expmax= trialssummary$max * trialssummary$survivepct
    trialssummary$min=apply(trials,1,min)
    trialssummary$expmin= trialssummary$min * trialssummary$survivepct
    trialssummary$sd=apply(trials,1,sd)
    trialssummary$plus1sd=trialssummary$mean + trialssummary$sd
    trialssummary$expplus1sd= trialssummary$plus1sd * trialssummary$survivepct
    trialssummary$minus1sd=trialssummary$mean - trialssummary$sd
    trialssummary$expminus1sd= trialssummary$minus1sd * trialssummary$survivepct
 
    keep <- c("year", "expmean","expmin","expmax", "expplus1sd", "expminus1sd")
    trialssummary <- trialssummary[,keep]
    return(trialssummary)
}
 
chartsummary <- function(trialssummary, mytitle)
{
    meltframe <- melt(trialssummary, id = 'year')
    ggplot(data=meltframe, aes(x=year, y=value, colour=variable)) +
     scale_x_continuous() +
     ylab("Annual spending (% of initial portfolio)") +
     xlab("Retirement year") +
     theme_bw() +
     geom_line(size=1.4) +
     opts(legend.position="top",
          legend.direction = 'horizontal',
          plot.background = theme_rect(colour = 'black', fill = '#CCCCEE', size = 1, linetype='solid')) +
     scale_colour_manual(mytitle, breaks = c('mean', 'min','max','minus1sd','plus1sd'),
                         values = c("#000099", "#CC0000","#009900","#999999","#999999"),
                        labels = c('Average Spend', 'Minimum', 'Maximum', '-1 SD', '+1SD'))
}
 
library(ggplot2)
 
returns <- read.csv("~/Documents/returns.csv")  # http://blog.streeteye.com/blog/wp-content/uploads/2013/01/returns.csv
lifetable <- read.csv("~/Documents/Lifetable.csv") # http://blog.streeteye.com/blog/wp-content/uploads/2013/01/Lifetable.csv
 
returns$realstocks=returns$Stocks-returns$CPI
returns$realbonds=returns$Bonds-returns$CPI
returns$realbills=returns$Bills-returns$CPI
 
drops <- c("Stocks","Bonds","Bills","CPI")
realreturns <- returns[,!(names(returns) %in% drops)]
realreturns$portreturn = 0.6*realreturns$realstocks + 0.4 *realreturns$realbonds
 
#Figure 1
 
tempageframe <- ageframe(lifetable, 65, 100)
tempframe1 = cohortframe(tempageframe,realreturns,1)
tempframe2 = calcspending(tempframe1,1)
tempframe2$survival65 = tempframe2$survivepct * 10000
tempframe2$spend = tempframe2$spend * 1000
keep = c("year","spend","survival65" )
tempframe2 <- tempframe2[,keep]
meltframe <- melt(tempframe2, id = 'year')
 
ggplot(data=meltframe, aes(x=year, y=value, colour=variable)) +
     scale_x_continuous() +
     ylab("Spend") +
     xlab("Year") +
     theme_bw() +
     geom_line(size=1.4) +
     opts(legend.position="top",
          legend.direction = 'horizontal',
          plot.background = theme_rect(colour = 'black', fill = '#CCCCEE', size = 1, linetype='solid')) +
     scale_colour_manual("Annual spending, retirement age 65, 60/40 portfolio, spending factor 1, 1928 cohort", breaks = c('spend', 'survival65'),
                         values = c("#CC0000", "#000099"),
                        labels = c('Annual Spend', 'Survivors from starting 10,000'))
 
 
#Figure 2
trials = testspendingfactor(realreturns, 1, 65, 100)
trialssummary = calctrialssummary(trials)
trialssummary$survival = ageframe65$survivepct*10000
keep <- c("year", "mean","min","max","minus1sd","plus1sd")
trialssummary <- trialssummary[,keep]
chartsummary(trialssummary, "Average Spending, All Cohorts")
 
 
# Figure 3
facetframe = data.frame (year = integer(), variable=character(), value=double())
 
for (index in 1:24) {
    spendingfactor = index/20
    trials = testspendingfactor(realreturns, spendingfactor, 65, 100)
    trialssummary = calctrialssummary(trials)
    trialssummary$spendingfactor = spendingfactor
    meltframe <- melt(trialssummary, id = c('year','spendingfactor'))
    facetframe <- merge(meltframe, facetframe, all=TRUE)
}
 
ggplot(data=facetframe, aes(x=year, y=value, colour=variable)) +
    scale_x_continuous() +
    ylab("Annual spending (% of initial portfolio)") +
    xlab("Retirement year") +
    theme_bw() +
    facet_wrap(~ spendingfactor, ncol = 4) +
    geom_line(size=1) +
    opts(legend.position="top",
         legend.direction = 'horizontal',
         plot.background = theme_rect(colour = 'black', fill = '#CCCCEE', size = 1, linetype='solid')) +
scale_colour_manual("Average Spending, All Cohorts, Spending Factors 0.05 to 1.2", breaks = c('mean', 'min','max','minus1sd','plus1sd'),
                    values = c("#000099", "#CC0000","#009900","#999999","#999999"),
                    labels = c('Average Spend', 'Minimum', 'Maximum', '-1 SD', '+1SD'))
 
############################################################
 
 
ageframe65 = ageframe(lifetable,65,100)
nsfactors = 40
sfactorsummary <- data.frame(1:nsfactors)
colnames(sfactorsummary) = "year"
sfactorsummary$sfactor <- 0
sfactorsummary$expspend <- 0
sfactorsummary$minspend <- 0
sfactorsummary$maxspend <- 0
sfactorsummary$plus1sd <- 0
sfactorsummary$minus1sd <- 0
sfactorsummary$sdspend <- 0
sfactorsummary$shortfall <- 0
 
for (index in 1:nsfactors) {
    spendingfactor = index/20
    trials = testspendingfactor(realreturns, spendingfactor, 65, 100)
    trialssummary = calcexpectedspending(trials, ageframe65)
 
    sfactorsummary$sfactor[index] = spendingfactor
    sfactorsummary$expspend[index] = sum(trialssummary[,"expmean"])
    sfactorsummary$minspend[index] = sum(trialssummary[,"expmin"])
    sfactorsummary$maxspend[index] = sum(trialssummary[,"expmax"])
    sfactorsummary$plus1sd[index] = sum(trialssummary[,"expplus1sd"])
    sfactorsummary$minus1sd[index] = sum(trialssummary[,"expminus1sd"])
    sfactorsummary$sdspend[index]=rowMeans(apply(trials, 2, sd)/trials[1,])
 
    trialstmp=trials
    for (index2 in 1:ncol(trialstmp))
        trialstmp[index2]=trialstmp[index2]/trialstmp[1,index2]
    trialstmp[trialstmp<0.75]=-1
    trialstmp[trialstmp>0]=0
    trialstmp[trialstmp<0]=1
    for (index2 in 1:ncol(trialstmp))
        trialstmp[index2]=trialstmp[index2] * ageframe65$survivepct
    sfactorsummary$shortfall[index]=mean(apply(trialstmp,2,max))
}
 
 
chart1=sfactorsummary
keep <- c("sfactor","expspend","minspend","maxspend","plus1sd","minus1sd")
chart1 <- chart1[keep]
 
meltframe <- melt(chart1, id = 'sfactor')
 
ggplot(data=meltframe, aes(x=sfactor, y=value, colour=variable)) +
    scale_x_continuous() +
    ylab("Lifetime spend expectancy (% of initial portfolio)") +
    xlab("Spending factor") +
    theme_bw() +
    geom_line(size=1.4) +
    opts(legend.position="top",
         legend.direction = 'horizontal',
         plot.background = theme_rect(colour = 'black', fill = '#CCCCEE', size = 1, linetype='solid')) +
scale_colour_manual("Lifetime Spend Expectancy v. Spending Factor", breaks = c('expspend', 'minspend','maxspend','plus1sd','minus1sd'),
                    values = c("#000099", "#CC0000", "#009900","#999999","#999999"),
                    labels = c('Mean', 'Worst investment outcome', 'Best investment outcome','+1SD','-1SD'))
 
chart2=sfactorsummary
keep <- c("sfactor","shortfall")
chart2 <- chart2[keep]
meltframe <- melt(chart2, id = 'sfactor')
 
ggplot(data=meltframe, aes(x=sfactor, y=value, colour=variable)) +
    scale_x_continuous() +
    ylab("Shortfall probability") +
    xlab("Spending factor") +
    theme_bw() +
    geom_line(size=1.4) +
    opts(legend.position="top",
         legend.direction = 'horizontal',
         plot.background = theme_rect(colour = 'black', fill = '#CCCCEE', size = 1, linetype='solid')) +
scale_colour_manual("Probability of 25% Spending Shortfall v. Spending Factor", breaks = c('shortfall'),
                    values = c("#000099"),
                    labels = c('Shortfall probability'))
 
 
# fixed 4% rule
trials = testspendingfactorfixed(realreturns, 1, 65, 100)
trialssummary = calctrialssummary(trials)
ageframe65 = ageframe(lifetable,65,100)
trialssummary$survivepct=ageframe65$survivepct
trialssummary$expmean = trialssummary$mean * trialssummary$survivepct
sum(trialssummary$expmean)
 
trials[trials<4]=-1
trials[trials>0]=0
trials[trials<0]=1
trials[,]=trials[,] * ageframe65$survivepct
mean(apply(trials,2,max))
 
# 0.5 spending factor -> look up from sfactorsummary

What’s the worst that could happen?

StreetEYE — Sat, 19 Jan 2013 04:45:29 +0000

It’s not whether you get knocked down, it’s whether you get up. – Vince Lombardi

Playing around with DataNitro, an add-in that lets you run Python in Excel¹.

What is the worst that could happen to someone who owns stocks, bonds, bills, over a 1-, 5-, 10-year time-frame? Here are the worst rolling periods for each asset class for 1928-2010, adjusted for inflation.

	Stocks		Bonds		Bills
1-year	-38.2%	1937, 1974	-15.5%	1980	-17.8%	1946
2-year	-52.5%	1972-1974	-26.2%	1978-1980	-24.6%	1945-1947
5-year	-44.7%	1936-1941	-37.5%	1976-1981	-27.3%	1945-1950
10-year	-37.5%	1964-1974	-43.2%	1971-1981	-43.9%	1940-1950
20-year	10.7%	1961-1981	-40.8%	1961-1981	-48.9%	1932-1952
30-year	243.5%	1964-1994	-39.3%	1950-1980	-43.4%	1932-1962

Worst case real returns for rolling periods from 1 to 30 years, 1928-2010

Over short timeframes, stocks can do quite a bit worse. The worst 2-year period is -52.5% for stocks, v. -26% for bonds and -25% for bills. Around year 8, stocks cross over. The worst 20-year period for stocks sees you up 10.7%, and the worst 30-year period sees you up 243%! When bonds and bills get hurt by inflation, they stay down for very long periods.

Rerunning the analysis for the post-war era doesn’t change much. Most of the worst-case periods for stocks and bonds were after 1946, but bills did worst around the war and better afterwards.

Worst case real returns for rolling periods from 1 to 30 years, 1946-2010

Spreadsheet here.

^?View Code PYTHON

import numpy as np # not used in this example, but works!
 
def rolling_return(series, n):
    "given a series of m returns, compute m-n rolling n-period returns"
    m = len(series)
    retarray = []
    for i in range(m-n):
        rr = 1.0
        for j in range(n):
            rr = rr * (1+ series[i+j])
        retarray.append(rr-1)
    return retarray
 
active_sheet("Returns")
stocks = CellRange("Equities").value
bonds = CellRange("Bonds").value
bills = CellRange("Bills").value
cpi = CellRange("CPI").value
 
realbonds = [bonds[i]-cpi[i] for i in range(len(cpi))]
realbills = [bills[i]-cpi[i] for i in range(len(cpi))]
realstocks = [stocks[i]-cpi[i] for i in range(len(cpi))]
 
active_sheet("Data_1928")
for i in range(1,31):
    tempbonds = rolling_return(realbonds,i)
    tempbills = rolling_return(realbills,i)
    tempstocks = rolling_return(realstocks,i)
    Cell(i+1,2).value = min(tempstocks)
    Cell(i+1,3).value = min(tempbonds)
    Cell(i+1,4).value = min(tempbills)
 
active_sheet("Data_1946")
 
realbonds46=realbonds[18:]
realbills46=realbills[18:]
realstocks46=realstocks[18:]
 
for i in range(1,31):
    tempbonds = rolling_return(realbonds46,i)
    tempbills = rolling_return(realbills46,i)
    tempstocks = rolling_return(realstocks46,i)
    Cell(i+1,2).value = min(tempstocks)
    Cell(i+1,3).value = min(tempbonds)
    Cell(i+1,4).value = min(tempbills)

¹Why is Python a good thing? Lots of very powerful packages for data manipulation, optimization, statistical analysis, machine learning are available in Python. Also, Python is a powerful, expressive, readable language that makes it easy to manipulate complex data structures.

‘Big Data’

StreetEYE — Sat, 05 Jan 2013 12:24:47 +0000

If ‘The Graduate’ were made today, Benjamin Braddock might hear a well-meaning uncle stage-whisper ‘Big Data’ instead of ‘Plastics.’ (Runners-up: ‘The Cloud’, ‘Social Discovery’, ‘Gamification’, the list goes on.) ‘Big data’ is a buzzword that people throw around a lot. What does it mean? Large data sets are not new. The IRS, the Census, Walmart, money center banks have always had big data sets.

What’s changed?

Look at Google. You type ‘Justin,’ and within milliseconds it offers ‘Bieber’ as a completion. You hit enter, and it fires off a query to a bunch of computers (guessing low hundreds) that each have part of the Google index in RAM. They return answers, and the results are ranked. Then the page ids are fired off to a different set of computers that has the entire text of the relevant Web pages in RAM, and the relevant excerpts are retrieved. Finally, a Search Engine Results Page (SERP) gets built, with personalized ads etc. All this happens in milliseconds.

This was a revolutionary approach to working with large datasets. What’s changed is not just that there are more large datasets, but also, the tools Google pioneered have gone mainstream, and new computing stacks let you apply powerful algorithms in real time.

Near the bottom of the stack is the database. Oracle databases can handle any amount of data. Could Google have been built on top of Oracle? Probably. Should they have? No. First, Oracle’s business model isn’t designed for it, licensing for millions of servers would cost a fortune! Second, Oracle is poorly adapted to the problem. It can do complex queries across massive disks pretty efficiently. But if you want to split data across hundreds of servers and keep it in memory, you have to give up much of the functionality Oracle provides, and it would still have complexity and features that would take up resources, driving up the cost and degrading performance.

A traditional SQL database is built on ACID principles: (Atomicity, Consistency, Isolation, Durability), that make databases reliable for banks and accounting systems. A transaction either works or it doesn’t. It can’t be half-completed, where money is debited from one account and not credited to another. Database activity can be viewed as a single series of transactions and queries – any query will see the database in a consistent state in the chain. Each transaction is isolated from every other, even if they happen concurrently, they don’t affect each other, as if they happened sequentially. And once a transaction is committed, it stays committed, regardless of power loss, network failure, etc.

ACID imposes a cost in complexity and performance. There’s a computer science result called Brewer’s CAP theorem (longer version, by Brewer). Pick any two: Consistency, knowing a single up-to-date state of the data; high Availability for writes; and Partition tolerance, no impact from network failure. The CAP theorem means that when you have a distributed system that communicates over a network, you have to make tradeoffs. If you have a master database and a few slaves mirroring the master for reporting, web queries, there will always be an instant where the master has been updated and the slaves are still out of date. There is no perfect disaster recovery (DR) system, where two systems are 100% in sync and if one fails, you can immediately fail over to the other with a guarantee of no data loss. There is always that instant where if the network fails, things are not 100% in sync. ACID and distributed systems don’t mix perfectly.

But Google doesn’t need ACID. Nobody cares much if two similar queries at the same time yield slightly different Google results, or Facebook walls. They do care if they don’t get it back in milliseconds.

And networks have grown incredibly fast, large and reliable. Meanwhile, CPUs are still getting cheaper, but individual CPU cores are not gaining performance as quickly, and speeding them up is less energy-efficient than buying more low-power CPUs. So companies like Google, Facebook, etc., use a different set of non-ACID tools in big server farms that let lots of cheap servers connected over fast networks work together.

Wags sometimes refer to the new databases, in contrast to ACID, as ‘BASE’: Basic Availability, Soft state, Eventually consistent. ‘NoSQL’ is more commonly used. Different NoSQL database systems are optimized for different use cases: key-value stores, document databases, graph databases. There has been an explosion of new database systems – the growing blue area in this chart.

What does the Big Data stack look like?

Clusters of virtual machines on Amazon Web Services, Heroku, Rackspace, etc.: PaaS (platform as a service)
NoSQL databases for non-ACID applications: MongoDB, Cassandra, CouchDB, Redis, Riak (a popularity chart, Google Trends).
Platforms to distribute algorithms over those clusters – Hadoop, MapReduce.
Analytical software to run complex machine learning algorithms on those platforms.

The last bullet point is what people most commonly associate with ‘Big Data’. Not only can Google return the most frequent completion as you type, it can update the most frequent searches in real time. More critically for the Google business model, it can predict which ad link you are most likely to click on, based on your recent activity, current location, everything it knows about your demographics and psychographics from your searches, your clicks on all the web sites in Google’s ad network, your emails in Gmail. Powerful machine learning algorithms like support vector machines (SVM) can run on clusters in a highly parallel manner to model behavior using hundreds of features and make predictions in real time.

If you have all the parameters to make a decision, given enough data and training, machine learning will generally find a good model and make useful predictions. Nevertheless, ad networks still present Christian singles ads to old married non-Christians. If you don’t collect the right parameters and feed them in the right form, you get garbage in, garbage out. If the data you have isn’t predictive, or you have the wrong model, having a lot of data isn’t going to help you ¹.

Bottom line: A new computing stack, pioneered 15 years ago by Google, has gone mainstream. It consists of ultra-cheap small computers connected by fast networks, plus distributed algorithms, for new applications that run at web scale and don’t have strong consistency requirements, and can make sophisticated predictions and decisions in real-time.

People call it ‘Big Data’, but what’s new and big is the computing power that can be directed at data in real-time. The distinguishing feature of this paradigm is dealing with more data than can fit in a computer’s memory by splitting up the task over many computers working in parallel in real-time, instead of doing them sequentially on a disk in batch mode.

In that sense, a traditional SQL data warehouse with petabytes of transaction data, batch processing business intelligence for online analytic processing and data mining, is not in the new Big Data paradigm, in the same sense as a real-time machine learning ad-serving engine that may have a smaller database.

It’s not the data that got big, it’s the computers that got small, fast, very numerous, and very very clever.

“One word: Cat pictures. Wait, that’s two words.”

¹ It’s undeniable that bad models were a signficant cause of the financial crisis. But it’s an error to say that the financial crisis was caused by big data gone awry. First of all, the bad models just so happened to be the ones that made the bankers the most money in the short run. It wasn’t so much bad big data as it was misaligned incentives, agent-principal problems, in some cases outright looting. Second, to my knowledge, there was exactly one firm that had a big data system that gave an integrated real-time picture of firm risks: Goldman Sachs, which averted life-threatening losses (although they were at risk if AIG or the whole system went down). Goldman’s system priced every instrument, however exotic, in real-time using the same curves, forward rates, volatilities, and correlation matrices, and allowed them to compute risk and test shocks against the whole firm. They were the only ones who said, we’re losing money every day on this when our model says we shouldn’t, let’s take a close look at this… and then cut losses and go short. Unlike Goldman Sachs, which grew organically, every other firm was cobbled together with mergers, and had different groups which did their own analytics in disparate systems which someone then struggled to integrate at the top level. They lost billions because, unlike Goldman, they didn’t use a big data paradigm. And also because “At Goldman, …when they say get out, they get out. At [competitor], when [the president] says get out, people start negotiating.”

What I Learned

StreetEYE — Sun, 30 Dec 2012 22:55:50 +0000

I didn’t really post as much as I would have liked this year. I envy people whose thoughts come out in a more or less coherent, finished form. When I post something, I always think of what I really wanted to say after hitting ‘publish’.

Today, I’m going to just try to write for an hour and post what comes out, hopefully resisting the temptation to ninja-edit.

My buddy Josh does a post with quotes where a bunch of people say what they learned over the last year. So what did I learn?

Startups are hard. Saw VC Ben Horowitz talk a couple of times. This was the gist of one of his talks, I recommend it for anyone in startup land. What stuck with me was what he said about meeting Michael Jackson and mentioning how outrageous one of his moves in The Wiz was. MJ replied, it was harder than it looked, he had to do that going uphill! The point is, whenever you see someone excelling on stage, at the World Series, in Olympic gymnastics, they make it look easier than it is. When you have to do something new and worthwhile, however hard it looked before you started, you discover new layers of difficulty you didn’t know existed. Doing what your teacher showed you how to do is a linear or maybe polynomial problem. Doing something new and figuring it out as you go along is highly nonlinear, or non-polynomial. Until you figure it out, and it’s old hat and you wonder why it seemed so hard.

At the start of the year I launched this little mad science experiment, Linkfest.com. The basic idea is that everyone goes to the ‘most read’ page at Bloomberg, Wall Street Journal, Financial Times etc. But today, where everyone shares everything socially, and everyone is a curator, it should be possible to create a ‘most read’ page across all financial media by finding what the best ‘power curators’ are sharing. And ideally, ultimately create an ecosystem like reddit, where the best rises to the top, and the spam gets trashed, the best content creators and the best curators all get the following they deserve.

I learned that community filtering and crowdsourcing work pretty darn well. Linkfest.com got a pretty regular following of a couple of hundred visitors, thanks particularly to Josh, Fred, Will. But it’s sitting pretty solidly in the ‘trough of sorrow’ phase. Not totally sure if I haven’t quite got the features right…or maybe the design needs to be sharper…or maybe I just need to make it more viral and market it better. I don’t know. It’s also possible it’s overfitted to what I want, and it’s not the 10x improvement users need to alter their habits, and the real power users build their own custom news feeds with Twitter, Google Reader, Flipboard.

So I also learned a lot about startups, news aggregation, etc., but not nearly enough. So as a next step I’ll do a survey to find out what people want from Linkfest.com, and what could be better. Watch this space. Or, just let me know what sucks and what could be better.

From mook to MOOC. I went through the Coursera Machine Learning course and learned more than I expected. The massively open online course (MOOC) education revolution is real, education is going to be disrupted as badly as music, news, finance, the post office, etc. (There’s a Shirky law that, if it is traditionally housed in a building with a Greco-Roman pediment, it’s going to get disrupted, because it’s a fat business model dependent on a trust heuristic that the Internet will tend to invalidate) If you haven’t already, check out some online courses and TED talks in 2013.

Online education startups: Coursera, Udacity
Universities and consortia: MIT Open Courseware, Harvard/EdX, Yale, Stanford
MOOC aggregators: Class Central, Open Culture

Next, will take the Stanford databases course, since everything I know about databases I learned myself, badly; and Damodaran’s valuation class.

Rise of the Machines. The other thing I learned is that, given decent data, with tools within the reach of a bright high schooler, machines can make decent decisions. We’ve reached the age of the smart machine. Any kid in high school today who grows up to be any kind of knowledge worker, is going to be primarily managing and assisting machines to do the work of diagnosing, solving problems, navigating, etc. Automation has hit the skilled manual worker pretty hard, but now it’s hitting the most skilled, and the consequences are going to be far-reaching. The future of jobs and wealth distribution is one aspect. We’re all going to have amazing robot assistants, autonomous drones, self-driving cars, robot cheetahs.

But we also have a surveillance state where everything you do is stored, and through some strange interpretation of the Fourth Amendment, there is no expectation of privacy or protection against the government capturing all manner of information (possibly including all domestic communications – they won’t tell us, and possibly interpret a search as taking place only when it’s investigated, not when it’s intercepted), storing it forever, as long as they get some kind of rubber stamp before using it against you.

And we’re literally one step away from machines that can repair, design and build themselves.

We learned easy money cures all, at least in financial markets, in the short run. It remains to be seen what it cures in the real economy. The problem is the trade and investment and financial imbalances, and financial incentives, that led up to 2008 haven’t really been rectified. The US is in better shape because it had a more aggressive fiscal policy (initially) and monetary policy, and the fact that everyone wants to own US bonds. Super hard landing in China and euro zone breakup seem off the table for now. So the most likely scenario is continuing gradual pickup in growth, until one day everyone realizes the boat is leaving the station to get cheap loans, houses, and stocks, and a real boom in growth and inflation picks up. The Fed wants a little inflation to make people less underwater on houses, and debt easier to pay back. But once the inflation genie is out of the bottle, it’s awfully hard to put back. At some point downward pressure on inflation from globalization becomes a spent force. Demographics reduce supply without a corresponding reduction in demand. Unclear whether energy and commodities will remain as benign as global demand increases and long-term sustainability of US oil boom is questionable. And once foreigners realize the Fed wants to inflate and make them lose money on their bonds, the dollar drops, and rates go up, and then the long-awaited global trade/investment/financial sector rebalance is on. Helicopter Ben is living up to his name, but some day this war’s gonna end.

If voting changed anything, they would make it illegal. In politics, we went through a dysfunctional election full of sound and fury that doesn’t seem to have signified or resolved anything. Despite failing to get a majority of votes for Congress, Republicans are still in the majority of districts due to gerrymandering and an urban-rural cultural divide. They are in safe seats and impervious to threats from the left, but highly vulnerable to primary challenges from the right, and don’t pay any cost for stonewalling and prevaricating. So that’s what we get.

We learned that despite living mostly in the immaculate cyber-world of our mobile devices, Mother Nature always wins. The South Ferry station, just built with massive 9/11 rescue funds (thanks America) is still closed and needs to be completely rebuilt after Sandy. And the A train causeway to Far Rockaway is also a total loss. After an unbroken string of major storms, one can only hope this will not be an annual occurrence.

What ties all these threads together? We really need to design structures that align the interests of the individual with the interests of the whole, and allow both to adapt to meet new challenges. The world is changing too fast to keep up. And we’re failing badly. It’s all about market design, and our systems suck.

Social capital – or, the lost art of not taking a dump in the community pool

StreetEYE — Mon, 05 Nov 2012 01:16:56 +0000

The first casualty when war comes is truth. – Hiram Johnson

Everybody talkin’ to their pockets
Everybody wants a box of chocolates
And a long-stemmed rose
- Leonard Cohen

Let’s talk a little about social capital.

According to studies, Greeks work the longest hours in Europe, and their retirement age is in the middle of the pack. Same goes for a lot of developing countries, and even some US inner cities. People work themselves to the bone, and they don’t get ahead.

Why are those countries in such a mess?

One reason is physical capital. Most of the difference between the productivity of an Indian coolie carrying bricks on his head and a US truck-driver is the truck.

Another reason is human capital. Poorly performing countries don’t have as many literate, well educated, highly trained workers who can handle complex, productive tasks with advanced tools and techniques. Not to pick on India, but the literacy rate is 74%.

The last reason is difficult to quantify and therefore not as much discussed in economics, but it’s absolutely critical: social capital.

Social capital is the set of social values, norms, and behaviors that improve the productivity of economic and social interactions. In the poor countries and communities, examples of dysfunctional norms and values abound. Girls shouldn’t be educated, or school isn’t cool. Certain jobs and places for university students should be reserved for certain races/castes (goes both ways). Cheating and bribery are endemic. Little things like people don’t show up on time, don’t wait on line, don’t observe traffic rules, literally dump waste in the drinking water, get away with murder. People don’t have incentives to be productive. And antisocial behaviors are not punished, but held in high regard, and reap rewards.

In advanced countries, social networks and hierarchies allow people to cooperate and pursue shared goals. Good behavior is rewarded and bad behavior punished, not just in courts where rule of law applies to all, but in the court of public opinion. Trust evolves – you don’t have to count your change, you can give your credit card to the waiter and not worry about them stealing. You don’t need to double-lock your door and put bars on your window and worry about rushing home before dark. Status accrues to people based on their abilities, achievements and character, not based on connections.

There were a lot of reasons for the financial crisis. But one was “I’ll be gone, you’ll be gone” when the losses turn up. In many cases the people in a position to know knew loans were bad, deals were bad, ratings were bad. But the incentive was to get the deal done, book the profit, cash the bonus. Clients turned into counterparties, it was OK to blow them up. Market makers were replaced by HFT programs with no responsibility for orderly markets. A looter mentality replaced building for the long term.

Division destroys social capital. You can have social capital with a left and a right. But you can’t have it when one section of society demonizes the other, assumes that the other side only cares about itself and not the country or broader good. You can’t have it without a basic consensus on the character of society, without the ability to develop institutions because the rules are changing. And bad behavior drives out good, when you assume one side is going to gain an advantage from lying, cheating, or stealing, it becomes an excuse to do it yourself.

We’re having an election Tuesday. Each side accuses the other side of lying – in all too many cases justifiably (1 – 2). If it’s close, one side will accuse the other of stealing it. If a poll or even an economic number (!) is seen to favor one side, it’s proof that the other side is cheating. There’s a war on facts, with economic data collection and reports de-funded if it’s believed they provide evidence for the other side.

We’re not going to turn into Greece because of the debt. We just won’t grow as fast if we spend on the wrong things. We will turn into Greece if we become a society of looters and moochers. And once we see each other that way, that’s how we behave.

You can’t have progress or a civilized society with people literally, or figuratively, taking a dump in the community pool. As a society, we need to call people out and exact a price when they do it.

Broken Windows

StreetEYE — Fri, 02 Nov 2012 02:05:50 +0000

So, some people are talking about Hurricane Sandy putting people back to work, and others are pointing out that this is the ‘broken windows fallacy.’ True, a massive superstorm is usually not a good thing. Nevertheless, three quick points.

English: Aerial view of roadbed collapse near the interface of the cantilevered truss sections of the San Francisco-Oakland Bay Bridge. View northwestward. Cropped from original version to better fit San Francisco–Oakland Bay Bridge article. (Photo credit: Wikipedia)

1) The recovery spending is not a deadweight loss. You jump into the pool and forget to take your telephone out of your pocket. Rats! this is not a good thing, it’s going to cost to replace it. But then you say to yourself, well, I was due for a new one anyway. The loss is the remaining useful value, not the amount you have to spend on the new phone. An earthquake damages the Bay Bridge, and $6B is spent on a new bridge. But the existing bridge was obsolete and prone to collapse, and the new one is better. The loss is not the $6B cost of the new bridge, just the remaining value of the old one, which may not have been worth much at all. The impact may just be to bring forward necessary spending a little earlier.

2) The recovery spending is not all lost opportunity cost. Suppose you’re a farmer harvesting the crops, and your truck breaks down, and you have to fix it, while the crops rot in the field. That’s a bad break. Now, suppose you’re unemployed and watching TV, feeling sorry for yourself, and your car breaks down. You spend some time fixing it, feel a lot better for yourself, and decide to get a job as a mechanic. That’s not such a bad break. When there’s high unemployment and low interest rates, the situation is more like the latter. Borrowing is not going to raise interest rates and crowd out other useful investment, and hiring recovery workers is not going to have to bid up wages to get them from productive work in other industries. To some extent, we’re just taking idle resources and putting them to work. And an unemployed worker is a wasting asset, you’re never getting back his or her productive years, and the longer they stay unemployed, the more rusty their skills get, the more potential employers question their value, and the less likely they are to return to a similar job. In the sense of forcing people to spend cash on something productive, Sandy is doing what Ben Bernanke is trying to do.

3) The world is a complicated dynamic system. You never know the outcome of the alternative history that could have been. Only simple models give clear answers, and the world does not oblige by acting like one. The hurricane could be pulling a finger out of a dike, Wall Street will feel obliged to move some people away, the tax base will diminish, and New York will go back to the 70s or turn into Detroit. (Unlikely, we survived 9/11, but eventually all shining cities enter a cycle of decay.) Alternatively, the infrastructure could get fixed up, and it could result in surviving an even more massive disaster unscathed that otherwise would have destroyed us.

A natural disaster is not something to be wished for, but we’ve been through worse, it could have come at a worse time, and what doesn’t kill us makes us stronger.

The Paul Ryan plan

StreetEYE — Sun, 12 Aug 2012 15:25:56 +0000

The Paul Ryan plan ‘Promotes saving by eliminating taxes on interest, capital gains, and dividends; also eliminates the death tax.’

So people like Mitt Romney, and all his heirs in perpetuity, would never pay another dime in income tax. How sweet is that? How fair is that?

It’s shocking that in this day and age, someone could make such a proposal, and be considered a serious person and politician.

The GOP has come an awful long way from its founder, who said, “Capital is only the fruit of labor, and could never have existed if labor had not first existed. Labor is the superior of capital, and deserves much the higher consideration.” – Abraham Lincoln