A few months ago, I got quite interested when Trading Blox announced that they introduced a new walk-forward functionality in their latest version. I just got round to upgrading, and giving that walk-forward testing a go. Amongst other things, some of the chart features have been improved – as can be seen in the eye candy above.

How it works

You can check this previous post for an explanation of how walk-forward works, from a general point of view. Robert Pardo (usually credited with the invention of walk-forward) has also written his bookfew books on the subject.

This new feature is a combination of small enhancements to the core application (ie. silent tests, dynamic date and starting equity settings, etc.) and a semi-custom script implementing the walk-forward testing procedure. It feels more like a (good) hack leveraging the core stepping functionality, rather than a functionality built from the ground up, but it does the job rather well (I looked at the underlying code and its ingenuity is actually pretty cool).

A minor issue is that you have to work out the number of optimization/out-of-sample cycles. If you get that wrong, the test will not cover the desired time interval and it makes the process less automated. Apart from that, the settings are pretty straight-forward, you can choose lengths of optimization and out-of-sample phases as well as other options:

In the settings above, there are 8 cycles, each having a 5-year optimization phase and 1-year out-of-sample phase

Objective Function

At the end of the optimization phase of each cycle, the system picks the best set of parameters to be used in the out-of-sample phase. However there are many ways to determine the best system. Do you use raw CAGR, MAR, Sharpe ratio, etc.?

There is a need to define the objective function (also called bliss function, as explained here, which will be used to determine the best system performance.

Trading Blox calculates standard statistics (MAR, CAGR, Drawdown, etc.) which can all be used as the objective function, but also allows the implementation of custom statistics calculations. These custom statistics can then be used as the objective function.

Output

This is where you realise the functionality is not fully integrated into the app. The system shows the individual results of each out-of-sample runs:

The stats calculated relate to each out-of-sample run (1 year in my example), whereas ideally these should all be spliced together, to form one system with its own stats. Same applies to equity curves, which are charted year by year (as individual charts). It is still possible to access the Daily Equity log file to do this yourself but it would be nice to have it better automated. The fact that it is a script and not compiled core code might make it possible to customize it to implement this automation.

Another log file also allows you to investigate the results of each optimization run and check their performance:

Starting optimization run ,225,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
Run ,225, of ,256, Stepped Parameters: ,
Name,Step Value,Goodness Measure of ,64.497285767,
Run ( Index ),8.000000000,
Optimization Run,1.000000000,
Entry Breakout (days),20.000000000,
Exit Breakout (days),10.000000000,
Starting optimization run ,226,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
 
----------------------------------------,
Best goodness of ,64.497285767, was on run ,1.000000000,
----------------------------------------,
 
Starting out of sample test ,241,2010-01-01, to ,2010-09-07, with starting equity ,166470686.069881530,

Test Results

For my first “test ride” of the walk-forward functionality, I used the good ol’ Donchian system.

The system parameters being tested are the Entry and Exit breakout lengths ranging from 20 to 50 days for the entry and 10 to 25 days for the exit.

The parameters for the walk-forward procedure are a 5-year optimization phase and 1-year out-of-sample phase with the plain CAGR stat being used as the objective/bliss/goodness function. The dates of the test go from 1998 to 2010 (but bear in mind that the first out-of-sample result is from 2003, ie. 1998 to 2002 are used as first optimization period).

The results (year by year) are the ones shown in the section above.

For an interesting comparison, I ran a standard stepped test using the same Donchian system and parameter values. Check below how the walk-forward system performance stacks up against the range of outputs from the stepped test:

All test results with the walk-forward result in yellow

In hindsight, it would have been better to pick other systems that performed better, but the advantage of the walk-forward process is that it adapts itself to pick a system.

In Closing

The walk-forward functionality is definitely not a fully-fledged feature of Trading Blox (yet?) – however it does allow for some automation of the process. The fact that it is “open” is probably a good thing as it can be tinkered with and improved easily. A nice addition to the product…

Related posts:

1. How can Walk-Forward testing keep your system a step ahead?
2. MMDI Portfolio Filter in Trading Blox
3. TradersStudio: the best Systems testing software?

Detailed Results

Below is the chart of each strategy over the month of August 2010, with the composite average of all systems:

And the results in tabular format:

 
Apart from 2 Bollinger Breakout, most systems are showing a good performance for last month.

2010, through the Composite Index

The composite index just about breaks even for the year with this strong August performance:

Appendix: System Details

System Rules and Parameters

All the systems were tested with the same simple position sizing rules of 1% per new trade. No other Money/Risk Management rules were used. No slippage was considered and a $15 RT commission applied. No return on margin is added to the system performance

The system rules are detailed on the Trading Blox online documentation.
The MA Crossover system was used with moving average pairs of 10-20, 20-50 and 50-200 days. The stops/position sizes are set at 2x, 3x and 5x ATR respectively.
The Bollinger Band system is the classic use of the Bollinger Bands with entries taking place at Breakouts. The parameters used were 20, 50 and 200 days with 2 standard deviations.
The Triple moving Average system was used with moving average triplets of 10-20-50, 20-50-200 and 50-200-800 days. The stops/position sizes are set at 2x, 3x and 5x ATR respectively.
The Donchian System is a simple version (with no Trade Direction filter) with channel lengths of 20, 50 and 200 days for entries (and 10, 25, 100 for exit). The stops/position sizes are set at 2x, 3x and 5x ATR respectively.

Portfolio Instruments

Covering over 50 instruments across Equities, Interest Rates, Currencies, Agriculturals, Metals and Energies, from around the world, the portfolio contains the following futures (CSI Symbols): AD, BP, CC, C, CD, CFC, CL2, CT, CU, YM, EBL, EBM, EBS, ED, EOX, ESM, FC, FEI, FFI, GC, HG, ICL, IND, IRB, JK2, JP2, JP6, JR2, JRB, JSK, JTI, JY, KC, KPO, KTB, KWR, LC, LGO, LH, MFX, MP, NG2, RA, RS, SB, S, SF, SI, STW, SXE, TRY, US, W, YTC.
Click here for a tabular view with description and exchange information.

Result Normalization

The system performances are adjusted for volatility to normalize the results. See why and how here.

Related posts:

1. the State of Trend Following in July
2. the State of Trend Following in June
3. The State of Trend Following report – Draft V0.2

Trading hand signals - tradingpithistory.com

It’s nearly the end of the Summer and I hope you enjoyed a nice time. The blog will go on a short break until the end of August, while I (hopefully) enjoy some good weather in the French Cevennes and the English Lake District.

In the mean time, I’ll leave you with a link to a pretty cool website I found (Trading Pit history – sample pics above), which documents the various hand signals used on the futures trading floors. A typical part of the trading folklore, and a language of its own, probably in decline due to the advent of electronic trading.

New readers: Welcome to the blog! You might be interested in catching up on the posts from the main 2 topics of the Summer: roll yield and back-testing results statistical significance – or older posts (the “Popular Posts” section on the left-hand side is a good place to start).

You might also want to subscribe to the blog to make sure you are not missing on any of the action…

Roll yield posts:

• Crude Oil, Contango and Roll Yield for Commodity Trading
• Trend Following returns breakdown
• Roll Yield and Commodity Yield Curve
• Better Trend Following via improved Roll Yield
• VIX, Peso… Sometimes you just cannot trade it!

Back-testing results statistical significance posts:

• Evidence-Based Technical Analysis
• The Bootstrap Test: How significant are your back-testing results?
• Bootstrap – Take 2: Data Mining bias, Code and using geometric mean
• Monte Carlo Permutation: Test your Back-Tests

Tune back in at the beginning of September…

Related posts:

1. Blog Therapy: Why I write this blog?
2. The Bootstrap Test: How significant are your back-testing results?

The concept behind the Monte Carlo Permutation is similar to the Bootstrap method:

• Generate multiple random outputs based on the single sample data from the back-test.
• compare the random Monte Carlo outputs to the back-test output to evaluate its statistical significance.

The difference lies in how the multiple random outputs are generated. Whereas the bootstrap generates a sampling distribution for the back-tested rule return, the Monte Carlo Permutation focuses on the pairing of the rule positions with the instrument daily return. Its resampling randomly associates the rule positions with the market returns, without replacement.

The H0 hypothesis in the Monte Carlo Permutation test asserts that the returns of the rule evaluated are a sample from a non-profitable population, or, in other words, that rule positions are randomly correlated to market returns.

Monte Carlo Illustration

Imagine the following back-test result, presented day by day:

There are effectively two input time series:

• Rule Positions
• Market Returns

The way these two time series are linked (by date) produces the daily output for the rule return – and a mean return can be calculated.

The permutation of the Monte Carlo method will reshuffle one time series to produce random links, or pairing, and produce a different rule output.

Two examples can be found below. The market return time series has been randomly reshuffled to produce two different sample outputs:

The Monte Carlo Permutation produces a large number of these random outputs. The p-value of the original back-testing sample can then be computed (it is equal to the fraction of random rule returns equal or greater to the back-tested rule return).

Note that Aronson once again recommends to run the back-test evaluated by the Monte Carlo Permutation on detrended data. It is also mentioned that Timothy Masters (who got the idea of applying the Monte Carlo method to rule testing) has performed tests showing that the bootstrap and Monte Carlo Permutation methods produce similar results when using detrended data.

Step by Step with Data Mining Bias Handling

Of course when applying this method to more than one rule, data mining bias comes into play.

The methodology for the Monte Carlo Permutation for data mining back-testing can be broken down as follows:

1. N back-tests are run on detrended data. Both rule position and market return time series are collected for the back-tested rules.
2. The market return time series is randomly reshuffled and paired with each of the N rule position time series to produce a new daily rule output time series for each rule. The same pairings must be used for all rules to ensure that the potential correlation structure present in the rules is preserved.
3. A mean daily return is calculated for each of the N rules – the best return is selected as the value for the sampling distribution in this iteration
4. Repeat steps 2 and 3 a large number of times
5. Form the sampling distribution of the best means generated in the steps above.
6. Derive the p-value of the best back-test mean return based on the sampling distribution.

Some “Criticisms”

Aronson mentions that since the Monte Carlo Permutation does not test a hypothesis about the rule’s mean return (H0 is about random correlation of positions and market returns) it is not possible to use it to derive confidence intervals – as could be done with the bootstrap sampling distribution.

The method also requires access to more information than the bootstrap (which only needs the daily rule returns). It makes it impossible to apply to “black box” systems or programs. For example, the Monte Carlo Permutation method would not enable us to check the statistical significance of a Trend following Wizard as was done in bootstrap post #2.

The same remark concerning the use of arithmetic mean return instead of geometric mean return applies here also, but that can be easily modified.

Finally, the method, as formulated, only considers extremely simple cases of money management with identical size for all positions. The method would need to be adapted to be used for rules with more complex money management strategies.

I’ll let you come to your conclusions and experimentations but it does seem like the Monte Carlo Permutation method has more weak points than the Bootstrap test.
 
 

Related posts:

1. The Bootstrap Test: How significant are your back-testing results?
2. Evidence-Based Technical Analysis
3. Au.Tra.Sy blog Summer Break

Please find below all individual results for July 2010:

Notes
<sup>* YTD: Year-To-Date performance.
** AUM: Assets Under Management.
1. Abraham Trading was founded by Salem Abraham, after he was introduced to Managed Futures and Trend Following by Jerry Parker. He is considered as a “second-generation” Turtle.
2. Altis Partners started trading in 2001 and now manage over a $1B with their Altis Global Futures Portfolio. The figures referenced in the performance table are not provided by Altis Partners and no reliance should be taken as to their accuracy, and as a consequence the figures may not be in accordance with any CFTC / NFA performance reporting requirements.
3. The four founders of Aspect (Eugene Lambert, Anthony Todd, Michael Adam and Martin Lueck) were significant members of one of the most succesful funds in managed futures – AHL (Adam, Harding and Lueck).
4. BlueTrend, from BlueCrest Capital, is one of the largest Trend Following funds – headed by Ms. Leda Braga
5. Campbell & Company is one of the oldest Trend Following firms, operating for around 4 decades.
6. Chesapeake Capital was founded by Jerry Parker, a former Turtle.
7. Clarke Capital was founded by Michael Clarke in 1993.
8. Drury Capital, Inc., was founded in Illinois in 1992 by Mr. Bernard Drury.
9. Dunn Capital was founded by Bill Dunn.
10. Eckhardt Trading is the firm managed by William Eckhardt, who co-led the Turtle experiment with Richard Dennis
11. EMC Capital was founded by Liz Cheval, a former Turtle.
12. Hawksbill Capital was founded by Tom Shanks, a former Turtle.
13. Hyman Beck & Co. main principals are Alexander Hyman and Carl Beck.
14. JWH & Co. was founded by John W. Henry, Owner of the Boston Red Sox.
15. Originally ED & F Man. Became a succesful CTA under Larry Hite and went on to form part of The Man Group plc, which subsequently bought AHL to form the Man AHL: the systematic trading division of the Man group.
16. Millburn Ridgefield have been trading Trend Following models since the early 1970’s. As they report performance figures one month later, last month performance is not reported in this report and their YTD, AUM stats are from the month before.
17. Rabar Market Research is the company of Paul Rabar, a former Turtle.
18. Saxon Investment was founded by Howard Seidler, a former Turtle.
19. Superfund founder and CEO: Christian Baha.
20. Transtrend is a Trend follower CTA based in Netherlands
21. Winton Capital is a London-based CTA founded by Dave Harding (also co-founder of AHL).
 
Note that the figures referenced in the performance table are not provided directly by any of the funds/CTAs featured in this report, but are sourced from other publications such as hedge fund/CTA websites.</sup>

These are top of the range CTAs/Managed Futures funds in the Trend Following space.
Most of the traders behind these funds have been involved in the Turtle Trading experiment (2 excellent books on this topic: Complete Turtle Trader – featuring the actual turtle rules and The Way of the Turtle), featured in the legendary books by Jack Schwager: Market Wizards and New Market Wizards, or in Michael Covel’s dedicated Trend Following book.
 
 
 
 

Related posts:

1. Trend Following Wizards – June 2010
2. Trend Following Wizards: negative month (May), 2010 now in the red…
3. Trend Following Wizards – April 2010

In part 1 of this bootstrap post, we looked at how to apply the method to establish the statistical significance of a single trading rule. In Part 2, we’ll look at how to deal with the data mining bias, the impact of geometric vs. arithmetic mean return. The code implementing the bootstrap test is available for download at the bottom of this post.

Dealing with Data Mining Bias

The approach described in the single rule test is not valid when performing data mining (whether testing different rules or different parameter values of the same rule). As per the data mining bias (explained previously), the (best) rule selected from the data mining process will invariably owe a large part of its over-performance to random (good) luck.

The way the bootstrap test deals with the data mining bias is by implementing a concept introduced in White’s Reality Check. The Reality Check derives the sampling distribution appropriate to test the statistical significance of the best rule found by data mining.

In effect, the concept is fairly simple – and similar to the single-rule bootstrap: assuming N rules have been tested in the data mining process, each resample iteration will perform a resample with replacement for each rule and the best mean return will be kept as this resample iteration’s test statistic:

1. N back-tests are run on detrended data. The mean daily return, based on x observations, is calculated for each back-tested rule.
2. Each rule’s mean daily return is substracted from the rule’s set of daily returns (zero-centering), This gives a set of adjusted returns for each rule.
3. For each “higher-level” resample (to form the sampling distribution of the best-performing rule in a universe of N rules), perform a “lower-level” resample with replacement on every rule. For each rule select x instances of adjusted returns at random and calculate their mean daily return (rule bootstrapped mean). Compare each rule bootstrapped mean and select the highest one: this is the test statistic of this “higher level” resample (bootstrapped best mean).
4. Perform a large number of “higher level” resamples to generate a large number of bootstrapped best means.
5. Form the sampling distribution of the best means generated in the step above.
6. Derive the p-value of the best back-test mean return (non zero-centered) based on the sampling distribution derived above

In effect: for each iteration, resample each rule, take the best return, keep it as this iteration test statistic and move on to the next iteration. The sampling distribution is formed of each iteration’s best return.

White Reality Check Related Papers

I have not yet found searched hard for White’s paper on the Reality Check but I did find the two following papers which seem to be worth a read:
Stepwise Multiple Testing as Formalized Data Snooping – Romano & Wolf

Abstract:

It is common in econometric applications that several hypothesis tests are carried out at the same time. The problem then becomes how to decide which hypotheses to reject, accounting for the multitude of tests. In this paper, we suggest a stepwise multiple testing procedure which asymptotically controls the familywise error rate at a desired level. Compared to related single-step methods, our procedure is more powerful in the sense that it often will reject more false hypotheses.

Unlike some stepwise methods, our method implicitly captures the joint dependence structure of the test statistics, which results in increased ability to detect alternative hypotheses. We prove our method asymptotically controls the familywise error rate under minimal assumptions. Some simulation studies show the improvements of our methods over previous proposals. We also provide an application to a set of real data.

Re-Examining the Profitability of Technical Analysis with White’s Reality Check and Hansen’s SPA Test
Abstract:

In this paper, we re-examine the profitability of technical analysis using White’s Reality Check and Hansen’s SPA test that correct the data snooping bias. Comparing to previous studies, we study a more complete universe of trading techniques, including not only simple rules but also investor’s strategies, and we test the profitability of these rules and strategies with four main indices. It is found that significantly profitable simple rules and investor’s strategies do exist in the data from relatively young markets (NASDAQ Composite and Russell 2000) but not in the data from relatively mature markets (DJIA and S&P 500). Moreover, after taking transaction costs into account, we find that the best rules for NASDAQ Composite and Russell 2000 outperform the buy-and-hold strategy in most in- and out-of-sample periods. Our results thus suggest that the degree of market efficiency may be related to market maturity. It is also found that investor’s strategies are able to improve on the profits of simple rules and may even generate significant profits from unprofitable simple rules.

Geometric or Arithmetic Mean?

In part 1, I introduced the idea that the mean arithmetic return being positive is not equivalent to the strategy being profitable (ie. this is not a sufficient condition). On the other hand, the mean geometric return being positive is a necessary and sufficient condition to the strategy being profitable (ie. both conditions are equivalent).

Therefore bootstrapping using the mean geometric return as the test statistic should provide a better evaluation of the system’s profitability statistical siginificance.

I will not go into detail of how the calculation is done as it is very similar to the arithmetic mean return, but using log of returns instead. Note that the geometric mean will be a stricter test than the arithmetic mean (a rules can have a significantly positive arithmetic return but a negative geometric return).

To illustrate the multiple applications of the bootstrapping methodology, I decided to run the test on one of the Trend Following Wizards track record (set of monthly returns). I picked Chesapeake and ran the monthly returns (from 1988 to 2009) through the bootstrap test.

The p-value calculated using the arithmetic mean is 0.000098 (less than 1 chance in 10,000 that this kind of results are due to random luck). Using the geometric mean, the p-value is 0.00022. The values are extremely low, which is not surprising given Jerry Parker’s 20-year track record with only one losing year and a monthly average return of 1.7%.

Many people would point out that survivorship bias should be considered, and obviously it depends on how you look at it. The main point of this dual test is that the geometric p-value is higher than the arithmetic p-value, verifying that it is a stricter test of statistical significance.

Bootstrap Code

Finally, here is a tool coded to implement the bootstrap test for a single strategy – available for download. Note that this is distributed “as is”, with no guarantee (but that’s the one I have been using so I still think it does the job…). It should run on any Windows machine with the .Net framework installed (XP or higher should do fine).

It simply takes three parameters (separated by space):

1. Returns file path and name
2. Number of resamples
3. Flag for Arithmetic (A) or Geometric (G) mean calculation

It also generates a file in the same directory with all of the resamples test-statistic values (to draw the histogram).

Simply place the bootstrap.exe in your directory of choice and run it from the command prompt as below:

Run the bootstrap.exe from the command line

Download here:
bootstrap.exe

Related posts:

1. The Bootstrap Test: How significant are your back-testing results?
2. Monte Carlo Permutation: Test your Back-Tests
3. Evidence-Based Technical Analysis

As mentioned in the Evidence-based Technical Analysis review post, the main value of the book lies in the presentation of the two methods allowing for computing the statistical significance of trading strategy results, despite having a single sample of data:

Both methods solve the problem of estimating the degree of random variation in a test statistic when there is only a single sample of data and, therefore, only a single value of the test statistic.

Today, let’s look at the bootstrap test, with a practical application of it.

In very brief terms, the concept uses hypothesis testing to verify whether the test statistic (such as mean return of the back-testing sample) is statistically significant. This is done by establishing the p-value of the test statistic based on the sampling distribution. (Aronson covers the basics of statistical analysis earlier in the book. I have also mentioned previously The Cartoon Guide to Statistics, which covers these concepts too)

The problem with back-testing is that the results generated represent a single sample, which does not provide any information on the sample statistic’s variability and its sampling distribution. This is where bootstrapping comes in: by systematically and randomly resampling the single available sample many times, it is possible to approximate the shape of the sampling distribution (and therefore calculate the p-value of the test statistic).

Bootstrap on Single Rule Back-Test

In the context of hypothesis testing, the bootstrap tests for the null hypothesis that the rule does not have any predictive power. In practical terms, this is translated to the population distribution of rule returns having an expected value of zero or less.

The bootstrap uses the daily returns of a back-test (run on detrended data) and performs a resampling with replacement.

In practice:

1. A back-test is run on detrended data and the mean daily return, based on n observations, is calculated.
2. The mean daily return is substracted from each day’s return (zero-centering), This gives a set of adjusted returns.
3. For each resample, select n instances of adjusted returns, at random (with replacement), and calculate their mean daily return (bootstrapped mean).
4. Perform a large number of resamples to generate a large number of bootstrapped means.
5. Form the sampling distribution of the means generated in the step above.
6. Derive the p-value of the initial back-test mean return (non zero-centered) based on the sampling distribution

A Practical Application

To illustrate the concept, we can look at a back-test and apply the bootstrap method to its daily return series. I decided to look at a back-test I presented in Better Trend Following via improved Roll Yield. Remember: a standard 50/20 Moving Average cross-over system applied to Crude Oil was improved by adding a roll yield optimisation process.

In that instance, the benchmark is the standard strategy and we want to check that the strategy improvement was not the result of random chance. In Aronson’s book, benchmarking is achieved by detrending the data. However, this case is different as the benchmark is the standard strategy. The improved strategy results can be thought of 2 distinct parts:

• Results from standard Trend Following strategy
• Results from Roll Yield Optimisation

I therefore generated a composite, “Roll Yield-only” equity curve (by removing from the improved strategy equity curve the returns that could be attributed to the Trend Following component). I then computed the daily returns based on that equity curve.

1. This set of daily returns is the original sample of 5120 observations, with an arithmetic mean of 0.216%.
2. Substracting 0.216% to all 5120 returns adjusts those returns (zero-centering), ready to be picked for resampling.
3. The 10,000 resamples all pick at random, with replacement, 5120 observations from the zero-centered, adjusted returns. A mean is computed for each resample.
4. Each of the resample means are used to form the sampling distribution of the mean return:
    
   

5. The last step is the comparison of the non-adjusted original sample mean (0.216%) to the sampling distribution to establish the p-value, which is 0.006 in this example.

Once the p-value is obtained, it is simply a matter of deciding which threshold qualifies for statistical significance. Scientists usually determine the statistical significance threshold at 0.05 (ie. the null hypothesis would be rejected for any p-value less or equal to 0.05).

Note on Arithmetic Mean vs. Geometric Mean

As discussed above, the assumption that the rule does not have predictive power is translated to the arithmetic mean of its returns being equal to zero. In the bootstrap method, rejecting the null hypothesis occurs when the mean arithmetic return is statistically significantly positive.

I am usually no big fan of arithmetic mean of returns as it is a flawed indicator of profitability. In effect, a system can have a positive mean arithmetic return and still be unprofitable – think about a return of 50% followed by a return of -40%: arithmetic mean return is +10%, yet the overall return is minus 10%

Proving that the mean arithmetic return is significantly positive, and deducing that the trading system is therefore profitable is flawed. It is ironically amusing that Aronson spends quite a lot of time talking about logic reasoning and usual traps people fall into, to actually present a flawed deduction logic. To use an example from the book:

A dog having four legs (a profitable rule having a positive mean arithmetic return) does not imply that any four-legged animal is a dog (ie. any rule with a positive mean arithmetic return is not necessarily profitable).

On the other hand, any profitable rule has a positive mean geometric return, and any rule with positive mean geometric return is profitable. On that basis, using the mean geometric return as the test-statistic in the bootstrap must be more appropriate.

I’ll be running this post in 2 parts, and this concludes part 1…
In part 2, we’ll look at how the bootstrap method can be modified to handle the data mining process and its associated bias. I’ll also make the code used for the practical application above available for download (this will be a simple bootstrap resample tool developed on the .net platform for Windows). Finally we’ll explore the idea of using the geometric mean return as the test-statistic instead of its arithmetic cousin.

Related posts:

1. Bootstrap – Take 2: Data Mining bias, Code and using geometric mean
2. Monte Carlo Permutation: Test your Back-Tests
3. Evidence-Based Technical Analysis

I have been going on about roll yield and term structure for a few posts, and through two very concrete examples we’ll see how it can affect your trading and system development

A reader recently mentioned a paper (pdf by Sloyer and Tolkin) presenting a theoretical trading strategy which improves the risk-return profile of standard equity-bond portfolio by adding allocation to equity volatility represented by the VIX index. The idea sounds good on paper (no pun intended), but a “small” assumption might render the strategy impossible to implement practically:

VIX futures can realistically be included as an asset in a passively managed portfolio as the futures can be rolled relatively cheaply from one contract to the next as each contract expires.

The Current VIX Situation

Taking a look at the current VIX futures curve clearly invalidates the assumption above:

VIX futures curve - click to zoom in

At the current levels, the contango rate is over 100% annualized – definitely no “relatively cheap” roll yield. As we’ve seen with the contango exhibited in Crude Oil in 2009, futures performance failed to match the spot price – and with such a high contango rate in the VIX futures, the same would happen: spot price returns would be “eaten away” by the negative roll yield. Indeed, prices would have to raise by an annualized 100% just to counter the contango.

Sometimes a good theoretical idea fails at the practical implementation stage…

Mexican Peso: Same Concept, Opposite Effect

Coincidently, another reader was offering me a friendly warning regarding the use of spot market to drive signals for a futures trading strategy – as was described in Better Trend Following through improved Roll Yield (note: for practical reasons, the test in that post was done using front-month contract as a proxy for the spot market).

In effect, as is the case in the VIX futures, the roll yield part of the total return sometimes trumps the spot price moves. In these cases, looking solely at the spot market can be flawed.

This is very well illustrated by this Peso chart sent from our reader:

The black curve almost continuously going down is the spot price, whereas the blue curve is a continuous futures contract (back-adjusted using front-month contracts) - click to zoom in

The strong divergence between both series meant that an investor/trader going short on the Peso, would have been “right” (ie. the Peso unarguably went down), yet would have lost money if using futures to implement her trade (as highlighted by the futures continuous contract going up).

This is another case where the roll yield has a stronger impact than the spot price move.

Two Conclusions

Despite the spot price usually grabbing most of the attention, the roll yield can be the driving factor to a futures market’s total return. This can seem counter-intuitive but on long-term timeframes, roll yield explains most of the market’s performance (as discussed in this separating the wheat from the chaff paper I have linked to previously).

When looking at potential trades (in futures, or ETFs, which are sometimes no more than futures “wrappers” or even forex with cost of carry), one should weigh the spot price return potential against the term structure implied return. From a system development point of view, an idea might be to add a portfolio ranking/filter based on the implied yield vs. ATR (or other measure of “volatility”).

The second conclusion is that the market seems to reward those who trade against conventional wisdom. In that period covered by the Peso chart, most people were worried about further devaluations and were paying dear to hedge against this outcome. The hedge ended up costing them more than the cost of holding on to a devalued Peso. Alpha is finite and flowed to the minority being long the falling Peso…

Maybe time to short the VIX futures?

Related posts:

1. Trend Following returns breakdown
2. Crude Oil, Contango and Roll Yield for Commodity Trading
3. Roll Yield and Commodity Yield Curve

 
Today I’ll be talking about an excellent book, which was recommended on several “quant” blogs I read: Evidence-Based Technical Analysis by David Aronson. One of the main reasons I picked this book is because it teaches you to fish (instead of giving you a fish). So, if you’re after a book with great trading strategies or indicators, this might not be the ideal one, however if you want to learn about strategy testing and methodology, it’s probably a great addition to your trading library. It had been on my list for a while and I wish I’d read it earlier as it has the potential to add cornerstone methods to trading research and testing procedures. Read on for a summary with a review right at the end…

Intro

One of the early quotes from the book defines the concept it covers:

The scientific method is the only rational way to extract useful knowledge from market data and the only rational approach for determining which TA methods have predictive power. I call this evidence-based technical analysis (EBTA).

Aronson introduces early on the concept of objective (TA) vs. subjective (TA). An objective claim is a meaningful proposition, which can be unambiguously verified. For us mechanical system trading developers: a set of rules that can be back-tested. On the other hand, subjective technical analysis would consist of approaches such as Elliot Wave Analysis.

However, objective technical analysis is not sufficient on its own: you still need rigourous statistical inference to draw conclusions on its predictive power.

Part One: the Foundations

Part one of the book establishes the methodological, philosophical, psychological and statistical foundations of EBTA.

The first topic covered is the need for benchmarking to evaluate objective rules and introduces the concept of detrending, which I have previously discussed.

The second topic deals with cognitive psychology and gives examples of different types of behavioral biases that can fool us and make us believe in subjective technical analysis:

• Pattern recognition
• Confirmation bias
• Hindsight bias
• Over-confidence
• Illusory correlations
• Mis-perception of randomness

The antidote for these “mind traps” is the scientific method. The generic scientific method is covered in the third chapter with some history and philosophy of science and logic reasoning. The scientific method – which can and should be applied to Technical Analysis – contains 5 stages:

1. Observation
2. Hypothesis
3. Prediction
4. Verification
5. Conclusion

Subjective TA does not conform to the scientific method and the author presents an interesting study of objectification of a subjective TA pattern (Head and Shoulders) to make it testable (it shows that Head and Shoulders is worthless on stocks and has doubtful value on currencies).

Statistical Analysis of Back-Test Results

The next three chapters introduce and cover statistical analysis. The beginning of this part gives a good refresher on statistical inference, starting with concepts such as frequency distribution, standard deviation, probabilities and p-values. The example of sampling and statistical inference using beads in a box makes for a good illustration and a fairly clear parallel with the world of trading rules back-testing.

The book moves on to concepts such as hypothesis testing, statistical significance and confidence interval, etc. and how they relate to rule testing.

One of the main issue of back-testing results is that they only represent one sample of how the systems/rule(s) perform. Aronson presents the classical statistical approach to derive the sampling distribution (required to perform the statistical inference) based on a single observation/sample. However this assumes normality of the distribution, which is unlikely to be correct when dealing with financial data.

New Scientific Methods for Back-Testing

This last concept leads to the introduction of the two alternative methods to derive the sampling distribution and perform statistical inference on the back-tested results. These are two computer-based methods:

• The Bootstrap
• The Monte Carlo permutation

Both methods estimate the sampling distribution by randomly resampling (reusing) the original sample of observation. A test statistic is then computed for each resample.

In practice, the bootstrap method uses resampling with replacement of the daily strategy returns to generate numerous random test statistics used to approximate a sampling distribution.
The Monte Carlo permutation method achieves the same result by decoupling and permuting the position direction (ie. long or short) with the daily instrument returns.

Using the statistical inference covered in earlier chapters, one can decide whether results found in the back-test are statistically significant or the product of random chance.

These two methods are the main take-away from the book, as they are valuable to identify the degree of randomness in a back-tested rule. This should probably be part of a standard trading system research methodology and I will cover these two methods in more detail in later posts.

On Data Mining

The methods above only deal with one rule/back-test. However, we rarely test the one rule in isolation: most back-testing would test multiple parameter values, rules and combinations to try and identify the best performing ones: this is data mining.

It is however wrong to expect future performance of the best performing systems to keep in line with past, back-tested results. The best performing systems might have intrinsic value, but some of their over-performance is due to random variations. If you run 1,000 different rules with no predictive power, all of them will contain some random chance producing a variable departure from the zero-mean. The “most lucky” rule will be furthest away on the right-hand side of the zero-mean (and therefore picked up by the data miner), despite having no intrinsic value.

Data mining introduces a bias, which overstates the value of the “best” rule compared to expected random variations. The data mining bias is linked to several factors:

• Increases with the number of rules back-tested
• Decreases with sample size used in back-testing.
• Decreases with the correlation of back-tested rules results.
• Increases with the frequency of outliers in the back-test sample.
• Decreases with the variation in back-tested returns among rules considered.

This is illustrated with examples and charts. The rest of the chapter concentrate on methods to reduce/correct for the data mining bias and adapts the bootstrap method (using White’s reality check) and Monte Carlo permutation to be used in “data mining” mode (instead of single rule testing).

In conclusion, data mining is a valid method to discover the best rule(s) but the researcher should ensure that the results are statistically significant to avoid the risk of discovering “most lucky” rules.

A Tour of the EMH and Application of Methods

The following chapter deals with the Efficient Market Hypothesis, which takes a bit of a beating by the author. The main point is that both from an empirical and theoretical point of view, the EMH contains flaws, which supports the idea of succesful TA.

The last part of the book presents a diverse set of rules and parameters (6,402 combinations) and attempts to test for their statistical significance. The rules are fairly simple and the results do not highlight significant predictive power in any rule.

Review Conclusion

This book is a very interesting read, on the long side, with 450+ pages. Even though I enjoyed it throughout, I was sometimes finding myself hoping for the author not to expand so much on some introductory topics (the history and philosophy of science is quite interesting but could well be skim-read to get to the “juicier” parts quicker). If you’re in a rush I’d advise to concentrate on chapters 4, 5 and 6 where the actual bootstrap and Monte Carlo methods get presented and discussed, and the discussion on data mining bias is interesting and very relevant. For a reader new to these concepts, the initial chapters would provide a comprehensive introduction of the foundational concepts of scientific reasoning and statistical analysis before putting them all together in application.

For more info, some of the reviews on amazon are quite insightful (mostly positive – although the book’s got its share of 1-star reviews). There is also a companion website to the book with more info and detailed results of the tests performed in the last part of the book.

Related posts:

1. The Bootstrap Test: How significant are your back-testing results?
2. Monte Carlo Permutation: Test your Back-Tests
3. Bootstrap – Take 2: Data Mining bias, Code and using geometric mean

Report Changes

As discussed last month, the system performances are still adjusted for volatility to normalize the results.

The portfolio has been subject to a few amendments, mostly a cleansing operation to remove/update irrelevant instruments (eg. not trading any more, no/very low volume). The updated list can be found at the end of the post with system details. These changes did not impact past results much.

Results for July 2010

Below is the chart of each strategy over the month of July 2010, with the composite average of all systems:

And the results in tabular format:

A bit of a better performance from the short-end side of systems but not so from the medium/long-term end of the scale.

2010, through the Composite Index

Below is how the composite index looks so far for 2010 (no rebalancing in the index):

Appendix: System Details

System Rules and Parameters

All the systems were tested with the same simple position sizing rules of 1% per new trade. No other Money/Risk Management rules were used. No slippage was considered and a $15 RT commission applied. No return on margin is added to the system performance

The system rules are detailed on the Trading Blox online documentation.
The MA Crossover system was used with moving average pairs of 10-20, 20-50 and 50-200 days. The stops/position sizes are set at 2x, 3x and 5x ATR respectively.
The Bollinger Band system is the classic use of the Bollinger Bands with entries taking place at Breakouts. The parameters used were 20, 50 and 200 days with 2 standard deviations.
The Triple moving Average system was used with moving average triplets of 10-20-50, 20-50-200 and 50-200-800 days. The stops/position sizes are set at 2x, 3x and 5x ATR respectively.
The Donchian System is a simple version (with no Trade Direction filter) with channel lengths of 20, 50 and 200 days for entries (and 10, 25, 100 for exit). The stops/position sizes are set at 2x, 3x and 5x ATR respectively.

Portfolio Instruments

Covering over 50 instruments across Equities, Interest Rates, Currencies, Agriculturals, Metals and Energies, from around the world, the portfolio contains the following futures (CSI Symbols): AD, BP, CC, C, CD, CFC, CL2, CT, CU, YM, EBL, EBM, EBS, ED, EOX, ESM, FC, FEI, FFI, GC, HG, ICL, IND, IRB, JK2, JP2, JP6, JR2, JRB, JSK, JTI, JY, KC, KPO, KTB, KWR, LC, LGO, LH, MFX, MP, NG2, RA, RS, SB, S, SF, SI, STW, SXE, TRY, US, W, YTC.
Click here for a tabular view with description and exchange information.

This month’s amendments were:
- Replaced CL with CL2, DJ with YM.
- Removed ET, LAC, LZC, IRB, JSK, KWR, M10, NGV, SJG

Related posts:

1. the State of Trend Following in August: good performance
2. the State of Trend Following in June
3. The State of Trend Following report – Draft V0.2