Net Prophet

March Madness Contest

2012-02-24T15:27:00.000-05:00

Danny Tarlow and Lee-Ming Zen over at This Number Crunching Life have announced their annual March Madness prediction contest. To compete, you use data from this season and past seasons (which Danny & Lee will provide), build a computer system that fills out a bracket, then pit yourself against the field of silicon competition. The posts from last season's tournament can be found here.

I personally know the winner from last year and the previous year, and I can only say that I have the utmost respect for their dedication, intelligence, and ruggedly handsome good looks.

3PT Attempt Percentage

2012-02-23T11:14:00.000-05:00

Ken Pomeroy recently made a couple of blog postings concerning defense, and specifically a statistic he calls the "3 Point Attempt Percentage" (3PA%). He defines this statistic as the "percentage of field-goal attempts that are from three-point range." Ken Pomeroy thinks this is a better measure of defense than 3PT%. His reasoning is that most teams only take 3 point shots when they are relatively unguarded; the effect of defense is not to make these shots harder, but to cut down on the number of opportunities. Hence the claim that it's really how many 3 pointers your opponent takes that reveals the quality of your 3PT defense. Near the end of the second posting he says:

People that are unaware of 3PA% (which is to say nearly everyone) are missing a very telling statistic that explains a lot of how defense works.

This is a strong statement and worthy of a little research to see whether it is true (at least so far as predicting outcomes is concerned).

3PA% is similar to Effective Field Goal Percentage, one of Dean Oliver's Four Factors. I have previously considered the Four Factors and concluded that they didn't add any predictive value to my models, but 3PA% captures a slightly different slice of information.

When I recently looked at derived statistics, one of the derived statistics was pretty close to 3PA%:

(Ave. number of 3PT attempts by the opposing team)
------------------------------------------------------
(Ave. number of FG attempts by the opposing team)

This isn't quite the same statistic, because it is using game averages rather than cumulatives, but it is close. This statistic turned out to have no predictive value, but a couple of statistics based upon 3PT attempts did have value:

(Ave. number of 3PT attempts by the opposing team)
-----------------------------------------------------
(Ave. number of turnovers)

(Ave. number of 3PT attempts by the opposing team)
-----------------------------------------------------
(Ave. number of rebounds)

Note that these statistics are relating the number of 3PT attempts by the opponent to a statistic for the defending team. I'm not entirely sure what these statistics are capturing, but I don't think it is 3PT defense. (The latter might be indirectly saying something about how a team defends against the three pointer, from how it is positioned to rebound effectively or not after a taken three pointer.)

That aside, I modified my models to generate four new statistics: the 3PA% for the home team in previous games, the 3PA% for the away team in previous games, the 3PA% for the home team's opponents in previous games, and the 3PA% for the away team's opponents in previous games. I then tested the model both with and without these statistics:

Model	Error	%Correct
Base Statistical model	11.06	72.7%
Base Statistical model + 3PA% statistics	11.05	72.7%

There's a very small improvement in RMSE with the added 3PA% statistics. So at least for my model, the 3PA% statistics don't seem to add any significant new information.

Performance Versus "The Line"

2012-02-17T13:27:00.000-05:00

As I've mentioned earlier, I use my models to bet (in some theoretical sense) against the "the line". Typically I bet the games where my model differs significantly from the line (e.g., >4 points or so). As I've documented here, I have a number of different models, all of which have around the same performance (~11 points RMSE).

In the past I've usually averaged the predictions of these models for betting purposes, but for some time I've wondered whether they all perform equally well against the line. Although they all have similar errors, it's possible that some of the models error more consistently to the winning side of the line. To test this, I gathered three seasons worth of Vegas closing line data (about 7700 games) and tested each model for how often its predictions were correct versus the line. (The predictor is "correct" if it would make a winning bet given the line.) I also looked at each predictor's error versus the line (i.e., how accurately it predicted the line).

Model	Performance vs. Line	Error vs. Line
TrueSkill	49.89%	3.75
Govan	49.28%	3.49
BGD	49.58%	3.51
Base Statistical	50.12%	4.34
Statistical w/ Derived	50.15%	4.34
All	52.00%	3.49
All (Difference > 2)	53.15%

The "All" model here is a linear predictor using all the inputs to TrueSkill, Govan, BGD and Statistics w/ Derived. (I also tested some voting models, but they all under-perform the Statistical/All models.)

There are a couple of interesting results.

Most noticeably, the "All" predictor is at break-even versus the line. (Due to "house cut" on sports bets, you need to win about 52% of your bets to break even.) If we restrict ourselves to bets where the predictor differs from the line by at least two points, performance moves into (barely) positive territory. This is very good performance; the best predictors tracked at The Prediction Tracker do not even break 50%. (Furthermore, I am using the "closing" line, which is a tougher measure [by about one point] than the opening line used at the Prediction Tracker.)

It's also intriguing that TrueSkill/Govan/BGD all underperform the line but track it noticeably better than the statistical predictor. This suggests to me that the line is set not by wily veteran gamblers in the smoky back rooms, but by a computer program using some kind of team strength measure.

A (possibly interesting) side-note: All models that under-perform the line are going to fall into the seemingly miniscule range of 48-52%. (If a model performs worse than 48% against the line, we would simply bet against the model.) Pick any crazy model you like -- "Always bet the home team," "Always bet on the team whose trainer's name is first alphabetically," etc. -- and the performance is almost certainly going to fall in that 48-52% range against the line. (If it doesn't, you've found the key to beating Vegas!)

The Continued (Slow) Pursuit of Statistical Prediction (Part III)

2012-02-10T17:41:00.000-05:00

As promised last time, we'll now look at a different type of derived statistic. We're going to look at statistics which are the ratio between the two teams of the same base statistic, e.g.,

(Ave # of offensive rebounds for the home team / Ave # of offensive rebounds for the away team)

The idea here is that it may be more predictive to look at the relative strengths of the teams rather than the absolute strengths.

The first statistics I want to try this upon are the strength measures like TrueSkill and RPI. Suppose that Syracuse, with an RPI of 0.6823, plays Missouri, with an RPI of 0.6234, and the same night UCF, with an RPI of 0.5723 plays Oregon State with an RPI of 0.516. Would we expect the same outcome in those games? In both cases, the better team is about 0.06 better in RPI. But Syracuse is about 10% better than Missouri, while UCF is about 12% better than OSU. If it's the relative strength that matters, we would expect UCF to win (on average) by more than Syracuse.

To test this out, I generated the relative strengths for measures like TrueSkill and ran them through my testing setup. In every case, the relative strengths had no predictive value above and beyond the value of the absolute strengths. And when the relative strengths alone were used for prediction, they underperformed the absolutes used alone.

I then did the same thing for the statistical attributes like offensive rebounding and got the same result. The relative strengths of the two teams provided no additional predictive accuracy.

I find this result fairly intriguing. My strong intuition was that at least a portion of the game outcome would be better explained by the relative strengths of the two teams. It's hard to believe that Syracuse should win its game against Missouri by more points simply because they're both stronger teams than UCF and OSU. But (as has often proven to be the case!) my intuition was just wrong, and relative strength is much less important than I would guess.

The Continued (Slow) Pursuit of Statistical Prediction (Part II)

2012-02-03T13:20:00.000-05:00

Continuing on from last time, I had set up the infrastructure to allow me to easily test the value of derived variables in statistical prediction. Before testing any of these derived variables, we need a baseline. In this case, the baseline is the performance of a linear regression using all the base variables. I don't know that I've ever documented the base variables, but they are basically all that can be created from the full game statistics available at Yahoo! Sports. These are averaged by game, so for example one of the base statistics is "Average free throw attempts per game." I also have the capability to average statistics by possession (e.g., "Average free throw attempts per possession" but unlike some other researchers, I've never found per possession averages to be any more useful than per game averages, so I generally don't produce them.

For most statistics, I also produce the average for the team's opponents. So to continue the example above, I produce "Average free throws per game for this team's opponents." I also produce a small number of simple derived statistics, such as "Average Margin of Victory (MOV)", and winning percentages at home and on the road.

When we get to predicting game outcomes, of course we have all of these statistics for both the home and the away team. (And that home/road distinction is important, obviously.) If we use all these base statistics to create a linear regression, we get the following performance:

Predictor	% Correct	MOV Error
Base Statistical Predictor	72.3%	11.10

This is the same performance I have reported earlier, and tracks fairly well with the best performance from the predictors based upon strength ratings.

Now we want to augment that predictor with derived statistics to see if they offer any performance improvement. As mentioned last time, we have 1200 derived statistics, so we have to do some feature selection to thin that crop for testing.

One possibility (as discussed here) is to build a decision tree, and use the features identified in the tree. If we do that (and force the tree to be small), we identify these derived features as important:

The home team's average margin of victory per possession over the overall winning percentage
The away team's average number of field goals made by opponents over average score
The home team's average assists by opponents over the field goals made
The home teams average MOV per game over the home winning percentage

That is, you'd have to admit, quite a goulash of statistics. I can probably come up with some rationale about some of those, but I won't bother. All I really care about is whether they will improve my predictive accuracy.

To test that, I add those statistics to my base statistics and re-run the linear regression. In this case, what I find is that while some of the derived statistics are identified as having high value by the linear regression, the overall performance does not improve.

There are other methods for feature selection, of course. RapidMiner has an extension focused solely on feature extension. This offers a variety of approaches, including selecting based on Maximum Relevance, Correlation-Based Feature Selection, and Recursive Conditional Correlation Weighting. All of these methods identified "important" derived statistics, but none produced a set of features that out-performed the base set.

A final approach is a brute force approach called forward search. In this approach, we start with the base set of statistics, add each of the derived statistics in turn, and test each combination. If any of those combinations improve on the base set, we pick the best combination and repeat the process. We continue this way until we can find no further improvement.

There are a couple of advantages to this approach. First, there's no guessing about what features will be useful -- instead we're actually running a full test every time and determining whether a feature is useful or not. Second, we're testing all combinations in our search space, so we know we'll find the best combination. The caveat here is that we assume that improvement is monotonic with regards to adding features. If the best feature set is "A, B, C" then we're assuming we can find that by adding A first (because it offers the most improvement at the first step), then B to that, and so on. That isn't always true, but in this case it seems a reasonable assumption.

The big drawback of this approach is that it is very expensive. We have to try lots of combinations of features, and we have to run a full test for each combination. In this case, the forward search took about 54 hours to complete -- and since I had to run it several times because of errors or tweaks to the process in ended up taking about a solid week of computer time.

In the end, the forward search identified ten derived features, with this performance:

Predictor	% Correct	MOV Error
Base Statistical Predictor	72.3%	11.10
w/ Forward Search Features	74.0%	10.73

This is a fairly significant improvement. The most important derived features in the resulting model were:

The away team's opponent scoring average over the away team's winning percentage.
The away team's offensive rebounding average over the away team's # of field goals attempted
The away team's scoring average over the away team's winning percentage
The away team's opponent treys attempted over the away team's rebounds

The ten statistics were actually evenly divided between home team statistics and away team statistics, but it turned out that the most significant five were all the away team statistics.

I'll leave it to the reader to contemplate the meaning of these statistics, but there are some interesting suggestions here. The first and third statistics seem to be saying something about whether the away team is winning games through defense or offense. The second and fourth statistics seem to be saying something about rebounding efficiency, and perhaps about whether the team is good at getting "long" rebounds. (The statistics for the home team are completely different, by the way.)

Next time I'll begin looking at a different set of derived statistics.

The Continued (Slow) Pursuit of Statistical Prediction

2012-01-20T16:00:00.000-05:00

When we last met on this topic, I was inspired by the Four Factors to look at derived statistics created from the ratio of two existing statistics, e.g.,

Offensive Balance = (# 3-Pt Attempts) / (# FG Attempts)

My previous work in this area has convinced me of the value of looking at all possibilities, no matter how non-intuitive, my approach was to look exhaustively at all the possible ratios between the ~35 base statistics. That leads to some crazy statistics such as:

(Average # of fouls per possession by the home team's previous opponents) /
(Average offensive rebounds per game by the home team in previous games)

There turn out to be a number of difficulties with this approach. (Perhaps not unsurprisingly, although crazy nonsensical statistics are not one of them.)

First, it's a lot of work just to generate the 1060 derived statistics. (Only 1060 because I avoided inverse ratios, and avoided "cross-ratios" between the two teams.) Initially I was generating a subset of these from within the Lisp code that pre-processes the game data. That was painful to set up and slow to execute. Eventually I discovered a way to generate the derived statistics within RapidMiner. I was able to drive this from a data file, so I wrote a small Lisp program to generate the data file that RapidMiner could use to construct all 1060 derived statistics.

Second, this amount of data tended to overwhelm my tools. With the derived attributes, each game has about 1200 attributes total. My training corpus has about 12K games. The combination tended to break most of the data modeling features of RapidMiner, usually by overwhelming the memory capacity of Java. Even when the software was capable of handling the data volume, operations like a linear regression might take hours (or days!) to complete, so testing and experimenting was laborious at best.

One way to reduce this problem is to thin the dataset, by testing on (say) a tenth of the full corpus. But that introduced a new problem: overfitting. If I used (say) a tenth of the data, I had about 1200 games in my test set -- just about the same number of test games as attributes. The result of that is almost invariably a very specific model that does extremely well on the test data and very poorly on any other data.

Another approach is to thin the attributes. This is the feature selection problem. The idea is select the best (or at least some reasonably good) set of features for a model. The stupid (but foolproof) way to do this is to try every possible combination of features, and select the best combination. But of course that's infeasible in many cases (such as mine), so a variety of alternative approaches have been created. RapidMiner has some built-in capabilities for feature selection, and there's a nice extension to add more feature selection capabilities here.

I experimented with a variety of different feature selection approaches. I was hopeful that different approaches would show overlap and help identify derived attributes that were important, but for the most part that did not happen. However, taking all the attributes recommended by any of the feature selection approaches did give me a more reasonable sized population of derived statistics to test.

More on this topic next time.

Basketball Season Underway

2012-01-17T15:28:00.000-05:00

I spent the last few days scraping game data, dusting off code and generally getting the basketball predictor back online. The current version of the predictor uses an average of 4 linear regressions. These models are based upon: (1) the Govan rating, (2) the TrueSkill rating, (3) a Batch Gradient Descent (BGD) rating, and (4) a rating based on a wide variety of statistical measures (such as "offensive rebounds per possession"). Individually, each of these models has a RMSE of less than 11 on my test corpus. Unfortunately, they're all highly correlated, so the combined model doesn't do any better than the best of the underlying models. Currently it has an RMSE of 10.79 on my test corpus.

During the season I compare the model predictions against the line and "bet" games where the prediction differs significantly from the line. "Significantly" is a relative term. When I first started doing this, my model often differed from the line by 10 points or more. As the model has improved, those differences have narrowed considerably. (As would be expected. The line is usually the best predictor.) In my testing so far this year, I've only seen a difference of more than 5 points once. There is some good mathematical work on sizing wagers based upon bankroll, perceived advantage, etc., but I've gone to a simple approach of betting $10 with an advantage of < 5 points and $20 with an advantage of >5 points. (Adopted after the 1/14 games shown below.)

Here are the games the model has "bet" so far (no real money was harmed):

Date	Home	Score	Away	Score	MOV	Line	Pred	Adv	Risk	Win	Result	Won	v.Line
1/14	Tennessee St.	52	SIU Edwardsville	49	3	16	8.8	-7.2	20	17.39	17.39	1	1
1/14	LA Lafayette	87	Florida Intl.	81	6	10	5.1	-4.9	20	19.05	19.05	1	1
1/14	Murray St.	81	Tennessee Tech	73	8	12	16.5	4.5	20	18.18	-20	1	0
1/14	Houston	55	Memphis	89	-34	-8.5	-4.2	4.3	20	17.39	-20	1	0
1/15	Ohio St.	80	Indiana	63	17	13.5	9.1	-4.4	10	9.09	-10	1	0
1/15	Bradley	78	Northern Iowa	67	11	-10	-7.2	2.8	10	8.70	8.70	0	1
1/15	USC	47	UCLA	66	-19	2	1.5	-0.5	10	9.09	9.09	0	1
1/16	Syracuse	71	Pittsburgh	63	8	13.5	17.3	3.8	10	9.09	-10	1	0

So far this season the model is 50% against the line (and subsequently down about $5) and 75% picking the correct outcome. The (evolving) model picked 38 games last year, and over the two seasons so far is at a 63% win percentage and 60% versus the line (+$133). Both are probably short-term aberrations -- the model has a 74% win percentage when tested against my corpus of 12K games.

I won't generally be posting predictions, but I will try to summarize the model's performance a few times during the season, as I'm sure it makes for interesting reading :-).

Football Wrap-Up

2012-01-11T16:49:00.001-05:00

A quick wrap-up of my performance predicting NCAA football.

This experiment started around the beginning of October, when some friends challenged me to use my program to predict football against a couple of other guys. In addition to predicting games, we would be "betting" against the line. We could use any betting strategy we desired to allocate $40 per week. The default strategy was to bet the biggest differences between the prediction and the line, allocating bets of $10, $8, $6... etc. My own betting strategy was a bit more complex. I allocated money according to the formula:

$$ =  80*ABS(Prediction-Line)/(100+5*ABS(Line)))

.The idea being to scale the bet to the relative magnitude of the difference between the prediction and the line. A difference of 3 points is much more significant when the line is 3 than when the line is 27.

I predicted games from Oct 16 through the end of the bowl season. My program doesn't account for neutral site games, so the bowl games were treated as home games for the higher-ranked team. (This works well in practice on the basketball side for the NCAA tournament.) I predicted a total of 224 games. The results:

Measure	Performance
Correct game winner	73%
Correct pick against the line	56%
Betting result	+$29

Overall, better results than I expected. 56% against the line is sufficient to be a winning bettor (if it can be maintained).

A Call from Bill Hancock

2012-01-05T11:15:00.000-05:00

(I promise to get back to the prediction stuff soon -- after a dalliance with NCAA football I've started to ramp back up for basketball. In the meantime, this imagined scenario from last night, inspired by my earlier comment that the NCAA was only allowing ten players on defense to spice up the bowl season.)

Phone call at spacious Turner Mansion last night:

(Ring, Ring)

Me: Hello? Oh, hello Mr. Hancock. How is your job as BCS Executive Director going?

Hancock: (mumble mumble mumble)

Me: Well, you're welcome. I'm glad my suggestion to only play ten players on defense has worked out so well.

Hancock: (mumble mumble mumble)

Me: *Nine* on defense? No, I'm not sure that's a good idea. We've been counting on the fact that most sports writers can't count past ten. So far they haven't noticed. But you put nine players out there and someone is going to write about it. And where does it all end? Eight players? Seven players?

Hancock: (mumble mumble mumble)

Me: No, sir, that was a joke. I'm not recommending seven players on defense. Listen, I don't think this is a good idea. Baylor just obliterated the points scoring record for a bowl game. This is Baylor, the doormat of the Big 12, a university whose only men's championship is in *tennis*. And then you had Wisconsin -- Wisconsin of all teams! -- throwing the ball all over the field and scoring 38 points. That's more than the Wisconsin basketball team scored last season. I realize you want to turn it up to eleven for the Orange Bowl, but this is not a good idea.

Hancock: (mumble mumble mumble)

Me: You're worried about Clemson's defense? With all due respect, sir, Clemson is an ACC team. The last time the ACC won a meaningful bowl game it was actually played for a bowl. If you gave the ACC space lasers they couldn't defend Fort Knox against a Boy Scout troop.

Hancock: (mumble mumble mumble)

Me: True, it is West Virginia.

Hancock: (mumble mumble mumble)

Me: No, sir, West Virginia is part of the United States.

Hancock: (mumble mumble mumble)

Me: No apology necessary. It's a common misconception.

Hancock: (mumble mumble mumble)

Me: Well, you do what you have to do, sir. Personally, I'm a traditionalist. Just tell the officials the result and let them take care of it. That's worked for Duke basketball for decades and no one's the wiser. Do they have a "charging" call in football? I can't remember. But I'm sure you'll make a good decision.

Hancock: (mumble mumble mumble)

Me: "Bet the over"? Ha, ha, good one, sir.

k-NN Prediction

2011-11-15T16:45:00.001-05:00

"Andywocky" commented not too long ago on my Prediction by Similarity posting asking whether I'd looked at k-nearest neighbors (k-NN) algorithms. At the time I made the original posting I hadn't, but shortly thereafter I had a "D'oh" moment and realized that what I was doing was re-creating k-NN. So I re-created some of the work I'd done using RapidMiner's k-NN operator.

The basic idea behind k-NN is that we predict the outcome of a new game by finding some number of similar past games, and then use those (say by averaging) to create a prediction for the new game. The "k" in "k-NN" refers to the "some number" of similar past games -- k might be 5 or 50, indicating that we were using the five most similar, or 50 most similar past games. "Nearest Neighbor" is just another way of saying similar. If we think of the games living in a multi-dimensional space -- say a dimension for each statistical value for the game (e.g., rebounds per minute, free throw percentage, etc.) -- then the most similar games are the ones that are the nearest neighbors in this multidimensional space.

There are some subtleties in how this works. For example, team free throw percentage might vary from (say) 50% to 100%, while rebounds per minute might vary from 0.00 to 0.056. If we don't normalize those dimensions, one or the other is likely to be far more important in determining the nearest neighbor than the other. But a reasonable starting approach is to characterize each game with as many statistical properties as we have, normalize those to similar scales, and then predict MOV by averaging the MOVs of k nearest-neighbors.

Here's the result of doing that with k=10. For comparison, I include the performance of the best linear regression predictor based upon the same statistical properties.

Predictor	% Correct	MOV Error
Best Statistical Predictor	72.3%	11.04
k-NN, k=10	59.7%	11.65

This isn't tremendous performance, but we have a few tweaks we can perform. First, we can try varying k to see if some different number of neighbors provides better performance. Some searching around produces the best performance in this case when k=41:

Predictor	% Correct	MOV Error
Best Statistical Predictor	72.3%	11.04
k-NN, k=10	59.7%	11.65
k-NN, k=41	71.2%	11.44

Interestingly, this shows a lot of improvement in games correct with only modest improvement in MOV error.

Another tweak we can look at is weighting our results. Instead of doing a flat average of the 41 nearest neighbors, we can weight each neighbor's contribution to the answer by how close it is to the new game. We can also try eliminating some of our dimensions to see if accuracy improves. This provides some further improvement:

Predictor	% Correct	MOV Error
Best Statistical Predictor	72.3%	11.04
k-NN, k=10	59.7%	11.65
k-NN, k=41	71.2%	11.44
k-NN, k=44, weighted subset	72.4%	11.17

With this tweak k-NN is competitive with the best linear regression. (Although they both trail the best predictors.)

I'm inclined to draw a couple of conclusions from these experiments. First, 40+ neighbors is a large number, suggesting that while games between statistically similar may be broadly comparable, there's not a strong relationship. Conversely, the improvement gained by using weighting suggests that closer is still better. It would seem that good performance with this approach requires a moderate amount of generalization to help "wash out" the random component in game outcomes.

Football Predictions (11/15/11)

2011-11-15T10:44:00.000-05:00

I hope to have some time in the next day or so for some postings, so here are the predictions for this week in NCAA football. I've been tracking this performance for a contest, and for the past three weeks I'm 56% against the line (and 71% winners). That might be anomalously good performance, but I've been positive against the line every week, so take that for what it is worth.

As always, heed the Net Prophet Disclaimer.

NCAA Football Predictions (11/15/11)
Home Team	Away Team	MOV
Air Force	Nevada-Las Vegas	16.1
Arizona State	Arizona	18
Arkansas	Mississippi State	13.1
Baylor	Oklahoma	-14.2
Bowling Green State	Ohio	-6.7
Brigham Young	New Mexico State	15.7
Buffalo	Akron	12.3
Central Michigan	Toledo	-14.9
Connecticut	Louisville	0.4
Duke	Georgia Tech	-6.9
East Carolina	Central Florida	-8
Florida State	Virginia	16.8
Georgia	Kentucky	24.5
Hawaii	Fresno State	9
Houston	Southern Methodist	18.3
Idaho	Utah State	-8.3
Illinois	Wisconsin	-9.9
Iowa State	Oklahoma State	-17
Kent	Eastern Michigan	6
Louisiana-Monroe	Florida International	-3.3
Memphis	Marshall	-15.4
Miami (Ohio)	Western Michigan	-2.4
Michigan	Nebraska	10.2
Michigan State	Indiana	20.9
Middle Tennessee State	Arkansas State	-11.5
Mississippi	Louisiana State	-26.3
Missouri	Texas Tech	15.2
Nevada	Louisiana Tech	1.8
North Carolina State	Clemson	-9.1
North Texas	Western Kentucky	2
Northern Illinois	Ball State	11.5
Northwestern	Minnesota	14.3
Notre Dame	Boston College	22.3
Ohio State	Penn State	1
Oregon	Southern California	14.9
Oregon State	Washington	-0.4
Purdue	Iowa	-1.6
Rice	Tulane	15
Rutgers	Cincinnati	0.8
San Diego State	Boise State	-13.6
San Jose State	Navy	-1.8
South Florida	Miami (Florida)	3.9
Stanford	California	20.3
Temple	Army	15.8
Tennessee	Vanderbilt	2.6
Texas	Kansas State	3
Texas A&M	Kansas	23.8
Texas Christian	Colorado State	24.4
Texas-El Paso	Tulsa	-10.4
Troy	Florida Atlantic	11.2
Uab	Southern Mississippi	-26.2
Ucla	Colorado	12.2
Virginia Tech	North Carolina	6
Wake Forest	Maryland	11.1
Washington State	Utah	-2.4
Wyoming	New Mexico	20.4

One Bad (Good) Game

2011-11-08T13:16:00.000-05:00

As mentioned in my previous posting, I recently looked at the effect of dropping a football team's best game (highest Margin of Victory) and their worst game (lowest MOV). The intuitive notion is that everybody has bad days, where everything goes wrong, and good days, where everything goes right, and maybe those days don't tell us anything useful about the real strength of a team. If that's so, then dropping those games might give us ratings that are more accurate.

To test this hypothesis I implemented this "drop the worst score" grading system for a couple of the rating systems I use for football and measured performance in the usual way. Here are the results for one of the rating systems:

Predictor	% Correct	MOV Error
BGD Baseline	73.7%	16.52
BGD w/o blowouts or lowouts	72.6%	16.77
BGD w/o lowouts	72.9%	16.69
BGD w/o blowouts	73.6%	16.62

Here I'm using the whimsical "lowout" to indicate the worst loss for a team.

As this shows, eliminating the blowouts/lowouts hurts predictive performance. For what it's worth, the losses seem to be more important than the wins. (I saw the same effect in basketball when I looked at this last year.)

The Impact of MOV Cutoffs in Football Ratings

2011-11-04T16:07:00.000-04:00

I was prompted to start my football predictions by a discussion on an email list of the value of MOV cutoffs in rating systems. Roger Dendy believed that capping the MOV in blowout victories improved his rating system. My testing of MOV cutoffs in basketball has shown just the opposite -- that no matter how big the blowout, there's always information in the margin of victory. Capping MOV at any level (in both blowouts and nailbiters) always reduces the prediction value of a rating.

Of course, just because that's true in basketball doesn't mean it's true in football. I was pretty sure it was true, but I believe in "trust but verify." So I put together the football predictor and tested a couple of different rating systems both with and without MOV caps.

I have many rating systems that use MOV, so I picked one and measured it's performance with a 100-fold X-validation across my archive of college football scores from 2005 to date. It had a RMS of 16.78 and predicted 71% of the games correctly.

Then I experimented with adding a cutoff to the MOV. I set the cutoff to 32 points, so that all the games where the MOV exceeded 32, it would be treated as 32. I just picked 32 arbitrarily as a good figure for a blowout win. The performance degraded to RMS=17.19 and 69%. I then bumped up the cutoff to 48 points, and the performance was RMS=17.01 and 70%.

The other rating system showed a similar pattern of performance.

What this shows -- at least for the two rating systems I tested and these performance metrics -- is that even huge margins of victory have value in assessing future performance. People argue intuitively that there's "no difference between winning by 48 and winning by 52" but that appears not to be true.

Recently I got to wondering if it might not make more sense to drop a blowout victory entirely. This would be like "drop your lowest score" grading in high school. The intuitive notion here is that sometimes teams just have a bad day -- a few unlucky bounces and worse goes to worse. Or lucky bounces and better goes to better, from the other side of the coin. More on that notion next time.

Football Predictions

2011-11-02T17:26:00.000-04:00

Shown below are predictions for this week's upcoming football games. I did a little tweaking and developed a new algorithm for this week's predictions so you'll see two predictions below.

The first is the new algorithm, the second is an ensemble of three algorithms. The new algorithm is similar to the "Homemade Sagarin Ratings" described here (although I do not use Excel Solver to calculate my ratings). The Sagarin ratings do very well at the Prediction Tracker, so I wanted to implement something similar and see how it did in comparison to my other predictors. Much to my surprise, it equals or surpasses my best football predictors. In past tests on the basketball data, this type of predictor did not perform well, but in the course of implementing it for football I found several problems, so I intend to retest this on the basketball data and look at some possible improvements if warranted.

If anyone knows a better description of the Sagarin PREDICTOR algorithm, please let me know.

As always when viewing my predictions, heed the Disclaimer.

+-------------------+----------------+---------------+---------------+
|Hname              |Aname           |prediction(1)  |prediction(2)  |

+-------------------+----------------+---------------+---------------+
|wisconsin          |purdue          |23.4           |19.2           |

+-------------------+----------------+---------------+---------------+
|west virginia      |louisville      |11.6           |9.3            |

+-------------------+----------------+---------------+---------------+
|maryland           |virginia        |4.1            |1.5            |

+-------------------+----------------+---------------+---------------+
|rice               |texas-el paso   |-.3            |1.0            |

+-------------------+----------------+---------------+---------------+
|texas              |texas tech      |11.1           |11.1           |

+-------------------+----------------+---------------+---------------+
|wyoming            |texas christian |-16.7          |-15.5          |

+-------------------+----------------+---------------+---------------+
|tennessee          |middle          |22.3           |20.5           |

|                   |tennessee state |               |               |
+-------------------+----------------+---------------+---------------+

|oregon state       |stanford        |-18.7          |-20.2          |
+-------------------+----------------+---------------+---------------+

|east carolina      |southern        |-13.0          |-10.4          |
|                   |mississippi     |               |               |

+-------------------+----------------+---------------+---------------+
|southern           |tulane          |19.0           |18.1           |

|methodist          |                |               |               |
+-------------------+----------------+---------------+---------------+

|san jose state     |idaho           |10.7           |9.9            |
+-------------------+----------------+---------------+---------------+

|san diego state    |new mexico      |32.9           |29.0           |
+-------------------+----------------+---------------+---------------+

|rutgers            |south florida   |2.1            |1.4            |
+-------------------+----------------+---------------+---------------+

|washington         |oregon          |-13.7          |-12.8          |
+-------------------+----------------+---------------+---------------+

|oklahoma state     |kansas state    |15.3           |14.2           |
+-------------------+----------------+---------------+---------------+

|oklahoma           |texas a&m       |17.7           |17.9           |
+-------------------+----------------+---------------+---------------+

|ohio state         |indiana         |21.7           |21.2           |
+-------------------+----------------+---------------+---------------+

|wake forest        |notre dame      |-10.0          |-10.5          |
+-------------------+----------------+---------------+---------------+

|north carolina     |north carolina  |-4.9           |-6.4           |
|state              |                |               |               |

+-------------------+----------------+---------------+---------------+
|nebraska           |northwestern    |12.2           |13.1           |

+-------------------+----------------+---------------+---------------+
|navy               |troy            |7.4            |6.5            |

+-------------------+----------------+---------------+---------------+
|baylor             |missouri        |2.0            |4.4            |

+-------------------+----------------+---------------+---------------+
|michigan state     |minnesota       |22.9           |22.8           |

+-------------------+----------------+---------------+---------------+
|iowa               |michigan        |-11.3          |-14.1          |

+-------------------+----------------+---------------+---------------+
|miami (florida)    |duke            |8.7            |8.4            |

+-------------------+----------------+---------------+---------------+
|fresno state       |louisiana tech  |-3.2           |-4.9           |

+-------------------+----------------+---------------+---------------+
|louisiana-lafayette|louisiana-monroe|11.3           |13.6           |

+-------------------+----------------+---------------+---------------+
|kentucky           |mississippi     |-2.1           |-.8            |

+-------------------+----------------+---------------+---------------+
|iowa state         |kansas          |19.3           |19.7           |

+-------------------+----------------+---------------+---------------+
|alabama-birmingham |houston         |-31.4          |-30.6          |

+-------------------+----------------+---------------+---------------+
|hawaii             |utah state      |1.2            |.7             |

+-------------------+----------------+---------------+---------------+
|georgia            |new mexico state|24.8           |25.2           |

+-------------------+----------------+---------------+---------------+
|western kentucky   |florida         |-3.3           |-3.9           |

|                   |international   |               |               |
+-------------------+----------------+---------------+---------------+

|florida            |vanderbilt      |8.6            |9.9            |
+-------------------+----------------+---------------+---------------+

|eastern michigan   |ball state      |4.2            |4.7            |
+-------------------+----------------+---------------+---------------+

|connecticut        |syracuse        |-1.6           |-3.3           |
+-------------------+----------------+---------------+---------------+

|pittsburgh         |cincinnati      |-1.3           |-2.0           |
+-------------------+----------------+---------------+---------------+

|california         |washington state|3.1            |3.5            |
+-------------------+----------------+---------------+---------------+

|nevada-las vegas   |boise state     |-30.6          |-28.2          |
+-------------------+----------------+---------------+---------------+

|florida atlantic   |arkansas state  |-12.7          |-15.1          |
+-------------------+----------------+---------------+---------------+

|arkansas           |south carolina  |.6             |1.0            |
+-------------------+----------------+---------------+---------------+

|ucla               |arizona state   |-7.5           |-7.2           |
+-------------------+----------------+---------------+---------------+

|arizona            |utah            |1.3            |-1.7           |
+-------------------+----------------+---------------+---------------+

|alabama            |louisiana state |3.9            |5.9            |
+-------------------+----------------+---------------+---------------+

|air force          |army            |11.1           |12.8           |
+-------------------+----------------+---------------+---------------+

|colorado           |southern        |-13.6          |-16.1          |
|                   |california      |               |               |

+-------------------+----------------+---------------+---------------+
|kent               |central michigan|5.5            |8.4            |

+-------------------+----------------+---------------+---------------+
|central florida    |tulsa           |-2.5           |1.0            |

+-------------------+----------------+---------------+---------------+
|miami (ohio)       |akron           |13.8           |13.3           |

+-------------------+----------------+---------------+---------------+
|boston college     |florida state   |-13.5          |-13.3          |

+-------------------+----------------+---------------+---------------+

NCAA Basketball Schedule Data

2011-11-02T17:17:00.000-04:00

I have provide on this page links to a file containing the currently published schedule of games for the upcoming basketball season. I scraped this today from Yahoo Sports so it may be missing some games that have not yet been scheduled, tournament games, etc. The format is self-explanatory and designed for easy ingest by Lisp, but should be easily translated to CSV or other format. All fields are enclosed with quotes for easy parsing.

At the same page I've also provided a listing of conferences and team names. The team names correspond to the names used in the schedule and on Yahoo Sports. This is the same conference file I used last year -- I don't believe there have been any conference changes, but if so let me know and I'll update the file.

Football Predictions

2011-10-28T13:46:00.000-04:00

Here are college football predictions for this week. I discovered a couple of different bugs in my input data since last weeks predictions; these should be somewhat better. Apologies as always for the old-school formatting, and heed my Disclaimer as well.

+--------------------+--------------------+--------------------+
|Hname               |Aname               |prediction(mov)     |
+--------------------+--------------------+--------------------+
|ohio state          |wisconsin           |-11.7               |
+--------------------+--------------------+--------------------+
|western michigan    |ball state          |15.5                |
+--------------------+--------------------+--------------------+
|washington          |arizona             |7.4                 |
+--------------------+--------------------+--------------------+
|duke                |virginia tech       |-4.3                |
+--------------------+--------------------+--------------------+
|utah                |oregon state        |1.5                 |
+--------------------+--------------------+--------------------+
|ucla                |california          |1.7                 |
+--------------------+--------------------+--------------------+
|central florida     |memphis             |26.0                |
+--------------------+--------------------+--------------------+
|tulsa               |southern methodist |5.4                 |
+--------------------+--------------------+--------------------+
|texas tech          |iowa state          |18.5                |
+--------------------+--------------------+--------------------+
|texas a&m           |missouri            |16.1                |
+--------------------+--------------------+--------------------+
|texas               |kansas              |22.7                |
+--------------------+--------------------+--------------------+
|southern california |stanford            |-10.0               |
+--------------------+--------------------+--------------------+
|texas-el paso       |southern mississippi|-8.1                |
+--------------------+--------------------+--------------------+
|tennessee           |south carolina      |-.1                 |
+--------------------+--------------------+--------------------+
|san diego state     |wyoming             |20.3                |
+--------------------+--------------------+--------------------+
|rutgers             |west virginia       |2.5                 |
+--------------------+--------------------+--------------------+
|penn state          |illinois            |5.0                 |
+--------------------+--------------------+--------------------+
|oregon              |washington state    |24.9                |
+--------------------+--------------------+--------------------+
|oklahoma state      |baylor              |7.3                 |
+--------------------+--------------------+--------------------+
|notre dame          |navy                |17.6                |
+--------------------+--------------------+--------------------+
|indiana             |northwestern        |-5.7                |
+--------------------+--------------------+--------------------+
|north carolina      |wake forest         |5.3                 |
+--------------------+--------------------+--------------------+
|new mexico state    |nevada              |-6.2                |
+--------------------+--------------------+--------------------+
|nebraska            |michigan state      |-6.6                |
+--------------------+--------------------+--------------------+
|kentucky            |mississippi state   |-11.1               |
+--------------------+--------------------+--------------------+
|michigan            |purdue              |22.1                |
+--------------------+--------------------+--------------------+
|miami (ohio)        |buffalo             |1.0                 |
+--------------------+--------------------+--------------------+
|maryland            |boston college      |6.8                 |
+--------------------+--------------------+--------------------+
|marshall            |alabama-birmingham |14.2                |
+--------------------+--------------------+--------------------+
|louisville          |syracuse            |-5.3                |
+--------------------+--------------------+--------------------+
|louisiana tech      |san jose state      |11.6                |
+--------------------+--------------------+--------------------+
|louisiana-monroe    |western kentucky    |-6.2                |
+--------------------+--------------------+--------------------+
|middle tennessee    |louisiana-lafayette |4.7                 |
|state               |                    |                    |
+--------------------+--------------------+--------------------+
|kansas state        |oklahoma            |-4.0                |
+--------------------+--------------------+--------------------+
|minnesota           |iowa                |-13.5               |
+--------------------+--------------------+--------------------+
|idaho               |hawaii              |-8.0                |
+--------------------+--------------------+--------------------+
|florida             |georgia             |2.4                 |
+--------------------+--------------------+--------------------+
|florida state       |north carolina state|13.2                |
+--------------------+--------------------+--------------------+
|east carolina       |tulane              |9.9                 |
+--------------------+--------------------+--------------------+
|nevada-las vegas    |colorado state      |1.3                 |
+--------------------+--------------------+--------------------+
|georgia tech        |clemson             |-4.4                |
+--------------------+--------------------+--------------------+
|akron               |central michigan    |-1.8                |
+--------------------+--------------------+--------------------+
|kent                |bowling green state |-5.2                |
+--------------------+--------------------+--------------------+
|auburn              |mississippi         |11.5                |
+--------------------+--------------------+--------------------+
|arkansas state      |north texas         |15.2                |
+--------------------+--------------------+--------------------+
|vanderbilt          |arkansas            |-5.3                |
+--------------------+--------------------+--------------------+
|arizona state       |colorado            |26.3                |
+--------------------+--------------------+--------------------+
|new mexico          |air force           |-16.8               |
+--------------------+--------------------+--------------------+
|florida             |troy                |11.7                |
|international       |                    |                    |
+--------------------+--------------------+--------------------+
|pittsburgh          |connecticut         |11.4                |
+--------------------+--------------------+--------------------+
|brigham young       |texas christian     |-10.1               |
+--------------------+--------------------+--------------------+
|miami (florida)     |virginia            |15.8                |
+--------------------+--------------------+--------------------+
|houston             |rice                |21.5                |
+--------------------+--------------------+--------------------+

Predicting the Oblong Ball

2011-10-20T23:15:00.000-04:00

I was recently challenged by some friends to predict NCAA college football, so I gathered up some historical data from this archive and adapted some of the better rating systems I've investigated to create a predictor. It's hard to judge the performance. It does not perform as well as the systems reported here according to my standard cross-validation testing, but my implementation of Sagarin's ELO also underperforms the reported performance. Since my implementation of ELO tracks the Sagarin performance very well in basketball, I suspect there's a systemic difference in how performance is measured.

At any rate, I don't intend to spend a lot of time on this, but just for amusement, here are the predictions for this weeks games:

alabama over tennessee by 10.6

arkansas over mississippi by 14.8

ball state over central michigan by 1.8

boise state over air force by 25.7

california over utah by -11.4

central florida over alabama-birmingham by 21.2

clemson over north carolina by 5.1

florida atlantic over middle tennessee state by -12.4

florida state over maryland by 5.9

hawaii over new mexico state by .5

houston over marshall by 12.9

illinois over purdue by 11.0

iowa over indiana by 6.5

kansas state over kansas by 18.7

louisiana state over auburn by 14.4

louisiana-lafayette over western kentucky by 6.5

miami (florida) over georgia tech by -15.4

navy over east carolina by 5.2

nebraska over minnesota by 14.4

nevada over fresno state by 7.8

north texas over louisiana-monroe by 2.7

northern illinois over buffalo by 3.1

notre dame over southern california by 2.5

ohio over akron by 21.1

oklahoma state over missouri by 9.9

oklahoma over texas tech by 9.4

oregon over colorado by 22.8

penn state over northwestern by 10.9

rutgers over louisville by 12.0

south florida over cincinnati by -6.9

southern mississippi over southern methodist by -4.4

stanford over washington by 17.7

temple over bowling green state by 17.2

texas a&m over iowa state by 14.5

texas christian over new mexico by 21.9

texas-el paso over colorado state by -1.8

toledo over miami (ohio) by 16.4

tulane over memphis by 10.2

tulsa over rice by 4.4

ucla over arizona by 1.6

utah state over louisiana tech by -2.9

vanderbilt over army by 5.1

virginia tech over boston college by 15.6

virginia over north carolina state by -4.4

wake forest over duke by -2.3

washington state over oregon state by 5.2

west virginia over syracuse by 3.2

western michigan over eastern michigan by 13.7

wisconsin over michigan state by 7.0

Apologies for the awful formatting -- I put this together in 3 days and didn't put much effort in to making pretty.

The Usual Disclaimers apply: Use this information at your own risk; it is not intended for gambling purposes and the Net Prophet does not encourage or recommend gambling on sports events.

More on Statistical Prediction

2011-10-12T11:06:00.004-04:00

I am continuing to explore statistical prediction. In particular, after implementing the Four Factors as described here, I became interested in examining other statistics generated from the base set of statistics. A subset of these generated statistics are ratios of the base statistics, like the "Offensive Balance" statistic I defined in my earlier post:

Offensive Balance = (# 3 Pt Attempts) / (# FG Attempts)

You can probably come up with a few sensible statistics like these off the top of your head. But since I've seen time and again the value of exploring all options -- even the ones that make no "sense" -- I decided to calculate and test all of these sorts of ratios to see which of them (if any) have predictive value.

That's a more difficult job than you might imagine. In my data sets there are 13 base statistics per team per game (FG Made, FG Attempted, 3PT Made, 3PT Attempted, FT Made, FT Attempted, Offensive Rebounds, Total Rebounds, Assists, Turnovers, Steals, Fouls, Score, and MOV). For predictive purposes, we want to use the average of these over a team's previous games [1] and we can average by either game or possession - so that's 26 base statistics per team. There are 26*25 = 650 possible ratios of those statistics. But we also want to consider ratios not only of a team with itself but also of the team with its opponent, e.g., the ratio of the team's average number of 3 PT attempts in past games to it's opponents average number of 3 PT attempts in past games. That adds another 676 possible ratios. Finally, we also want to consider the statistics for a team's past opponents, e.g., the average number of 3 PT attempts in past games of a team's opponents in those games. Adding those in creates a lot more ratios. Multiply all that by the 12K games in my training data, and it's a lot of data.

My approach is to generate a subset of the possible ratios and test them for predictive value. For various reasons I settled on generating all the ratios with a particular numerator, e.g.,

(FG Made) / (# Fouls)
(FG Made) / (Opponent's # Fouls)
(FG Made) / (# Fouls by Opponents in Past Games)
etc.

This ends up adding about 96 new statistics to every game in the database. I can then take this expanded data and pump it through the usual linear regressions, etc., to find the statistics that have predictive value. But this is a slow process -- for each numerator, it takes hours to generate all the statistics and run them through iterations of the predictive model. (This has the disadvantage that I may miss some combination of generated statistics with different numerators that are only valuable in combination.)

So far, I haven't identified any ratios that result in significantly better predictions. But I have been surprised that (at least so far) the models have selected a number of unexpected ratios as being of value. For example:

(Away team's Average FG Made) / (Away team's Average 3PTs Attempted)
(Away team's Average FG Made) / (Away team's Average 3PTs Made)

These ratios seem to be capturing something about the Away team's offensive balance between inside and outside play. Interestingly, both the ratio with 3 PTs Attempted and 3 PTs Made are significant -- it may be that the first captures the "offensive strategy" (whether a team plays outside first or inside first) and the second captures something about how effective they are at executing that strategy. It's also interesting that these ratios are only significant for the Away team -- apparently the home team's performance doesn't depend strongly on what sort of offensive strategy it uses.

Another interesting statistic:

(Home team's Average FG Made) / (Home team's Past Opponents' Average Offensive Rebounds)

It takes a moment's thought to grasp this statistic. It compares the average number of FGs made by a team to the offensive rebounding of the opponents the team faced. If we take Offensive Rebounds as an indicator of how strongly teams are contesting inside play, then this ratio would seem to say something about how effective the home team's inside play has been relative to its opponents.

Hopefully working through all the ratio statistics will turn up a set of statistics that provide significantly better predictive value.

[1] Averaging isn't the only option here, and there are other possibilities for generated statistics that might be useful, but I feel that ratios are a reasonably fertile area for exploration.

2011-09-27T22:04:00.001-04:00

Blogger seems to be rolling out some new template options. You can view Net Prophet in the new templates here.

Statistical Prediction: Pace-Adjusted Statistics & The Four Factors

2011-09-21T15:58:00.000-04:00

There is much talk in sports statistics circles about pace-adjusted statistics. As Wikipedia puts it:

A key tenet for many modern basketball analysts is that basketball is best evaluated at the level of possessions.

The notion here is that because teams play at different paces, game-level statistics can be misleading. A team that averages 95 points per game is not necessarily better than one that averages 78 points per game. The higher-scoring team may simply be playing at a much faster pace. We can account for this by measuring statistics per possession rather than per game.

While this makes a lot of intuitive sense, I always like to test my intuitions. So I took the same set of statistics used in this posting and re-calculated them as per-possession statistics. (See here for how to estimate the number of possessions in a game.) Then I ran the prediction model using the per-possession statistics. (Obviously some statistics, like "Field Goal Shooting Percentage" are not calculated on a per-game basis, so those don't get pace-adjusted.) Here is the performance comparison:

Predictor	% Correct	MOV Error
Govan + Averaging	73.5%	10.80
Statistical prediction (per-game stats)	72.2%	11.09
Statistical prediction (per-possession stats)	72.2%	11.10

As you can see, the two approaches were indistinguishable. Not only was performance nearly identical, but they both selected the same statistics for the prediction model. So at least for this case, it doesn't appear that adjusting for pace improves performance.

My guess is that the relative unimportance of pace is due to the shot clock and the copycat nature of coaching. There probably isn't enough pace variation across teams to make it a significant factor.

If you search around for "pace-adjusted statistics" you'll eventually stumble across Ken Pomeroy's Four Factors page. The four factors are derived statistics that are intended to give additional insight into how teams play. The factors are:

Effective field goal percentage
Turnover percentage
Offensive rebounding percentage
Free throw rate

(Definitions can be found on Ken Pomeroy's page.)

"Effective FG%" is not of interest to me because the linear regression can adjust the relative importance of field goals versus three-point attempts. "Turnover %" is turnovers per possession; that's one of the statistics I calculated as part of the per-possession statistics experiment above. (It had no value in the predictor, fwiw.) "Offensive rebounding %" is a more interesting statistics, and since offensive rebounds are used by the statistical prediction model, this seems like a worthwhile statistics to investigate. "Free throw rate" seems to capture some notion about how often a team draws a foul. I think that's already captured, but it isn't difficult to generate this statistic.

If I generate these two new statistics and run the prediction model, I find that performance remains the same, but the "Offensive rebounding %" statistics replace the per-game or per-possession offensive rebounding statistics. ("Free throw rate" has no predictive value and is eliminated in the linear regression.)

Since three point shooting percentages are used in the predictor, I decided to define a new statistic to capture how much a team relies on the three-point shot (and impacts its opponents use of the three-point shot). I defined this as:

Offensive Balance = (# 3 Pt Attempts) / (# FG Attempts)

and re-ran the predictor. The new statistic has no predictive value. An alternative formulation is to look at the made 3 pointers versus the made field goals:

Offensive Balance = 3*(# 3 Pt Made) / 2*(# FG Attempts)

but again, this statistic has no predictive value.

I'm open to suggestions if anyone out there has any thoughts on similar "derived statistics" that might be of value in prediction.

Statistical Prediction: Normalizing Inputs

2011-09-19T11:25:00.002-04:00

One thing we want to consider in doing statistical prediction (or any sort of prediction where we have a variety of dissimilar inputs) is to normalize our inputs. The purpose of this is to be able to compare inputs that have different scales. For example, in my data set, home team scoring average varies from 43 to 102, while "steals by the away team" varies from 0 to 13, so it's hard to compare those two numbers directly. And we don't want our prediction model to favor one data over another just because it has a bigger absolute value. To address this we can "normalize" our data to similar scales.

I mentioned here that Brady West normalizes all the input data to his model by subtracting the mean and dividing by the standard deviation -- this is called "standard score." Instead of knowing that the home team scored 108 points, you'd know that they score 2.38 standard deviations above the mean. That sounds like a fine approach to me, but as it turns out, RapidMiner (the tool I'm using to do the predictive models) doesn't offer that as an option. It does, however, offer a z-transformation, which transforms the data so that it has a mean of zero and a standard deviation of 1. If we apply that to all of our inputs, we'll have more of an apple-to-apples comparison. For example, the home scoring average ends up ranging from -9.96 to 3.99, while the away team's FT percentage varies from -14.34 to 4.87 -- giving you some sense that there is more variance in FT shooting percentage.

If we apply the z-transformation to our inputs, there is no change in performance for the model that takes only scoring averages. That's reasonable, since the scoring averages are all basically on the same scale anyway. But when we throw in a second data point with a different scale, the difference becomes apparent:

Predictor	% Correct	MOV Error
Govan + Averaging	73.5%	10.80
Scoring averages	72.1%	11.18
Scoring + 3 pt % -- Without normalization	72.1%	11.18
Scoring + 3 pt % -- With normalization	72.1%	11.09

So as a matter of course I'll perform a normalization step as part of the prediction workflow. (In this case, it doesn't improve our best performance by much.)

It's also interesting to compare the coefficients in our linear regression. This is what we see if we look at the coefficients for the various scoring averages:

Datum	Coefficient
Home Team Scoring Average	5.886
Away Team's Opponent Scoring Average	-4.447
Away Team Scoring Average	-5.686
Home Team's Opponent Scoring Average	4.793

Naively, you might want to predict a team's score as exactly halfway between what the team usually scores (offense) and what the other team usually gives up (defense); but what this shows is that the best estimate actually weights offense slightly more -- 57% for the home team, 54% for the away team.

Statistical Prediction

2011-09-16T11:22:00.000-04:00

With this post, I'm going to start taking a look at predicting game outcomes based upon team-level statistical measures other than won-loss or MOV, i.e., measures like "team scoring average," "average number of offensive rebounds per game," etc.

There are a number of ways to slice & dice these statistics, but the most straightforward approach is to use season-to-date averages. So, when I'm trying to predict the Illinois-Purdue game on 2/15, I'll be looking at the statistics for those two teams averaged over all the games for that season before 2/15. And I also want to include average statistics for a team's opponents. So I want to know both Purdue's scoring average for all of its previous games, and also the scoring average of its opponents in those games. For every game, I'll typically have four values for a statistic: the home team's average, the home team's opponents' average, the away team's average, and the away team's opponents' average.

To begin with, let's look at how well we can predict games using the most obvious statistic: the scoring average. Using just the (four) scoring average statistics, and the usual methodology, here's our performance:

Predictor	% Correct	MOV Error
Govan + Averaging	73.5%	10.80
Scoring averages	72.1%	11.18

That's pretty encouraging. Just using the scoring averages delivers performance comparable with some of our better W-L and MOV-based predictors. The bad news is that this is still highly correlated with our best other predictors (around 96%), meaning that it probably can't be used in an ensemble to improve our overall predictive performance.

If we look at adding other statistics we find (as would be expected from the literature) that they offer little improvement. The best combination I could find (in order of importance) was (1) scoring, (2) 3 pt percentage, and (3) opponent's average offensive rebounding:

Predictor	% Correct	MOV Error
Govan + Averaging	73.5%	10.80
Scoring averages	72.1%	11.18
Scoring + 3 pt % + Opponent's off rebounding	72.2%	11.09

As you can see, the improvement was not huge. The inclusion of "average number of offensive rebounds by opponents" is interesting because it is not scoring-related. That statistic would seem to capture some aspect of a team's defensive performance -- a team that gives up a lot of offensive rebounds to its opponents is probably doing something wrong at the defensive end of the court. That suggests that we might want to think about a better measure of defensive performance -- for example, we might want to look at offensive rebounding percentage rather than just the raw total.

New Papers

2011-09-16T10:28:00.001-04:00

(All of the following papers have been added to the papers archive.)

[Gill 2008] "Assessing Methods for College Football Rankings," JQAS 2008

Summary: This paper purports to "...consider several mathematical methods for ranking college football teams based on point differential... [and] assess the predictive performance of these models using leave-one-out cross validation." The models considered are variants of least-squares fitting of rating values to point differential. Variants include different fitting methods (e.g., weighted least squares) and methods for limiting the impact of blowouts (e.g., cutting off the point differential at 14 or 28 points). Predictive performance is used to assess cutoff values for blowouts.

Comment: A disappointing paper for me; from the title and abstract I had hoped that this paper would analyze some set of football ranking approaches for their predictive value. Instead, the main conclusion of the paper seems to be that one can construct a rating system that emphasizes nearly any aspect of competition by selecting the right approach and tuning constants.

[Wigness 2010] "A New Iterative Method for Ranking College Football Teams," JQAS 2010
And see: WWR Rankings

Summary: This paper describes a method for ranking college football teams. The method uses (potentially) score, location (home or away) and time of season to create an initial value for each game, and then iteratively re-rates games until equilibrium is achieved. The method has a number of parameter/options, and the paper evaluates the performance of several combinations. Performance is measured by the % of correct predictions for bowl games. Over 9 seasons, the best combination predicts about 59% of the total bowl games correctly, and about 63% of the total BCS bowl games. In contrast, over the same span the BCS computer rankings have predicted about 57% of the BCS games correctly.

Comment: A fairly interesting paper, and apparently the work mostly done by an undergraduate. The approach is at least somewhat novel -- it involves creating a graph where the nodes are teams and the links are games between teams, and then summing all the simple paths originating from a team and going out "K" links. (Where K is a parameter, but K=4 was the best performing.) There's no intuitive (to me at least) meaning for doing this, but to some extent it captures the strength of opposition, the same way RPI uses OWP, OOWP, etc. I'd like to implement and test this system, but the naive implementation for calculating all the simple paths is likely going to be very slow, and if there's a clever matrix implementation it doesn't occur to me. I've put a question in to the authors asking about their implementation.

[Loeffelholz 2009] "Predicting NBA Games Using Neural Networks," JQAS

Summary: An ensemble of several different neural networks fed with team statistics was used to predict NBA games. Performance was assessed using "% Correct" and compared to consensus picks from five experts published in USA Today. The ensemble methods did not improve upon the best included baseline predictor. The best predictor (feed-forward NN) predicted 74% of the test games correctly (compared to 69% for the human experts).

Comment: There are number of interesting results in this paper. First, the authors looked at both (1) splitting team statistics based on home/away, and (2) using only the most recent 5 games, and in both cases found no value. This agrees with my own experiments with similar approaches. Second, the authors experimented with various combinations of statistics and had the best performance using only FG% and FT% for each team.

[Beckler 2009] "NBA Oracle," CMU Classwork
And see: Matthew Beckler's Home Page

Summary: This paper describes an effort to use various machine learning techniques to predict NBA game outcomes (as well as some related tasks). Inputs to the learning process were 62 features for each game -- most features were averages for the current season and the previous season of team statistics such as rebounding, shooting percentage, etc. The most effective technique was linear regression, which predicted about 70% of games correctly -- comparable to human experts. The most important statistics were team winning percentage in the previous season, and (in decreasing importance) defensive rebounds, points made by opposing team, number of blocks and assists made by opposing team.

Comment: A fairly straightforward attempt to predict NBA games based upon team statistics. Prediction accuracy is in line with similar work (although below Loeffelholz) -- around 70% seems to be fairly easy to achieve for NBA games. There's no attempt to predict MOV.

[Orendorff 2007] "First-Order Probabilistic Models for Predicting the Winners of Professional Basketball Games," JQAS 2007

Summary: This paper describes an effort to apply Bayesian Logic (BLOG) and Markov Logic Networks (MLN) to predicting NBA games. Inputs to the models are won-loss records. The MLN model performs best, predicting 76% of games correctly.

Comment: The methodology here is similar to my methodology -- the research uses a x-validation on the entire NBA season. However, there is one very important distinction. This research uses the entire season's data to predict the held-out games -- not just the season up to the time of the predicted game. This makes a huge difference in prediction performance, so take the authors' result of 76% accuracy with a grain of salt. It's likely that the accuracy using season-to-date data would be 15-20% lower.

[Trono 2010] "Rating/Rankings Systems, Post-Season Bowl Games, and 'The Spread'", JQAS 2010

Summary: This paper compares a number of simple systems for predicting college football bowl games.

Comment: This is a difficult paper to analyze. It is written in a very colloquial, unorganized manner and lacks a clear purpose. The systems analyzed are described in vague terms that make it difficult to understand the computational implementation, or even to attribute authorship of the systems. All that said, at least one system described has out-performed the Las Vegas line (by 1 game) over a 7 year period.

[West 2008] "A New Application of Linear Modeling in the Prediction of College Football Bowl Outcomes and the Development of Team Ratings," JQAS 2008

Summary: This paper uses linear regression to build a predictive model for college football bowl games. The inputs to the model are average statistical measures (e.g., "Offensive yardage accumulated per game"). The model predicted 19 of 32 bowl games correctly (59.4%).

Comment: This paper is of particular interest to me at the moment because I've also turned to looking at prediction using statistical team measures. This work seems to agree with my result that only a few measures (mostly related to scoring) have significance in the final model. Also of interest here is that West pre-conditions his statistical measures by expressing all of them in units of "standard deviations from the mean."

Prediction by Similarity

2011-08-31T23:01:00.000-04:00

Well, I've been all over the map with my blog postings lately and this one is no exception. I'm going to post today about what I call "prediction by similarity." I don't know what it's really called by the machine learning community -- back in the day when I did artificial intelligence we would have called it case-based reasoning and for all I know that may still be perfectly fine nomenclature.

The basic idea (as it applies to college basketball) is that if I want to know how Illinois @ Michigan State is going to turn out, I can look at historical examples of similar games and see how they turned out. Hopefully they'll be a good guide for predicting the Illinois @ Michigan State matchup.

The first challenge in this method is to find similar games. The easy solution would be if Illinois had already played (say) five games at Michigan State this season. Presumably those results would be an excellent guide to how the current game would turn out. Of course, it's never the case that we have five previous matchups to look at. Even when the two teams have played before, it's usually the other end of a home-and-home series, and because of the impact of home court advantage, it's hard to use even that. On the other hand, there are 15K games in my corpus -- and about twice that if I include the 2006-2008 seasons as additional historical examples. Out of 30,000 games, I ought to be able to find a few pretty similar to Illinois @ Michigan State.

About our only option for measuring the similarity of two games is to compare the season statistics (including things like ratings) of the four teams. If the two home teams have very similar statistics, and the two visiting teams have very similar statistics, we might have some hope that the past game can be used to predict the current game. So what statistics are important? I have no idea (and as far as I know, no one has looked at the question) but we can make some educated guesses. The relative strengths of the teams is probably important -- that is, we want to look at past games where the home team was about as strong as MSU and the visiting team as strong as Illinois. The absolute strengths might be important, too. If we use (say) TrueSkill as our strength rating, a matchup between an 850 team and an 800 team might be quite different than a matchup between a 450 team and a 400 team. (Or maybe not.) Pace of play might be important -- a fast 850 team playing a fast 800 team might have a different result than a fast 850 team playing a slow 800 team. In fact, I can probably make an argument for just about any statistic as being important.

As usual, I intend to compensate for my lack of knowledge with persistence and processing power. I'll try different combinations until I find one that works well or convince myself that further search won't be fruitful. Unfortunately, this search will take a lot more processing power. By the end of the 2011 season, I have to rate and sort 30K games for each predicted game. That goes a lot slower than (say) just updating RPI. It takes several hours to make one run.

Let's take a look at what sort of "similar" games we find with a similarity function that uses TrueSkill, average points per possession (PPP), average points per possession allowed (PPPA), and average number of possessions (POSS):

Similarity	Date	Home	TrueSkill	PPP	PPPA	Poss	Away	TrueSkill	PPP	PPPA	Poss	MOV
0.00	1/24/2011	Pittsburgh	104.7	1.21	0.92	66.4	Notre Dame	70.61	1.13	0.98	66.7	-5
0.18	3/25/2007	Florida	93.46	1.19	0.91	68.0	Oregon	71.66	1.12	0.98	66.8	8
0.20	3/21/2009	Connecticut	104.3	1.23	0.92	68.7	Texas A&M	68.39	1.07	0.98	66.9	26
0.22	1/29/2009	UCLA	91.3	1.16	0.93	65.8	California	70.50	1.12	0.97	67.8	15

Our current game (the one we're trying to predict) is in the first row: the 1/24 conference game between Pittsburgh and Notre Dame. The next three lines are the closest matches found. In the closest match, Oregon's statistics are amazingly close to Notre Dame's. Florida's aren't quite as close a match to Pittsburgh -- notably, Florida's strength (as measured by TrueSkill) is significantly less. In all of the similar games, the home team wins handily -- we would predict Pittsburgh by 16 points based upon these games. And that probably wouldn't be a bad prediction: Notre Dame's victory was considered a significant upset.

So how well does this work as a predictor? So far, not very well. About the best performance I've found is a MOV Error of 12.5 and a % Correct of about 68%. There are a couple of positives: I'm still working on the code, so I may find some bugs or improvements. And secondly, the predictions from this model are only correlated about 60% with (say) TrueSkill, which may make it useful as part of an ensemble model.

Correlation Between Predictors

2011-08-27T23:39:00.000-04:00

Danny Tarlow was kind enough to give me some comments on a paper I'm writing about the work reported in this blog, and one of his suggestions was to look at whether the predictors I've tested are picking up on the same signals. This is a significant question because if the predictors are picking up on different signals, then they can be combined into an ensemble predictor that will perform better than the individual predictors. (Dietterich 2000) showed that

"...a necessary and sufficient condition for an ensemble of classifiers to be more accurate than any of its individual members is if the classifiers are accurate and diverse."

A classifier is accurate if is better than random guessing. Two predictors are diverse if they make different errors. Intuitively, an ensemble will perform better than the base predictors if the errors in the base predictors are uncorrelated and tend to cancel each other out. Our predictors are all obviously accurate, but are they diverse?

To test this we can measure the correlation between the errors made by the different predictors. If they are uncorrelated, then it is likely that we can construct an ensemble with improved performance. I don't have the time and energy to test all combinations of the predictors I've implemented, but here are the correlations between the top two won-loss based predictors (Wilson, iRPI) and the top two MOV-based predictors (TrueSkill+MOV, Govan):

	Wilson	iRPI	TrueSkill + MOV
iRPI	0.99
Trueskill+MOV	0.93	0.93
Govan	0.95	0.95	0.98

Not unsurprisingly, the highest correlations are between the two won-loss predictors and the two MOV-based predictors. But all of the predictors are highly correlated. The least correlated (by a hair) are Wilson and TrueSkill+MOV. Putting those two predictors into a combined linear regression or an averaging ensemble results in performance worse that TrueSkill+MOV alone.

On the other hand, perhaps using the best predictors is the wrong course. Perhaps its more likely that the worst predictors are uncorrelated with the best predictors, and a combination of one of the worst with one of the best would be fruitful.

	Wilson	iRPI	TrueSkill + MOV	Govan
1-Bit	0.83	0.83	0.80	0.79
Winning Percentage	0.97	0.98	0.92	0.93

As this shows, even the 1-Bit predictor ("the home team wins by 4.5") is highly correlated with the better predictors, and using just the winning percentage shoots the correlation to 0.92+. Adding these predictors to an ensemble with the better predictors also results in worse performance.

Of course, it's always possible that some combination of predictors will improve performance. There's been some interesting work in this area -- see (Caruana 2004) in Papers. But for right now I don't have the infrastructure to search all the possible combinations.