Rational Pastime Blog

Rating Systems Challenge: Equations are Red, Experts are Blue

2013-04-08T13:35:00.000-04:00

Georgia Dome
Site of the 2013 Final Four
Source: Wikimedia Commons

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

While tonight's contest will feature a battle between two squads of college students who represent the pinnacle of their sport, the winner will also determine whether the best prognosticators of NCAA Tournament Basketball were a council of experts—the survey samples for the ESPN and AP Preseason Polls—or advanced models developed by Nate Silver and ESPN.

In The Signal and the Noise, author Nate Silver makes a convincing argument that experts are typically bad prognosticators and groups of experts are seldom better. Instead, we should look at rigorous, testable models based on objective, verifiable data. However, even advanced models can fail to predict outcomes in low-information systems where minor variations can cascade into major disruptions.

Weather forecasting is one attempt to simplify such a system. Forecasting tournament play in sport is another.

To borrow more from Mr. Silver, in both meteorology and sports prognostication, simple historical baselines do not provide us with much useful information. Sure, there may be a 20% chance of rain* on any given April 10th in Atlanta, GA, but that doesn't tell us whether Atlantans should take their umbrella when they leave for work tomorrow. Likewise, while it may be true that the ultimate champion in ~58% of NCAA Tournaments has been a one-seed,** that tells us very little about whether 1 Louisville is going to beat 4 Michigan tonight in the Georgia Dome.

*not a researched estimate
**since the NCAA Tournament expanded to sixty-four teams, according to CBS Sports

Instead, we try to build models that rely on first principles (adiabatic heat exchange, Log5) and spit out probabilistic assessments. These models are most likely to fail when we have imperfect information about outcomes based on multistage causality. This is why, in a given year, a council of experts in November might outwit the most advanced models—one designed by Mr. Silver himself—with the most up-to-date information.

A win for Louisville is a win for math: the FiveThirtyEight and ESPN Computer brackets may be trailing, but a Cardinals title would put them over the top. Conversely, a win for Michigan is a win for the experts—or at least November's experts. The preseason polls expected Indiana to win it all so, while they may lead now, a Michigan win is the only way they can hold onto that lead.

Either way, do not let tonight's outcomes distract from the fact that the postseason polls underperformed chalk yet again. Maybe the experts—be they journalists or coaches—aren't that smart after all.

Follow us on Facebook and Twitter—or stay tuned—to find out which system won Rational Pastime's 2013 Rating Systems Challenge.

Rating Systems Challenge: Final Four Preview

2013-04-06T04:12:00.001-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

The 2013 Rating Systems Challenge heads into the Final Four with only one top seed and two original picks remaining: 1 Louisville, which eighteen of twenty systems expected to make it this far, and 4 Michigan, which only the preseason polls expected to survive. Joining them are 4 Syracuse and, 2013's Cinderella school, 9 Wichita State. It should be pretty obvious by now which school the systems expect to win it all.

The implications are now clear for the Rating Systems Challenge since the systems that picked Michigan to reach the Final Four do not expect the Wolverines to advance further, and since those are the systems that are currently winning. If the Cardinals fall short of winning the National Championship, the ESPN Preseason Poll will have been 2013's best prognosticator. If Louisville does win it all, that honor will fall to the FiveThirtyEight and the ESPN Computer systems.

Regardless of who wins, one interesting fact is that the systems as a whole did a poorer job of predicting the tournament this year than last year. Look at this graphic below depicting the number of correct picks each system made in 2012 (out of a possible 63):

Last year's most successful system, Survival, picked 47/63. The median success rate was 41/63. This year, the most correct picks any system will make is as many as last season's median performance: 41. The current leading system, ESPN Preseason, has correctly picked 40/60 thus far, and has only one possible correct pick remaining: Louisville in the semifinals. The systems that might surpass it in the ESPN scoring system made 38 correct picks so far. With only two games remaining they have no chance to catch up.

If Wichita State upsets Louisville, no system will perform as well as last year's median system in terms of correct picks.

The numbers look even worse using ESPN's weighted scoring system. This year, the best possible score is 1160 (the final score for ESPN Computer and FiveThirtyEight if Louisville wins it all). Such a score would have ranked 14th in last year's Rating Systems Challenge and beat only ~75% of brackets nationwide. However, with such a volatile tournament this season, it's possible that 1160 will be enough to finish among the upper echelon of ESPN entrants.

Follow us on Facebook and Twitter—or tune in tomorrow—to follow each system's performance in the NCAA Tournament semifinals.

Rating Systems Challenge: Mascot Pun Does Basketball Thing to Math

2013-03-31T03:11:00.000-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

Yes, that's a lazy title, but it is about as much as my fatigued, basketball-ingesting mind can regurgitate this morning.

It would have made sense to come up with some sort of title illustrating how the upset victories of 9 Wichita State and 4 Syracuse threw the rating systems for a loop, but I'm not sure anything makes sense in sports at this point. Besides, my mind is still far too blown to think through anything of the sort. Just imagine a post title that incorporates the word "shocking" and, maybe, "Otto."

I'm not sure which damage is more irreparable: that to the pleasure center of my brain or to the nation's brackets. Probably the latter, and we have evidence. RP Blog is tracking twenty rating systems, yielding forty picks for East and West representatives in the Final Four. These forty picks represented five different schools.

Today, halfway through the Regional Finals, each and every one of these five schools has been eliminated. Inspect the wreckage below:

The last school standing from this half of the bracket was 2 Ohio State, which The Wichita State University took care of last night after deflecting a late Buckeye charge. What sort of effect did this have on the standings in the Rating Systems Challenge? Check the leaderboard below the jump.

This chart and leaderboard might look familiar. It should, because yesterday's games had essentially no effect whatsoever.

Not one of the systems picked either 3 Marquette or Syracuse to advance, so not one of the rating systems picked up any points yesterday. Even in terms of remaining points, yesterday's games had little effect. Four systems—the ESPN National Bracket, Sonny Moore and the preseason polls—did project Ohio State to reach the Final Four, but none had the Buckeyes advancing any further.

Moreover, it seems the rest of the nation had as much faith in the Orange and the Shockers as our rating systems did. After yesterday's action, the ESPN Preseason Poll is still beating 98% of submissions. To make a long story short (too late) the standings remain where they were two nights before: with the preseason polls doing quite well but in a weak position going forward.

This will not be the case again tonight.

Both Regional Finals match-ups have major implications for several ratings systems. The RPI systems pick 2 Duke to beat 1 Louisville on the way to a National Championship. Two systems—the preseason polls—expect 4 Michigan to beat 3 Florida; nine systems send the Gators to Atlanta instead. A Blue Devil upset would yield the biggest impact, cutting down the eighteen systems that picked overall #1 Louisville to win it all.

Follow us on Facebook and Twitter—or tune in tomorrow—to see how each system performs in the final day of Regional play.

Rating Systems Challenge: Preseason Polls Dominating? Don't Be So A-maized.

2013-03-30T02:27:00.000-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

Was that awful pun really necessary? Yes, yes it was.

Okay, no it wasn't, but I hope it drew your attention to something that seems amazing but is actually rather common. The preseason polls from AP and ESPN tend to do rather well in forecasting NCAA Tournament results. Last year, the AP Preseason Poll finished fourth overall in the Rating Systems challenge, six spots ahead of the AP Postseason Poll.

On average, the preseason polls are actually slightly more accurate than postseason polls at predicting postseason success. Perhaps the preseason polls are better at measuring the baseline talent level. Perhaps the postseason polls are poisoned by recency bias. Either way, this is the pattern, and it's a pattern that is playing out again this season. Largely thanks to 4 Michigan's (get it? maize?) upset of 1 Kansas, the ESPN and AP Preseason Polls rank #1 and #3, respectively.

Will their lead hold up? Maybe, but most likely not. Read why after the jump.

Like I said, the preseason polls are performing well so far—the ESPN Preseason Poll is doing so well that it's beating 99.6% of all brackets in the ESPN Tournament Challenge. However, since they both picked Indiana to win it all, their opportunities to earn more points are limited. For these systems to have a shot at winning the 2013 Rating Systems Challenge, 3 Florida will have to miss the Final Four. Otherwise, Sports Reference, Survival and Pomeroy will take the lead. Even if Florida loses, a 1 Louisville title means that the preseason polls will probably lose to the FiveThirtyEight.

You may notice that, for the first time in this tournament, a significant number of the systems have outperformed the Chalk bracket. With three top seeds eliminated, expect this trend to continue through the NCAA Championship.

With three 1-seeds eliminated, there's a lot of room for lower seeds to advance to the Final Four. Four systems picked 4 Ohio State to advance out of the West—the only lower seed that any of the systems expect to advance tomorrow. 9 Wichita State's Cinderella run makes this even more likely.

In terms of Sunday's games, there is at least one system that picked a lower seed to advance in both contests. Two systems pick 2 Duke to overcome top-seeded Louisville. Two systems (not the same ones) put Michigan in the Final Four, and fully nine systems expect the Gators to reach Atlanta.

With five projected Final Four picks still alive, each and every one of these games has major Rating Systems Challenge implications.* Also, I suppose they impact the Tournament in some way. Follow us on Twitter @RationalPastime or tune in tomorrow to see how well each system performed in the Regional Finals.

*Okay, not so much Syracuse-Marquette

Rating Systems Challenge: The Bitter Taste of Bracket l'Orange

2013-03-29T01:29:00.001-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

The Shockers of 9 Wichita State may have been the lowest seed to advance last night, but 4 Syracuse's upset of 1 Indiana was by far the most disruptive. All twenty systems picked Indiana to advance to the Elite Eight; nineteen picked them to reach the Final Four. Nine had the Hoosiers in the Finals, and three systems—the preseason polls and the ESPN Decision Tree—picked Indiana to win it all.

The only system to pick a different team to win the East Regional was NCAA RPI. That system picked 2 Miami, which fell tonight in another upset to 3 Marquette. In short, all of the East Regional champion picks by the twenty rating systems have been eliminated.

Marquette's win was a major boon for the ESPN Preseason bracket, the only bracket that put Marquette in the Elite Eight. ESPN's Preseason Poll took the lead tonight after Miami's loss. Think about that for a second: of all the systems, the most accurate one to-date was constructed by a survey of experts before the season even began. However, this bracket is in trouble the rest of the way, as it is among the systems that picked Indiana to win it all.

The best-positioned systems at the moment, in second and third place with the most possible points remaining, are the Sports Reference efficiency system and the HSAC Survival model. Going forward, look for Pomeroy's system to move up a few spots as well. Finally, if 3 Florida stumbles, we might expect the ESPN Computer and FiveThirtyEight simulations to make a big move.

The worst-positioned systems are, unsurprisingly, our last-place systems: the two RPI formulas and Nolan Power Index. All three systems performed so poorly that they gained zero points in last night's competition. Of these three, only NCAA RPI has a significant number of points remaining.

What sorts of upsets can we expect today? Well, Florida is fine position to reach the Elite Eight despite not being the top original seed in their region. Fifteen systems picked Florida to advance, so a third straight upset by the Florida Gulf Coast Cinderellas would put quite a few brackets on ice.

More likely (and more potentially disruptive) are upsets by 3 MSU, 4 Michigan and perhaps 12 Oregon. A win by Oregon would knock the consensus favorite, 1 Louisville, out of the NCAA Tournament. A win by Michigan would eliminate 1 Kansas and further clear Florida's path to the Final Four.

Tune in tomorrow to see how our systems fared in the second day of Regional play.

Rating Systems Challenge: Regional Preview

2013-03-28T05:47:00.001-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

The Men's NCAA Tournament returns tonight at 7PM EST, bringing with it the potential for some exciting upsets on the way to the Elite Eight. Ten schools that were not the top seed in their region when the tournament began have a chance to advance. Two of those original top seeds—1 Gonzaga and 2 Georgetown—have been eliminated, clearing the way for Cinderellas such as 9 Wichita State, 13 La Salle and 15 Florida Gulf Coast.

In fact, with Wichita State and La Salle facing off against one-another, we are guaranteed to see a 9-seed or lower advance to the Elite Eight for the first time since, well, 11-seed VCU reached the Final Four two years ago.

Unfortunately—but unsurprisingly—none of the rating systems under review picked any of those underdogs to advance that far. The gutsiest picks of these systems were 4 Michigan (picked by the two preseason polls and Sports Reference), 3 Marquette (picked by the ESPN Preseason Poll) and 3 Florida.*

*Florida is such an "underdog" that fifteen systems originally picked them over Georgetown. Of course, Florida won't play Georgetown; they'll be attempting to teach FGCU what midnight feels like, a likely win nobody would call an upset.

Most of these upsets would play a role only as spoilers. If Florida advances, just about every system wins. If any of the major upsets occurs, few of the systems will benefit directly (some may benefit indirectly by eliminating their competitors' Final Four picks). However, a win by Michigan could do some damage, elevating the preseason polls and giving Sports Reference a significant lead heading into the Regional Finals.

It may in fact be more informative to focus on what the systems have lost rather than speculate on what they may gain. Every team in the Challenge has lost at least one Elite Eight pick, some more than one. Take a look at how this might affect the various systems:

Gonzaga's loss, while shocking, was not particularly disastrous. Since nearly every system made that mistake, the relative damage of that upset was small—and the systems that benefited (I'm looking at you, Nolan Power Index) made too many other risky choices to gain from this one. The systems that picked Georgetown (I'm looking at you, Nolan Power Index) are in a much weaker position, as are the systems that took chances on 3 New Mexico, 5 Wisconsin and 6 Butler.

Lunardi RPI is on the shakiest ground of all. Three of their eight picks to reach the regional finals—1 Gonzaga, Georgetown and 3 New Mexico—are all heading back to campus empty-handed. Seven systems are missing two picks, while the rest are missing only one. These systems are going to need big wins from Duke or Indiana to rectify their errors and separate themselves from the pack (a win by Louisville would win them points, but as the most favored team it would win points for too many other systems as well).

This illustrates the problem of making picks in March. You can't be too conservative, but when you take risks you have to take the right ones. We've seen a number of exciting upsets in the 2013 NCAA Tournament, but very few that the systems have been able to capitalize on. The best approach is to pick plausible upsets that are not too popular. From this standpoint, the systems that picked Michigan have the best shot going forward. Perhaps it's not too surprising that two of them—Sports Reference and the ESPN Preseason Poll—are in the lead.

Will they stay there? Following along with the 2013 Rational Pastime Rating Systems Challenge during tonight's action, then tune in tomorrow to see how these systems did.

Rating Systems Challenge: Eagles Soar, Explorers Explore, Brackets Sore

2013-03-25T12:49:00.000-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

While last night was one of the more exciting nights in recent history for the NCAA Tournament, it was not a particularly interesting night for the Rating Systems Challenge. Our leaderboard remains essentially unchanged.

How could it be otherwise? None of the systems expected a 13 LaSalle / 12 Ole Miss match-up in the Round of 32, let alone that the Explorers would advance. And they're joining Florida Gulf Coast as the first 15-seed to reach the Sweet Sixteen?! Please. If anyone picked this--human or computer--you're not smart, just lucky (or a loyal alum).

Today's leader is the same as yesterday's leader (Sports Reference efficiency rating) and today's laggard is the same as yesterday's laggard (Nolan Power Index). With all the surprises, the disparity between best and worst is only five picks out of forty-eight. Read more about those surprises after the jump.

Heading into the Round of 32, there were nineteen teams with a chance to advance to the Sweet Sixteen despite not being the top seed in their Super Region. Of these, only five advanced--three of which were facing a 7-seed or lower. Two of these schools were playing lower seeds than themselves.

Out of this field of potential Cinderellas, there was only one school that the systems initially and successfully picked to reach the Sweet Sixteen: 6 Arizona, who took care of 14 Harvard on Saturday. Two systems (the preseason polls) had San Diego State in the field of sixteen. And they would have gotten away with it too if it weren't for a squad of high-flying underdogs and their chicken dance.

How do the remaining systems stand to fare for the rest of the Tournament? What surprises do they expect in the Sweet Sixteen and Elite Eight? Tune in Wednesday to find out.

Rating Systems Challenge: Shocking Upset Busts Brackets

2013-03-24T01:50:00.000-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

Another exciting night of college basketball has shaken up the race for top dog in the 2013 Rational Pastime Rating Systems Challenge.

After the third full day of the NCAA Tournament we see the Sports Reference efficiency system take the lead, edging out Chalk and other brackets by the smallest of margins. Five other systems are tied with Chalk in total points but have more possible points remaining.

Interestingly, sitting in second place behind one of the more advanced rating systems is one of the least sophisticated entrants: the ESPN Preseason Poll. Both the Sports Reference and ESPN Preseason systems benefited from correctly picking 6 Arizona to reach the Sweet Sixteen. Bringing up the rear yet again is our perennial laggard: the Nolan Power Index. NPI unfortunately projected 5 Virginia Commonwealth and 6 Memphis to advance; Big Ten powerhouses from the Wolverine State easily felled them both.

Of course, the stories of the day were the two big upsets: 12 Oregon's tidy win over 4 Saint Louis and 9 Wichita State's "shocking" (sorry) victory over the postseason poll favorite, top-seeded Gonzaga. Keep reading after the jump to discover what effect this will have on the various systems' performance in the remainder of the NCAA Tournament.

As you can see on the leaderboard above, the postseason polls and the Nolan Power Index lost their NCAA Champion pick when the Wichita State Shockers upended Gonzaga. All three find themsevles at a significant disadvantage for the rest of the tournament. Gonzaga's fall hobbled NPI especially, which already lags 92% of brackets submitted at ESPN.com and has correctly picked only 55% of match-ups so far.

The damage is not confined to NPI and the postseason polls; a full fourteen rating systems placed Gonzaga in the Final Four. The LRMC Bayesian system expected Gonzaga to reach the finals (losing to 3 Florida). With the Bulldogs' departure, there are only four systems with an intact Final Four: the ESPN National Bracket, the two preseason polls, and the Sonny Moore bracket.

Today's games yield more bracket busting potential. A full forty-one Final Four picks are on the line tomorrow in the form of one-seeds Kansas and Indiana, two-seeds Duke, Miami and Ohio State, and three-seed Florida. The only systems that can directly benefit from any of these upsets, however, are the preseason polls who ranked 7 San Diego State high enough to send them to the Sweet Sixteen.

Ten teams who were not their Super Region's top seed will vie to advance. With Kansas State and Georgetown out of the way, two are guaranteed to be underdogs: SDSU and 12 Ole Miss are the highest remaining seeds in their respective Super Regions.

Best of all, tomorrow we get to see if the clock strikes midnight for our two Cinderellas: 13 LaSalle and 15 Florida Gulf Coast. Stay tuned to see how the various rating systems handle these and other developments as March Madness continues.

Rating Systems Challenge: Eagle-eyed Analytics

2013-03-23T03:28:00.000-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

A hearty day of upsets did some serious damage to the nation's brackets while shaking up the standings of the 2013 Rational Pastime Rating Systems Challenge.

While the systems performed poorly Thursday—missing twenty-five of twenty-eight upsets between them—Friday was a different story. Ten systems correctly picked 11 Minnesota's upset of 6 UCLA. Lunardi RPI correctly picked 9 Temple over 8 NC State. LRMC Bayesian picked 10 Iowa State over 7 Notre Dame.

In the best pick of the night, the Nolan Power Index correctly chose 12 Ole Miss over 5 Wisconsin.

On the flip side, five systems incorrectly favored 10 Colorado over 7 Illinois. Not one system divined 13 LaSalle's victory over 4 Kansas State or 15 Florida Gulf Coast's historic defeat of 2 Georgetown—but can you blame them? Frankly, any system that picked FCGU over the Hoyas is more likely broken than clever.

For the Round of 64, seventeen systems picked sixteen of forty-six (35%) of upsets between them. They missed four, each a 12-5 upset or better. Keep reading to see what Saturday and Sunday have in store, and which systems are outperforming overall.

Friday was a better day for the advanced analytical systems. After the Round of 64, the Survival, FiveThirtyEight and advanced ESPN systems share the top spot. They tied Pure Chalk in terms of total correct picks, but Chalk lags in terms of possible points remaining. Same applies to the two postseason polls, and NCAA RPI.

Lunardi RPI stands in the unique spot of tying for first in points but keeping last place all to themselves in possible points remaining. The Round's two biggest upsets—14 Harvard over 3 UNM and FGCU over Georgetown—knocked LRPI Final Four picks out of the tournament; Joe Lunardi's RPI had Georgetown in the final game.

Thanks to the phenomenal upsets in the first full round, there are nineteen teams with a chance to advance to the Sweet Sixteen that are not the top seed in their Super Regional. Of these nineteen, the various systems picked five.

Three lower-seeded teams have a shot to advance to the Sweet Sixteen against an underdog. The departures of Georgetown, New Mexico and Kansas State made the path that much easier for Arizona, San Diego State and Ole Miss—but don't discount a Cinderella run by the LaSalle Explorers, Harvard Crimson or Florida Gulf Coast Eagles.

Hey, it could happen!

Rating Systems Challenge: Veritas in Ludo

2013-03-22T13:34:00.001-04:00

This post is one in a series investigating how the various college basketball rating systems predict the men's NCAA Tournament. Click here to read the rest.

14 Harvard shocked the world last night, eliminating New Mexico and probably more than a few brackets from competition.

Other than the two systems (Lunardi RPI, NCAA RPI) that put New Mexico in the Final Four, the Crimson victory had very little affect on the Rating Systems Challenge--none of the systems predicted this upset.

In fact, the several systems were more wrong than right on the topic of Thursday upsets. The three most favored upsets--Mizzou/CO State, St. Mary's/Memphis, Bucknell/Butler--failed to materialize.The systems also missed California's 12-5 upset over UNLV. Overall, the combined systems were a paltry 3/28 if you count only the upsets they predicted, 3/30 if you include the upsets they ignored altogether.

It wasn't a complete wash. Two systems (ESPN Computer, ESPN Decision Tree) correctly identified Oregon's 12-5 upset of Oklahoma State, while one system (NCAA RPI) picked 9 Wichita State over 8 Pittsburgh.

With so many swings and misses (air balls? bricks?), it is no surprise that the standings after one day's play are exceptionally chalky. Read past the jump for the leaderboard and additional analysis.

NCAA RPI benefited especially from picking Wichita State over Pittsburgh and for avoiding trendy upset picks such as Mizzou and 11 St. Mary's. RPI is the only bracket outperforming chalk, while the top of the leaderboard is occupied by exceptionally chalky systems such as the AP and ESPN postseason polls and the ESPN National Bracket.

Despite its early lead, NCAA RPI is already at a disadvantage due to its New Mexico Final Four pick. Traditional laggards such as Sonny Moore and Nolan Power trailed as usual. Perhaps most surprisingly, Las Vegas had an especially bad day at the races, correctly picking only 63% of the contests. Don't worry, I'm sure they found a way to make money anyway.

Today features five upsets that at least one system picked, including 12 Ole Miss, trailing by 3 at the half as this post goes to press, which the Nolan Power Index picked to overcome 5 Wisconsin. If 11 Minnesota or 10 Colorado completes their upsets, expect major changes on this leaderboard.