Monday, March 25, 2013

Upset Analysis by Scott Turner

Posted by Danny Tarlow
This is a guest post by Scott Turner, who is a perennial Machine March Madness competitor, and who runs

There are 22 entries in the Machine Madness contest this year, so analyzing them is a much bigger task than in past years.  Nonetheless I dug through all the brackets and looked at all the first round upset predictions to see how well the machines did.

Correct Upset Predictions

Interestingly enough, every first round upset was picked by at least two of the predictors except for Harvard -- which no one picked -- and Florida Gulf Coast, which only "Larry's Upsetting Picks" predicted.   The only consensus upset pick was Minnesota over UCLA, which was predicted by exactly half of the predictors.  Iowa State also got broad support (40%) but none of the rest of the picks had more than 4 predictors in support.  Here's the full table of the upsets that occurred and who predicted them (stretch your window!):

Entry Sum Minnesota Iowa St Oregon Wichita St. Mississippi Temple California La Salle Fla GC Harvard
*Danny's Dangerous Picks 2 1 1
Andy's Astounding Bracket 2 1 1
Ask me about my T-Rex 3 1 1 1
Curtis Lehmann's Crazy Bracket 0
Dan Tran's Dazzling Bracket 3 1 1 1
Guess O'Bot 3000 0
K. V. Southwood's Fine Bracket 2 1 1
Larry's upsetting picks 4 1 1 1 1
LA's Machine Mad Pick 1 1
Leon's Super Legendary Bracket 1 1
Marginal Madness 1 1
Mark's LR bracket 4 1 1 1 1
MatrixFactorizer 3 1 1 1
natebrix's Neat Bracket 2 1 1
noodlebot 1 1
Predict the Madness 1 1
Ryan's Rank 1 Approximation 1 1
Scott Turner's Prediction Mach 2 1 1
ScottyJ's Grand Bracket 0
The Rosenthal Fit 2 1 1
TheSentinel 2 1 1
Tim J's Nets for Nets 4 1 1 1 1
   Ave Correct: 1.9 50% 41% 18% 18% 18% 14% 14% 9% 5% 0%

My conclusion here is that UCLA-Minnesota and Notre Dame-Iowa State were probably mis-seeded.

UCLA-Minnesota is an interesting case in human psychology.  Minnesota lost 11 of its last 16 games, finished 8th in its conference and lost in the first game of the conference tournament, while UCLA won 11 of its last 16, won the Pac-12 regular season conference title, and lost in the title game of the conference tournament.  It's no wonder UCLA got a 6 seed and Minnesota an 11.  But in fact, Minnesota was playing against much better competition through the conference games, and most of its losses came to ranked opponents and/or on the road.  Machines understand the concept of a "good loss" much better than people.  

The Notre Dame-Iowa State mis-seeding wasn't so egregious.  This probably should have been an 8-9 matchup instead of a 7-10, in which case a win by Iowa State would have hardly been surprising.

All of the rest of the games were probably true upsets.

Incorrect Upset Predictions

Most of the predictors also made a number of incorrect upset predictions.  Most predictors had one or two missed upsets, although six of the predictors made no missed upset predictions (primarily because they made mostly chalk predictions).  Here's the full table:

*Danny's Dangerous Picks 2 1 1
Andy's Astounding Bracket 0
Ask me about my T-Rex 2 1 1
Curtis Lehmann's Crazy Bracket 1 1
Dan Tran's Dazzling Bracket 2 1 1
Guess O'Bot 3000 3 1 1 1
K. V. Southwood's Fine Bracket 4 1 1 1 1
Larry's upsetting picks 7 1 1 1 1 1 1 1
LA's Machine Mad Pick 0
Leon's Super Legendary Bracket 8 1 1 1 1 1 1 1 1
Marginal Madness 0
Mark's LR bracket 4 1 1 1 1
MatrixFactorizer 2 1 1
natebrix's Neat Bracket 2 1 1
noodlebot 0
Predict the Madness 1 1
Ryan's Rank 1 Approximation 1 1
Scott Turner's Prediction Mach 2 1 1
ScottyJ's Grand Bracket 0
The Rosenthal Fit 0
TheSentinel 2 1 1
Tim J's Nets for Nets 4 1 1 1 1
Average Missed: 2.1 45% 36% 23% 23% 18% 14% 14% 9% 9% 5% 5% 5% 5% 5%

As a general rule, the predictors that made the most correct upset picks also made the most incorrect upset picks.  Notably, "Larry's Upsetting Picks" made the incredible call of the FGCU upset (and also called the second round upset) but also made seven incorrect upset picks.

There was almost a consensus (45%) on Colorado over Illinois.  That's an interesting contrast with the Minnesota pick -- Illinois should have benefited in most of the predictors from a tough B1G conference schedule, but many of the predictors thought Illinois was still vulnerable.  Illinois had a 16 point halftime lead in this game, but let it slip away and need some late game heroics to win, so this was certainly a reasonable prediction.

St. Mary's over Memphis was another popular pick.  Memphis won by 2 when a last-second shot by St. Mary's missed, so this also seemed like a reasonable upset pick.


Upset Profits

An important question is whether any of the predictors profited from their upset predictions -- that is, whether the points they gained from correct upset predictions were more than the points they lost from missed upsets.  In general, this is complex to calculate because we have to look at how the predictions affect the later rounds of the tournament.  But it's easy enough to look at just the first round scoring.  Here's the table:

Andy's Astounding Bracket202
The Rosenthal Fit 202
Ask me about my T-Rex321
Dan Tran's Dazzling Bracket321
LA's Machine Mad Pick101
Marginal Madness101
*Danny's Dangerous Picks220
Mark's LR bracket440
natebrix's Neat Bracket220
Predict the Madness110
Ryan's Rank 1 Approximation110
Scott Turner's Prediction Mach220
ScottyJ's Grand Bracket000
Tim J's Nets for Nets440
Curtis Lehmann's Crazy Bracket01-1
K. V. Southwood's Fine Bracket24-2
Guess O'Bot 300003-3
Larry's upsetting picks47-3
Leon's Super Legendary Bracket18-7

We see that a couple of the predictors ("The Rosenthal Fit" and "Andy's Astounding Bracket") came out two points positive, fifteen of the predictors gained one or zero points, and five of the predictors lost points.  Interestingly, both "The Rosenthal Fit" and "Andy's Astounding Bracket" made only two upset predictions and got both of them right -- and there was no overlap in their predictions.  Furthermore, neither of them predicted the "easiest" upset of Minnesota over UCLA.



None of the predictors performed very well at picking upsets, and there wasn't wide agreement on the upset picks.  The consensus would have selected only the Minnesota-UCLA upset and been +1 in scoring, but no individual predictor did that.  Most of the predictors did not hurt themselves with their upset picks (at least looking at only the first round), but none really saw significant benefit.  Given the potentially large downside of missing upset predictions, in future contests it wouldn't be an unreasonable strategy to force your predictor to make all chalk selections in the first round. 

1 comment:

Unknown said...

Scott, this is excellent analysis, and a great complement to the upset analysis you did last year.

I intuit that if one wanted to win money in NCAA bracket pools, the key would be to pick a limited number of upsets (and you present the most reasonable strategy that I have seen here), but vary them across a large number of entries in a large pool. Of course, the number of upsets in a given year can vary a lot, and someone else could do very well just via dumb luck, but it seems like it would be the best strategy (even if it still had a negative expected pay-off when played repeatedly)