College football fans like us hate the BCS.

Unless you work for ESPN and the BCS contributes to your paycheck, the idea of allowing only two teams to play for the national championship is criminal.

And if you’re reading this numbers based blog, you probably know about the problems with the computer polls used in the BCS rankings.

First, only one of the six ranking systems gives enough details so that others can reproduce the results. The other five black boxes are shrouded in mystery.

Second, the BCS forbids the computers from using margin of victory in their calculations. It does not matter that a 33 point loss says something much different about a team than a 1 point loss. In the name of sportsmanship, the BCS will not give teams the incentive to run up the score.

Last, you may have even heard that Richard Billingsley, the man behind one computer poll, is not a mathematician. As he admitted to the authors of Death to the BCS, “I don’t even have a degree. I have a high school education. I never had calculus. I don’t even remember much about algebra.”

But it gets worse.

Why Strength of Schedule and Margin of Victory Matter

I wasn’t looking for a flaw in a BCS computer poll.

I was thinking about strength of schedule and margin of victory. In college football debates, most people agree that a ranking system should account for these factors. The intuition is obvious. Northern Illinois does not play the quality schedule that Alabama does. Oregon’s 24 point win over a solid Oregon State team says something much different about the Ducks than a 1 point win. However, no one has provided any quantitative evidence to support accounting for schedule strength and victory margin in rankings.

Bowl games at neutral sites provide a simple quantitative test for ranking systems: how often does the higher ranked team win each game? For a system that incorporates neither schedule strength and victory margin, rank teams by winning percentage. For a system that accounts for strength of schedule but not margin of victory, rank teams with the Colley Matrix of the BCS. Last, my rankings account for both.

I spent some time digging into the details of the Colley Matrix.

Colley Matrix does not consider the results of each game

It was easy to get mesmerized by the beautiful mathematics behind Colley’s method. His paper discusses Laplace’s die problem, a symmetric positive definite matrix and solving a linear system of equations. I spent the weekend telling my wife that if college football games had only winners and losers, this would be a dandy little ranking algorithm.

Then thinking back on the equations, it hit me.

The method does not care who a team loses to in ranking them. It considers the win loss record of each team and the number of games played between each pair of teams. However, the specifics of who won each game are not an input to Colley’s method.

Omitting specific game results in a ranking system is like disabling the guiding system on a missile. The technology will do its job, but it will not be that accurate.

You can check this yourself by reading the descriptions of equation 18 and 19 in Colley’s paper. It’s possible to solve for the rankings (equation 17) knowing only each team’s record and how many games each pair of teams played.

As a mathematician, I find this omission appalling. To see why, take Alabama in 2012 as an example. The Crimson Tide lost to Texas A&M, a respectable loss to another top 10 team. But the Colley Matrix does not account for this. Suppose Alabama beat Texas A&M but lost to a bad Florida Atlantic team. Since a top team almost never loses to a bad team, this bad loss should lower Alabama’s rank. It doesn’t.

You can check this with your own example. Wesley Colley has set up a page on which you can add and remove games and recalculate the rankings.

When I first discovered this omission in 2012, Stewart Mandel of Sports Illustrated suggested looking into whether this flawed computer poll was helping Kent State. The Golden Flashes were 11-1 heading in the MAC championship game and ranked 17th in the BCS. If they moved up to 16th or better, they would earn a BCS bowl bid.

Sure enough, the Colley Matrix had Kent State ranked 15th, the highest rank in any computer poll. It did not consider that their lone loss came at Kentucky, a 2-10 team that won zero SEC games that year. This flawed computer poll played a small role in placing Kent State 17th overall in the BCS rankings. I wrote about this on SI.com.

“Sam Feng’s article is a perfect example of anti-science”

In response to my article, I got this tweet the next day. Amidst a flurry of four letter words, a blogger blasted the mathematics behind my analysis of Colley’s method.

I disapprove of Feng because I don’t know what the f*$% he was doing, and I don’t think he knows what the f*$% he was doing either.

I guess that can happen when your writing jumps from a small blog to SI.com. At least he could get my name right.

I promptly replied to his post, and a conversation ensued about the details of the mathematics. In the end, the blogger verified my main conclusion that Colley omits the results of each game. The post started with a rant about “anti-science”. It progressed with a dense mathematical discussion in the comments. It ended with this in the last comment.

I’d rather have Ed’s rankings making the decisions than Colley’s or a roomful of NCAA bureaucrats.

The blogger still had a problem with the example I used in the article, a criticism with some merit. The example is equivalent to the Alabama scenario above. In exchanging the loss to Texas A&M for a loss to Florida Atlantic, the records of these two opponents change. Since the Colley Matrix does consider each team’s record, the rankings do change. However, Alabama’s rank does not change. This makes no sense when a team loses to a cupcake.

To be precise, one can show the rankings remain exactly the same under certain changes of wins and losses. For an example in 2012,

• Stanford beat Oregon
• Oregon beat Washington
• Washington beat Stanford

Suppose we change the result in each game.

• Oregon beat Stanford
• Washington beat Oregon
• Stanford beat Washington

Since all teams have the same record, the rankings stay exactly the same. Oregon would remain 7th despite losing to an average Washington team. It just doesn’t make sense.

However, for the sake of simplicity, I went with the example in which one team traded a loss for a win. At the end of the day, Colley’s method disregards massive amounts of useful information.

Northern Illinois busts the BCS.

Before the MAC championship game in 2012, Kent State threatened to bust the BCS with their ranking of 17th. However, their opponent, Northern Illinois, wasn’t too far behind at 21st.

After winning the championship game, Northern Illinois jumped to 15th in the final rankings to earn a BCS bowl game against Florida State. In the computers, the Huskies made massive jumps in the polls of Richard Billingsley and Peter Wolfe.

Billingsley does not describe this ranking method, so no one knows why he bumped Northern Illinois from 19th to 12th.

However, Peter Wolfe describes his method and even offers a few references for his Bradley-Terry model. This nice academic article by Keener describes the model in some detail, making it possible to reproduce the results. After carefully reading this paper, I didn’t find any problems with the ranking method. It does matter that Kent State lost to Kentucky. The math seems to favor teams that play an extra game, which most likely helped Northern Illinois jump from 23rd to 12th after their win over Kent State.

A Playoff on the Horizon

I didn’t think anything could make me hate the BCS more. I was wrong.

At least the current computer polls will be banished when a four team playoff arrives in 2014. A selection committee similar to the group that determines the field for the NCAA men’s basketball tournament will pick the four teams. I only hope their debates will be aided by better algorithms for ranking teams.

Thanks for reading.