The Big 12 hired an analytics firm to advise them on the best way of making the College Football Playoff. After crunching the numbers, the firm suggested expanding to 12 teams, playing 8 instead of 9 conference games and having a championship game.
This idea of holding a championship game interests me the most. Navigate, the research firm, says this change alone would increase the Big 12’s playoff odds by 5 percent. (All three suggestions add up to an 10-15 percent increase.)
However, it doesn’t make sense that adding a championship game will increase a conference’s chance to make the playoffs.
Think back to 2015. Oklahoma finishes their regular season at 11-1 and 3rd in the committee rankings. Since the Big 12 didn’t have a championship game, the Sooners sat at home during the last week of the season. They couldn’t lose and fall in the rankings.
Here, we’ll use simulations to determine the effect of a championship game on a conference’s playoff chances. The results impact not only Big 12 teams but also Notre Dame.
The numbers suggest that the committee must place extremely high value on winning a championship game, not just being a conference champion, to see any kind of increase in playoff odds. Let me explain.
Analytics on conference championship games
In 2014, Bleacher Report asked me to calculate the chance that each team would make the College Football Playoff. I wrote about these simulation results in their weekly playoff odds report for the past two seasons.
I’m happy with the accuracy of these projections. The numbers didn’t like undefeated Utah in early in 2015, and they subsequently lost 3 games. I also predicted a difficult path for top ranked Mississippi State in 2014, a team that failed to make the playoff. You can make your own judgement based on the archive of predictions.
To look at conference championship games, I use the same methods to simulate the past three seasons retrospectively. Let’s look at the two key components to this calculation.
- College football team rankings. For each season, I use season ending rankings to calculate a win probability for each game. This uses the best estimate of team strength to retrospectively look at the season.
- Season long simulations. The simulation marches through each week of the season and picks winners at random according to the assigned win probability.
Each week, teams that lose drop in the committee rankings. Teams that win usually hold their place in the rankings but occasionally jump over teams ahead of them.
The simulation also determines the winners of divisions to hold conference championship games the last week of the season. The simulation accounts for conference champions before the final rankings. If a top four team is not a conference champion, they have a chance of falling out of a playoff spot (25% with one loss, 50% with two losses, 75% with three losses).
These assumptions on conference champions have an enormous impact on playoff probability.
Does a championship game give the Big 12 an edge?
Suppose the Big 12 added a conference championship game with its current 10 team structure. The simulation finds the top two teams by conference record and has them play in a championship game.
Over the past three seasons, the Big 12’s playoff chances drops an average of 17%. This differs from the Navigate Research results that show a 5% increase.
My story almost ended here. The simulations made the clear statement that a championship game would hurt the Big 12. Their top team can’t drop without a loss, so don’t hold a championship game.
And I probably should have stopped here. “Big 12 analytics firm makes stupid conclusions” makes a pretty good headline. But I wanted to see how far I could push these results. What if the simulation gave more credit to teams that played in a title game?
How will the committee view a conference championship?
Let’s push the boundaries of how the committee gives credit to a conference championship.
- 1. Dropping teams without a conference championship. Now, a one loss team has a 50% to drop out of a playoff spot, and all two loss teams get dropped.
- 2. Moving up for winners of conference championship games. A team that wins a conference title game has a 75% chance to move ahead of the team in front of them. This places a high value on “one last chance to impress the committee,” an opportunity Big 12 teams and Notre Dame do not have right now.
With only the first criteria of a higher drop rate for non-champions, the Big 12’s playoff chances decrease an average of 9.2% with a championship game. The drop is smaller than in my original model (17%) but still significant.
With both criteria, the Big 12’s playoff chances increase by 5.5% with a championship game. If you assume the committee will give a bump to teams that win a conference championship game, then this game helps the odds. This 5.5% increase is about the same result obtained by Navigate Research.
What about expansion to a 12 team league?
Navigate Research also suggested expanding to a 12 team league. It’s impossible to know this might impact the Big 12 without knowing which teams would join the conference.
However, a simple trick lets us study how a championship game impacts a larger conference. I can eliminate the championship game from each the other Power 5 conferences and study how this changes their playoff odds.
First, consider the strongest assumptions on a conference championship that include both criteria of the previous section. Then a conference championship gives these conferences a 1.2% advantage to make the playoff, a smaller increase than the Big 12 increase of 5.5%.
In my study over three seasons, I get three probabilities for the Big 12 and 12 for the other Power 5 conferences with more teams. This small sample size most likely explains the differences in the playoff odds between these two sets. This also suggests that the estimate of a 5.5% increase for the Big 12 is probably high.
What does analytics say about expansion?
My simulations confirm that a Big 12 championship game would increase their odds of making the College Football Playoff. However, there are numerous reasons to doubt whether this increase actually exists.
First, the simulation assumes that the committee will give a strong preference to teams that win a conference championship game. This includes the dropping of teams without a conference title and the bumping of teams that win a championship game, an opportunity that doesn’t currently exist for Big 12 teams or Notre Dame.
The results are extremely sensitive to the parameters. For example, the simulation assumes a 75% chance that a team that wins a conference title game jumps over a team without such a win. This leads to a 5.5% edge for making the Playoff if the Big 12 holds a title game.
If the simulation only gives a 50% chance to make this jump, then a championship game hurts the Big 12. Remember, this extra game exposes a team to a possible loss. With these parameters, the Big 12 playoff chances decrease by 2.8% with a championship game.
What’s the honest truth on analytics and a Big 12 championship game? We don’t know. It’s impossible for me to assign pinpoint values to the parameters of my simulations, even if I had ten years of data on the selection committee.
The committee does seem intent on giving winners of a championship game every advantage. In 2014, a flawed Florida State team jumped TCU after winning the ACC championship game. In 2015, Michigan State and Stanford both moved ahead of teams without championship game wins (Oklahoma, Ohio State respectively).
But no one knows if the committee will keep placing such a high value on conference titles. And what about top teams that lose their conference championship game? If top ranked Clemson lost to North Carolina in the 2015 ACC title game, would they have dropped from the top 4?
What do you think? Do my methods give too much credit to conference champions? Let me know in the comments.
I think attempting to predict subjective, rational decisions of a committee by using statistics is a pretty risky business in general. Individual committee member bias, publicity, and the feeling of needing to spread the wealth around year to year if teams look even, all factor in.
One thing to consider is an undefeated Notre Dame will undoubtedly always bump out a conference winner of the weakest conference with or without a championship game.
Regarding factor #1 also consider what is the probability of the B12 Champion being undefeated? and the probability of an undefeated B12 Champ bumping say a one loss ACC or P12 Champ, assuming the B12 does not have a championship game?
Hopefully at some point the playing field will be leveled and all P5 conferences will be required to have a championship game, and Notre Dame will be required to join a P5 conference or else no longer be considered a P5 team.
All good questions, Marc. I don’t know if an undefeated Big 12 champ moves ahead over a one loss ACC or Pac-12 champ with a championship game, but I do think they make it.
Notre Dame has a lot of pull, so they’ll always be a Power 5 team even if they don’t join a conference. It makes things complicated though. Does BYU deserve the same status as ND? Probably not, but…
The primary motivation for the NCAA is to get the big name and big money teams in the playoffs which is why they passed over TCU for Ohio State. So the Big 12/10 chances all depend on the team that stands to go. If it is OU or Texas, chances are excellent. If it is Iowa State, they probably have no chance at all if there is a biggest money school to choose.
The 5% increase calculated by the consultants means 1 trip in 25 years which is not exactly a good reason to put Cincinnati in your conference.
Given that it is easier for OU or Texas to run through the Big 12/10 without a loss versus facing another tough team in the championship and losing, my guess is the Big 10 (12) is much better off standing pat.
The bottom line is the chances cannot be calculated due to the corruption of the NCAA system which is personified by the Big 12/10 as academic institutions which refuse to use correct math for the number of teams in the conference for $$$ purposes.
In thinking further, the chances that the team to advance is going to be Texas or OU is fairly substantial, and there is basically no chance that K State or Iowa State will advance, the conference championship seems a major negative. Because if a Iowa State or K State should advance, they are fairly likely to get knocked off in the championship
So if you reduce Texas (which is almost 100% assured to make it regardless if the record puts them there) or OU and then force lesser schools to have the additional hurdle of the conference championship in addition to the already substantial hurdle of watching the ratings suddenly change dramatically as happened with TCU and curious decisions in the NCAA basketball tourney, it is basically impossible that the conference would be more likely to advance to playoffs with a championship game as it reduces the chances for both the elites and the rank-and-file conference members.
But of course, the whole reason for the championship game is to make more $$$ so we know what the research is going to conclude and they will have a championship game whether it increases or decreases the odds if there is more money on the table.
“The 5% increase calculated by the consultants means 1 trip in 25 years which is not exactly a good reason to put Cincinnati in your conference.” Bingo!