Would a Big 12 championship game be a mistake? — Analytics on College Football Playoff odds

The Big 12 hired an analytics firm to advise them on the best way of making the College Football Playoff. After crunching the numbers, the firm suggested expanding to 12 teams, playing 8 instead of 9 conference games and having a championship game.

This idea of holding a championship game interests me the most. Navigate, the research firm, says this change alone would increase the Big 12’s playoff odds by 5 percent. (All three suggestions add up to an 10-15 percent increase.)

However, it doesn’t make sense that adding a championship game will increase a conference’s chance to make the playoffs.

Think back to 2015. Oklahoma finishes their regular season at 11-1 and 3rd in the committee rankings. Since the Big 12 didn’t have a championship game, the Sooners sat at home during the last week of the season. They couldn’t lose and fall in the rankings.

Here, we’ll use simulations to determine the effect of a championship game on a conference’s playoff chances. The results impact not only Big 12 teams but also Notre Dame.

The numbers suggest that the committee must place extremely high value on winning a championship game, not just being a conference champion, to see any kind of increase in playoff odds. Let me explain.

Analytics on conference championship games

In 2014, Bleacher Report asked me to calculate the chance that each team would make the College Football Playoff. I wrote about these simulation results in their weekly playoff odds report for the past two seasons.

I’m happy with the accuracy of these projections. The numbers didn’t like undefeated Utah in early in 2015, and they subsequently lost 3 games. I also predicted a difficult path for top ranked Mississippi State in 2014, a team that failed to make the playoff. You can make your own judgement based on the archive of predictions.

To look at conference championship games, I use the same methods to simulate the past three seasons retrospectively. Let’s look at the two key components to this calculation.

• College football team rankings. For each season, I use season ending rankings to calculate a win probability for each game. This uses the best estimate of team strength to retrospectively look at the season.
• Season long simulations. The simulation marches through each week of the season and picks winners at random according to the assigned win probability.

Each week, teams that lose drop in the committee rankings. Teams that win usually hold their place in the rankings but occasionally jump over teams ahead of them.

The simulation also determines the winners of divisions to hold conference championship games the last week of the season. The simulation accounts for conference champions before the final rankings. If a top four team is not a conference champion, they have a chance of falling out of a playoff spot (25% with one loss, 50% with two losses, 75% with three losses).

These assumptions on conference champions have an enormous impact on playoff probability.

Does a championship game give the Big 12 an edge?

Suppose the Big 12 added a conference championship game with its current 10 team structure. The simulation finds the top two teams by conference record and has them play in a championship game.

Over the past three seasons, the Big 12’s playoff chances drops an average of 17%. This differs from the Navigate Research results that show a 5% increase.

My story almost ended here. The simulations made the clear statement that a championship game would hurt the Big 12. Their top team can’t drop without a loss, so don’t hold a championship game.

And I probably should have stopped here. “Big 12 analytics firm makes stupid conclusions” makes a pretty good headline. But I wanted to see how far I could push these results. What if the simulation gave more credit to teams that played in a title game?

How will the committee view a conference championship?

Let’s push the boundaries of how the committee gives credit to a conference championship.

• 1. Dropping teams without a conference championship. Now, a one loss team has a 50% to drop out of a playoff spot, and all two loss teams get dropped.
• 2. Moving up for winners of conference championship games. A team that wins a conference title game has a 75% chance to move ahead of the team in front of them. This places a high value on “one last chance to impress the committee,” an opportunity Big 12 teams and Notre Dame do not have right now.

With only the first criteria of a higher drop rate for non-champions, the Big 12’s playoff chances decrease an average of 9.2% with a championship game. The drop is smaller than in my original model (17%) but still significant.

With both criteria, the Big 12’s playoff chances increase by 5.5% with a championship game. If you assume the committee will give a bump to teams that win a conference championship game, then this game helps the odds. This 5.5% increase is about the same result obtained by Navigate Research.

What about expansion to a 12 team league?

Navigate Research also suggested expanding to a 12 team league. It’s impossible to know this might impact the Big 12 without knowing which teams would join the conference.

However, a simple trick lets us study how a championship game impacts a larger conference. I can eliminate the championship game from each the other Power 5 conferences and study how this changes their playoff odds.

First, consider the strongest assumptions on a conference championship that include both criteria of the previous section. Then a conference championship gives these conferences a 1.2% advantage to make the playoff, a smaller increase than the Big 12 increase of 5.5%.

In my study over three seasons, I get three probabilities for the Big 12 and 12 for the other Power 5 conferences with more teams. This small sample size most likely explains the differences in the playoff odds between these two sets. This also suggests that the estimate of a 5.5% increase for the Big 12 is probably high.

What does analytics say about expansion?

My simulations confirm that a Big 12 championship game would increase their odds of making the College Football Playoff. However, there are numerous reasons to doubt whether this increase actually exists.

First, the simulation assumes that the committee will give a strong preference to teams that win a conference championship game. This includes the dropping of teams without a conference title and the bumping of teams that win a championship game, an opportunity that doesn’t currently exist for Big 12 teams or Notre Dame.

The results are extremely sensitive to the parameters. For example, the simulation assumes a 75% chance that a team that wins a conference title game jumps over a team without such a win. This leads to a 5.5% edge for making the Playoff if the Big 12 holds a title game.

If the simulation only gives a 50% chance to make this jump, then a championship game hurts the Big 12. Remember, this extra game exposes a team to a possible loss. With these parameters, the Big 12 playoff chances decrease by 2.8% with a championship game.

What’s the honest truth on analytics and a Big 12 championship game? We don’t know. It’s impossible for me to assign pinpoint values to the parameters of my simulations, even if I had ten years of data on the selection committee.

The committee does seem intent on giving winners of a championship game every advantage. In 2014, a flawed Florida State team jumped TCU after winning the ACC championship game. In 2015, Michigan State and Stanford both moved ahead of teams without championship game wins (Oklahoma, Ohio State respectively).

But no one knows if the committee will keep placing such a high value on conference titles. And what about top teams that lose their conference championship game? If top ranked Clemson lost to North Carolina in the 2015 ACC title game, would they have dropped from the top 4?

What do you think? Do my methods give too much credit to conference champions? Let me know in the comments.