This is a guest post from Tony Kaminski, a recent University of Michigan graduate and engineer. You can follow him on Twitter or check out his site Big House Analytics.

How does Jim Harbaugh’s fourth down decisions stack up to analytics?

The numbers say that coaches should go for it on fourth down way more than they do. Berkeley professor David Romer started this research back in 2006, and Brian Burke, now with ESPN, has performed the most complete study based on NFL data.

My interest in this topic as a Michigan fan began with a conversation with Craig Ross, a member of WTKA MGoBlog Roundtable and author of two books on Michigan sports. In his book The Search for the Unified Field Theory (Football Version), he writes

So here’s the deal. Einstein’s follies aside, I believe there is a unified field theory in football, a fundamental set of ideas or equations that can help to explain “what happened” in a football game, why the game was won or lost. More than this, I think this theory allows a coach to make choices and structure his team and program in a way that squeezes out a few more wins than might otherwise be the case.

And therein lies our aim: finding this unified theory. Craig and others have good reason to believe that not punting very much, or at least substantially less than coaches tend to do currently, is a part of applying this theory to winning more games.

The natural starting point of this conversation is the work of David Romer, the Berkeley economist. An excerpt from a New York Times profile of Romer explains the motivation looking at 4th downs:

Romer, a lifelong sports fan who is a professor at the University of California, Berkeley, came up with the idea to rigorously examine fourth-down plays after listening to a radio broadcast of an Oakland Raiders game in his car about a decade ago. Although the Raiders had the ball in striking distance of the end zone, one of the commentators remarked that they would be smarter to kick a near-certain field goal rather risk going for a touchdown.

“I am pretty analytic,” Romer recalled telling himself. “That is a pretty shallow way of thinking about it.”

So, after stewing over the idea for a couple of years, he set out to tackle the great fourth-down debate.

Romer’s work has been largely met with skepticism in coaching circles because the difficulty in overcoming conventional wisdom. Coaches at big-time universities or professional franchises with nine-figure payrolls often stick to the proverbial book — and why not?

If you fail underneath the brightest lights, it’s easier to defend yourself when your decisions echo the conventional wisdom. Wins come at a premium, and no career has less job security or more turnover than football coach. So you do everything you can to hold onto that gig, because they don’t come around often.

But doing great things, in the private sector or on the football field, involves bucking convention. Romer has shown this means punting less. Way less.

You can read this article for a quick tutorial on Romer’s work and its application. While the New York Times model doesn’t explicitly reference the professor, their “4th Down Bot” comes to a very similar conclusion, just with more data to back it up. It boils down to this:

Especially near the middle of the field, coaches need to be more aggressive. In this article, we’ll look at Jim Harbaugh’s fourth down decisions since his arrival at Michigan. A few points on this analysis.

• While Romer and the New York Times used NFL data, Romer believes the findings also apply to the college game. (Craig recounts an email conversation that him and the professor had in his book where Romer explains why.)
• Bill Connelly’s S&P has designated garbage time as being up by 25 points or more in the second quarter, 22 or more in the third, or 17 or more in the fourth. For the 2015 analysis, I filtered this out arbitrarily, in a more liberal manner than Bill does, based on my recollections of the individual games. In 2016, I followed Connelly’s rules.
• This is not an indictment of Harbaugh’s decision making on fourth down, merely an analysis of how closely his decisions echo Romer’s advice. There are specific team, opponent, or situation-variant times where it would be best to not follow the chart above.
• This data does not reflect the final outcome of the decision made but only the decision to go for a fourth-down conversion or punt itself.

Let’s first look at Harbaugh’s 2015 fourth down decisions in these tables. The color green designates a decision that follows what analytics recommends. Yellow means it did not but was defensible given the score, situation, or general trend of the game (more on that later). Red designates a time when Harbaugh probably should have gone for it on fourth down opposed to punting.

Let’s discuss a few of the decisions marked yellow, or ones I found defensible given the game situation. (Maybe I was a bit too friendly to our coach, I’m a fan and proud alum, after all — leave a comment if you disagree with any of my designations, and we’ll continue the conversation).

For instance, Michigan punts on fourth and seven from the Michigan State 38 with 1:12 to go in the first half of a 10-7 game. That’s not technically the correct decision but consider the following:

• Michigan State is out of timeouts
• Michigan had allowed exactly seven points over the prior 17 quarters
• Michigan had one of the best punters in college football in 2015

Michigan State runs three quick plays without scoring before halftime.

Against Northwestern, it’s fourth and nine from their 34 with 4:19 to go in the second quarter. Michigan has a 21-0 lead. The chart says to try a field goal but that’s too deep for then-freshman kicker Kenny Allen, who had missed a 44-yard attempt earlier in the season. Michigan punts, and then Jourdan Lewis houses an interception on the ensuing possession that erases any doubt about the outcome.

This isn’t to say I agree with all of the times Harbaugh deviated from the research. I was upset by the decision to punt from Ohio State’s 36 on fourth and five less than five minutes into that game. Given the talent discrepancy between the Wolverines and Buckeyes, who littered the first round of the NFL draft (literally, probably), points would be at a premium, and we passed up an opportunity to get some early.

Ultimately, Harbaugh had a 77.8% “success rate” — out of Michigan’s 54 fourth down decisions, Harbaugh went with analytics or I found the punt defensible on 42 of them. If you want to go by the book (meaning the punts I deemed as defensible but not Romer-approved counting against our success rate instead of for it), our score is 59.3% (32 of 54).

The following table shows my results for the 2016 season through the Ohio State game.

On the following fourth downs, Harbaugh went for the first down.

This analysis shows a 91.3% success rate (42 of 46). The results were so staggering that I had to have a friend double check my work. If you went purely by the numbers, our success rate is 76.0% (35 of 46) . By either metric, this is a substantial improvement from last season.

A couple notes:

• Michigan blew out seven of their 12 regular-season opponents, so this chart was significantly easier to compile than last year’s.
• I did not chart the “defeat with dignity” portion of the Michigan State game, the final offensive play against Indiana (it was a full-blown blizzard in Ann Arbor at that point and IU had effectively conceded), or the fourth-down attempt in overtime against Ohio State since we obviously didn’t have a choice.
• There were only four punts I gave the “red” designation, and even then, I felt that I may have been being unreasonably harsh on Harbaugh, since these were coming up correct at a much higher clip than last season.
• Michigan was 12 for 18 on fourth-down conversion attempts in 2016 compared to six for 16 in 2015. More of these came in garbage time this year than last, hence the disparity in charted attempts.

Harbaugh’s decision making already agreed with analytics a good bit in 2015, but 2016 represents an even bigger step forward. I have a few hypotheses on why.

First, Michigan has a larger analytics team than under previous regimes. It’s not hard to believe that they had a part in this trend.

Second, Harbaugh had a different situation on defense these two years. In 2015, the defensive production fell off with the injury of Ryan Glasgow, most notably when Indiana gashed Michigan for 307 rushing yards in Bloomington and the dreadful Ohio State game.

This was not the case in 2016. Don Brown’s defense was stout all season long, which gave Harbaugh the confidence the defense could get a stop on a short field.