Football Analytics

Podcast: Dr. Eric Eager on football analytics in 2020

On this episode of The Football Analytics Show, I’m joined by Dr. Eric Eager, the Executive Director of Research and Development at Pro Football Focus. Among other topics, we discuss:

How PFF uses grades for the betting markets (3:06)
How PFF grades deal with cornerbacks (7:08)
The most important research PFF did this off season (10:48)
The insights from the PFF QB Annual, and two QBs to fade based on this data (15:56)
A bold prediction for the NFL in 2020 (36:02)

We also broke down a number of NFL teams:

San Francisco and fragility (23:09)
Kansas City and why not to go against them (27:52)
Green Bay and Minnesota based on week 1 (30:08)
Los Angeles Rams and the defense (31:58)
Philadelphia (33:20)
Tennessee (36:02)
Pittsburgh (38:10)

Dr. Eager is a superstar in combining data and watching football, and you’ll get a ton out of his insights during this episode.

To listen, click on the right pointing triangle.

To listen on Apple Podcasts, click here.

The Football Analytics Show is also available on Spotify, Stitcher and Google Podcasts.

How to predict interceptions in the NFL

By Dr. Ed Feng 5 Comments

This work is adapted from The Pick Report, my full study on how to predict interceptions. To check out the full product, click here.

Is Carson Wentz skilled at not throwing interceptions?

The raw data on the Eagles quarterback certainly supports this claim. While the NFL average interception rate has been 2.4% the past three seasons, Wentz has been better:

2017 – 1.6%
2018 – 1.7%
2019 – 1.1%

Over these years, Wentz had his best season in 2017 despite not having his best interception rate. The former #2 pick overall led Philadelphia to an 11-2 start before an injury derailed his season. Nick Foles finished the job as the Eagles won the Super Bowl.

Wentz had his best season of interception prevention in 2019. However, the offense struggled as he dealt with injuries to his receivers.

Will Wentz’s ability to prevent interceptions continue? Unfortunately, we can’t make any statements based on his past interception rate. The data shows very little predictability for interceptions, a concept I’ll make precise in the next section.

To predict interceptions, we need to first check the skill in throwing interceptions and then dig deeper into the data on NFL quarterbacks. Let me explain.

The predictability of interceptions

Why are interceptions difficult to predict?

To look at the most basic idea of predictability, consider a quarterback’s interception rate, or interceptions divided by pass attempts. We ask how sticky this quarterback statistic is from season to season.

Stickiness is determined by calculating the correlation of interception rate from one season to the next. This analysis considers the 2000 through 2019 seasons, and a quarterback gets included if he has thrown 200 pass attempts in consecutive seasons.

Interception rate the previous season explains only 7% of the variance in interception rate during the current season. To understand this result, think of a plot of data points that correspond to consecutive seasons. The interception rate from the previous season is on the x-axis, or the horizontal axis. The interception rate from the current season, or the variable we would like to predict, is on the y-axis, or the vertical axis.

If interception rate were a good predictor, the data points would hug a line. There would be scatter around the line, since no predictor is perfect. But the data would suggest a clear relationship between the two quantities.

There is no clear relationship between interception rate in consecutive seasons. Interception rate from the previous season explains 7% of the variance in the current season, and this 7% is known as the R-squared value. The plot of the data looks like a Jackson Pollock painting.

But are interceptions really that random and unpredictable? To think about this in a different way, let’s ask about the relative contribution of skill and luck.

The skill in interceptions

To understand how this works, let’s first assume there is zero skill in throwing interceptions. This means that every quarterback has a 2.4% chance of throwing an interception on a pass attempt. If this assumption matches up with data on players, then interceptions would be all luck.

However, this seems improbable. Consider Jameis Winston. Tampa Bay fans had high hopes when he was drafted as the top pick in the 2015 draft. They had even higher hopes in 2019 when Bruce Arians arrived as his head coach.

However, Jameis was a turnover machine in 2019, as he threw interceptions on 4.8% of his pass attempts. This percentage was a little bit of an anomaly. In his career before Arians, he had a 3% interception rate. That’s not good, but it’s not atrocious like his 2019 rate.

Over his career, Jameis has thrown 2,551 pass attempts. If interceptions were all luck, we would expect his interception rate to be close to the NFL average of 2.4%. With more pass attempts, the more likely his interception rate is close to this average. Remember, the assumption is that interceptions are all luck. This makes the math easy, as the distribution of Jameis’ interception rate is something called a binomial distribution.

With enough pass attempts, this distribution looks like a normal distribution, or the bell curve that you often see. For Jameis, this bell curve implies a 2 in 3 chance his interception rate is between 2.1% and 2.7%. This means the width, or standard deviation, is 0.3%. If Jameis threw another two thousand pass attempts, the width would get smaller.

Jameis’ actual interception rate is higher than expected from the randomness assumption. During his career, he’s averaged a 3.5% interception rate. Based on the width of 0.3%, Jameis is three and a half standard deviations worse than NFL average. That’s as bad as the acting of Hayden Christiansen as Anakin Skywalker in Star Wars.

How about the new quarterback in Tampa Bay? Tom Brady has thrown picks on 1.4% of his pass attempts over the last six seasons. Based on the randomness assumption and 5,500 pass attempts, he is four standard deviations better than the NFL average.

These extremes in interception rate suggest a contribution of skill to interceptions. To quantify this, I’m going to use an idea from Michael Maubossian in his book The Success Equation. He said that:

Outcome = Skill + Luck

Here, the outcome is interception rate. The skill part is related to the accuracy of the quarterback. A player who can regularly throw a football through a tire 50 yards away will throw fewer interceptions. Aaron Rogers has a 1.1% interception rate since the 2014 season.

A rate that low might not be the best idea. Minimizing interception rate should not be the goal in football. The goal should be to win football games. If a team is down late in the game and needs a touchdown, the optimal strategy is throwing to Davante Adams in double coverage, even though there’s a higher chance of a pick. This is better than dumping the ball off to a running back.

In addition to skill, there is luck involved in throwing interceptions. Even the most accurate quarterback will throw an errant pass at times. In addition, the speed of a modern NFL defense forces some tipped passes and pass break ups. The offense has no control over the trajectory of the ball once these plays happen. Despite their best efforts, Tom Brady and Aaron Rogers will never have a zero interception rate.

With the Maubossian model, let’s consider the variance of the outcome, or the variance in interception rate. This is given by the following:

Var(Outcome) = Var(Skill) + Var(Luck)

Usually when we take the variance of a linear equation, there’s a term that considers the correlation between skill and luck. However, by definition, there is no correlation between skill and luck. Luck, good or bad, does not discriminate between Jameis Winston and Tom Brady.

Before, we looked at the deviation in interception rate for Jameis Winston and Tom Brady. Let’s do the same for all quarterbacks the past six seasons. Based on the randomness assumption, we calculate a standard deviation from NFL average. This depends on the number of pass attempts, and the analysis only includes quarterbacks with 500 pass attempts during this time period.

If interceptions were all luck, this player data would produce a bell curve with a width of one. The preliminary analysis suggested a distribution with a larger width. The wider this distribution, the more skill in interceptions.

The key quantity is the fraction of variance in outcome that comes from variance in skill. This is similar to what we looked at with predictability.

With predictability, we were interested in the interception rate for a single season. We reported how much of the variance in this interception rate was explained by interception rate the previous season.

With skill, we consider a player’s interception rate over six seasons. We will report how much of the variance in this interception rate is explained by skill. There would be some variance in interception rate due to randomness. The simple Maubossian model attributes the excess variance to skill.

Based on the past six seasons, interception rate is 66% skill.

Interception rate has low predictability. The interception rate last season explains 7% of the variance in interception rate this season. However, it’s 66% skill.

We see the same trend with three point shooting in the NBA. For predictability, last season’s three point shooting percentage for a player explains 14.5% of the variance in this year’s three point shooting percentage. However, three point shooting is 78% skill.

Interceptions and three point shooting have low predictability but high skill.

This skill in interceptions suggests we should be able to do better at predicting interception rate. To do this, we need to consult a different variable.

A hidden variable

To better predict interceptions, let’s consult an idea first introduced by Bill Connelly during his time at SB Nation and expanded upon by Aaron Schatz of Football Outsiders.

To understand this idea, consider passes defended. An incomplete pass is considered to be defended if one of three events happens:

The pass gets tipped at the line of scrimmage.
The ball gets deflected by a defender in coverage.
A defender hits the receiver as the ball arrives and jars the ball loose.

These three events get counted as passes defended. In looking at a quarterback, we can track how often he puts a ball in a dangerous situation through these passes defended.

Consider the rate at which a quarterback allows the defense to break up a pass. His passes defended rate is passes defended divided by pass attempts.

First, let’s ask about the stickiness of this quantity from year to year. For quarterbacks, last year’s passes defended rate explains 20% of the variance in passes defended rate during the current season. That’s almost three times as much variance explained as interception rate.

What about skill versus luck in passes defended? Over the past six seasons, 75% of passes defended rate is explained by skill. This is more than the 66% skill with interception rate but less than the 86% skill with completion percentage.

The idea with passes defended is that a certain fraction of these balls will end as interceptions. In an ideal world, this fraction should not be sticky from season to season. We expect strong regression to the mean. This is Bill Connelly’s original analysis in college football.

To check this, consider interceptions as a fraction of bad balls. Bad balls are all passes in which a quarterback puts the ball in a dangerous situation. This includes interceptions and passes defended.

Now consider the fraction of bad balls that end up as interceptions. To calculate this, take interceptions and divide by interceptions plus passes defended. In the NFL from 2014 to 2019, 21.3% of bad balls ended up as interceptions.

This percentage of interceptions to bad balls is not sticky at all. There is zero correlation from season to season. This means we can expect strong regression to the mean from year to year.

We should also check the skill in this rate of interceptions to bad balls. The analysis reveals this rate is 29% skill. The rate of interceptions to bad balls is not all random. Compared to the NFL average of 21.3%, Jameis Winston is still at one extreme with a 27.6% pick to bad ball rate. Aaron Rodgers is at the other extreme with a 14.3% rate, but Tom Brady appears closer to the mean at 19.0%.

In the end, the skill in interceptions per bad balls at 29% is way less than the skill in fumble rate on sacks, which is 49% skill. In addition, with the low level of predictability, the picks to bad ball rate should regress to the mean for quarterbacks.

Carson Wentz in 2020

Will Carson Wentz continue to prevent interceptions? The data suggests no.

In 2019, 10.5% of his bad balls ended up as interceptions. This quantity shows strong regression to the mean, which implies that Wentz’s interception to bad ball rate will be closer to the NFL average of 21.3% in 2020.

It is not likely Wentz will have an interception rate as low as 1.1% again like he did in 2019. He will be closer to the NFL average of 2.4%.

This analysis was adapted from The Pick Report, my study on how to predict interceptions. To get the full report, click on “Add to Cart.”

Add to Cart.

The surprising truth about passing and rushing in the NFL

By Dr. Ed Feng 4 Comments

Before the 1991 Super Bowl, Bill Belichick told his Giants defense to let Buffalo running back Thurman Thomas rush for 100 yards.

As David Halberstam writes in Education of a Coach, it was a tough sell. The New York Giants played a physical defense that prided itself on not allowing 100 yard rushers.

No matter, the short, stout coach looked straight into the eyes of Lawrence Taylor and Pepper Johnson and said, “You guys have to believe me. If Thomas runs for a hundred yards, we win this game.”

Just in case his players didn’t listen, Belichick took it upon himself to ensure Thomas got his yards. He took out a defensive lineman and linebacker and replaced those large bodies with two defensive backs. In football lingo, the Giants played a 2-3-6 defense designed to struggle against the run.

Did Bill Belichick go insane? I certainly thought so when I first read this story years ago.

However, analytics is on Belichick’s side. Let me explain.

The truth about rushing in the NFL

To examine Belichick’s strategy, let’s use analytics to evaluate the importance of rushing compared to passing in the NFL.

To make this comparison, we must move past misleading statistics like rush yards per game. Teams with the lead run the ball to take time off the clock. Any team can rush for 100 yards if they run it 50 times.

To measure true skill, let’s consider yards per attempt, a powerful efficiency metric. A team can’t fake their way to 5 yards per carry by running the ball more.

In the NFL, I’ll define team efficiency for passing and rushing as yards gained per attempt on offense minus yards allowed per attempt on defense. Better offenses gain more yards per attempt. However, better defenses allow fewer yards per attempt, so subtracting these smaller values leads to higher team efficiency.

Team efficiency can come from the offense, defense, or both. For example, consider Carolina in 2015, the season they played in the Super Bowl. The offense gained 6.2 yards per attempt, 13th best in the NFL. This rank might seem low given that quarterback Cam Newton won Most Valuable Player award that season. The defense allowed 5.0 yards per attempt, 2nd best. This gives a team pass efficiency of 1.2 yards per attempt, 5th in the NFL.

In these numbers, sacks count as pass attempts. These negative yards count against the yards per attempt for an offense. In rush efficiency, the value comes directly from yards per carry.

To examine the relative importance of passing versus rushing, let’s look at how playoff and Super Bowl winning teams performed in team efficiency. Strong teams in passing and rushing will have positive efficiency values, or better than the NFL average of zero.

NFL playoff teams excel in passing, either by throwing the ball on offense, preventing the pass on defense, or both. From 1998 through 2017, only 39 of 252 playoff teams allowed more yards per pass attempt than they gained. This implies that 84.5% of playoff teams had a positive pass efficiency.

Super Bowl champions excelled in pass efficiency as well, as 15 of the 21 champions had pass efficiencies of a yard per attempt or more.

However, excellence in the air does not guarantee success in the small sample size of the playoffs. Three Super Bowl champions had negative pass efficiency during the regular season. Let’s look more closely at two of these Super Bowls.

After the 2007 season, the New York Giants beat New England, a team with Tom Brady and Randy Moss looking for an undefeated season.
After the 2001 season, New England beat St. Louis, the Greatest Show on Turf led by Kurt Warner and Marshall Faulk.

These are considered two of the greatest Super Bowl upsets ever.

The importance of the passing game in the NFL should not be a surprise. Quarterbacks dominate the headlines and earn the highest salaries. With the importance of throwing the ball on offense, it makes sense that teams with a strong pass defense also excel.

However, the insignificance of rushing in the NFL might surprise you.

From 1997 through 2017, only 57.5% of playoff teams (145 of 252) had a positive team rush efficiency. The visual of rush efficiency for playoff teams shows a random scatter of points with both positive and negative values. A strong run game or stout rush defense has little effect in helping an NFL team win enough games to make the playoffs.

Moreover, running the football doesn’t suddenly become important in the playoffs. If it did, the teams with positive values of team rush efficiency would make the Super Bowl. Instead, only 26 of 42 Super Bowl teams from 1997 through 2017 had a positive rush efficiency value, a rate of 61.9%.

For example, Indianapolis won the Super Bowl after the 2006 season despite having the NFL’s worst team rush efficiency during the regular season. Green Bay ranked 31st out of 32 teams in rush efficiency during the 2010 regular season, but the Packers won the Super Bowl anyway.

These teams had elite quarterbacks, as Peyton Manning of Indianapolis and Aaron Rodgers for Green Bay propelled these teams to regular and post season success. We’ll get back to these two later when we explore whether a team needs a run game to throw the ball effectively.

These raw yards per carry numbers can get skewed a bit. Teams that excel in pass efficiency will tend to have the lead late in games. In these situations, they will run the ball more to take time off the clock. The defense expects the run, which should hinder their rush efficiency.

From 2012 through 2016, teams up 7 or more points in the fourth quarter rushed for 3.29 yards per carry, well below the NFL average of 4.18. While this certainly hurts the rush efficiency of good teams, these carries only make up 12.0% of all carries during this period.

Teams down 7 or more points in the fourth quarter rush for 4.91 yards per carry, so these bad teams get a bump in rush efficiency. However, these carries only make up 5.1% of all NFL carries, as these teams opt to throw the ball as the best means to get back into the game.

Excluding these carries would not change the main claim about the insignificance of rushing in the NFL.

Sign up for football predictions

Hope you’re enjoying the article so far. You have a ways to go, as we haven’t even discussed whether you need the run to set up play action.

If you sign up for my free email newsletter, I’ll send a paper copy of this manifesto to read at your leisure. You’ll also get a weekly sample of football predictions usually reserved for paying members of the site.

To get this free service, enter your best email and click on “Sign up now!”

A guessing game of a team’s wins

In the NFL, running the ball does not affect winning as much as you think. To illustrate this point, consider this guessing game. Suppose you want to guess how many games a team will win during the regular season. Without any other data, it makes sense to guess 8, the average number of wins in a 16 game season.

From 1997 through 2016, this estimate would be wrong by 3.08 wins. In technical jargon, 3.08 is the standard deviation of actual wins from the guess of 8. In normal people language, this means 2 of 3 teams will be within 3.08 wins of the guess. About two thirds of NFL teams won between 5 and 11 games between 1997 and 2016.

How much better does your guess get knowing the rush efficiency for each team? To determine the correction, you plot wins as a function of team rush efficiency and draw a best fit line through the data. The bottom panel of the visual shows this relationship for all NFL teams from 1997 through 2016.

The regression line informs a new guess about the number of games a team will win. For example, suppose a team has a rush efficiency of 0.6 yards per carry. Instead of guessing 8 wins for this team, the line shows 8.7 wins for this team.

How much better are these new guesses? Not much. The error only drops from 3.08 wins to 3.01 wins. In technical jargon, rush efficiency explains only 5.0% of the variance in wins. (For a simple, visual explanation of the previous statement, click here.) You might as well guess at random.

The results of the guessing game get better with team pass efficiency, as the visual shows that better passing teams tend to win more games. The error in estimating wins drops from 3.08 to 1.95. Pass efficiency explains 60.2% of the variance in wins in the NFL.

In college football, rush efficiency correlates more strongly with wins than in the NFL. Teams like Alabama, Stanford and Wisconsin have won with a power running game and a physical front seven on defense. This level of insignificance in the rushing game is unique to the NFL.

To put these numbers in context, let’s look at two NFL franchises and how their fate depended on the efficiency with which they threw or ran the ball.

Indianapolis had a remarkable run from 2003 through 2010. Under the leadership of General Manager Bill Polian and quarterback Peyton Manning, the Colts won at least 12 games each year before slacking off with 10 wins in 2010.

The Colts achieved success through the air, ranking in the top 8 in team pass efficiency each year. Peyton Manning and his offense played the bigger role, but the pass defense helped out some years. The Colts ranked in the top 10 in yards allowed per pass attempt from 2007 through 2009.

However, Indianapolis was really bad in the run game. Only once in this era (2007) did they gain more yards per carry than they allowed.

In 2006, they had the worst team rush efficiency in the NFL, as their rush defense allowed a league worst 5.3 yards per carry. The rush defense performed better in the playoffs, as they allowed a near NFL average 4.1 yards per carry. However, Manning and the pass offense led Indianapolis to their Super Bowl win that year over Chicago.

In contrast to Indianapolis, Minnesota dominated the ground game during this same period. They featured running back Adrian Peterson on offense and had tackles Pat and Kevin Williams clogging up the middle on defense. From 2006 through 2013, Minnesota finished 1st in rush efficiency four times and 3rd another two times.

However, this strength led to ups and downs in wins. Minnesota went 3-13 in 2011 despite leading the NFL in rush efficiency. The next season, they led the NFL again behind a monster season from Peterson, who made a remarkable return from knee surgery. The Vikings had a 10-6 record that season.

The Viking’s best season over this stretch came in 2009. They finished 12th in rush efficiency that season, by far their worst rank. The difference? A quarterback named Brett Farve came out of retirement to play for Minnesota. The Vikings finished 7th in yards gained per pass attempt, went 12-4 and came within a late turnover against New Orleans of playing in the Super Bowl.

While these two franchises represent the extremes, the trend is clear. In the modern NFL, teams that can throw the ball and defend the pass win football games and championships. It matters little whether they can run the ball or stop the run.

Will this trend continue into the future? Will the NFL continue to prize the quarterback, cornerbacks and pass rushers? Or will running backs make a come back soon?

To guess at the future, we first need to look at the past and the evolution of the passing game.

Bill Walsh and the West Coast Offense

In the late 1960’s, Cincinnati’s offensive coordinator Bill Walsh had a problem: his players sucked.

As Michael Lewis describes in his book Blindside, the quarterback couldn’t throw the ball more than 20 yards down the field. The receivers defined the term “replacement level.” They may have struggled in a flag football game.

With this lack of talent, Walsh had to innovate.

Instead of a deep passing game that required a strong armed quarterback and fast receivers, he developed a passing game based on short drop backs by the quarterback and well-timed passes to the receivers. The ball traveled a shorter distance in the air, making an interception less likely. If the quarterback could hit the receiver in stride, the offense might gain as many yards after the catch as in the air before it.

Walsh got many of these ideas from San Diego offensive coordinator Sid Gilman, but he put his own spin on the West Coast offense. Soon, he coaxed quarterback Virgil Carter into leading the NFL in completion percentage in 1971.

Then Walsh made a pleasant discovery. This West Coast offense didn’t just work when your players sucked. It kept moving the ball efficiently with better players. Cincinnati made the AFC championship game in 1975 with Kenny Anderson at quarterback.

By the time Walsh became the San Francisco head coach in 1979, the NFL had tipped the rules in favor of the passing game. Offensive lineman could now use their hands when blocking, which made it easier to protect the quarterback. Defensive backs could now put their hands on a receiver only within 5 yards of the line of scrimmage instead of everywhere on the field.

Walsh thrived in San Francisco during the 1980’s. His teams won three Super Bowls based on his high powered aerial attack. With this success, the rest of the NFL began to adopt the West Coast offense and throw the ball more.

The fraction of pass plays rose drastically after 1978 and then continued to rise to 55% by the end of the 1980’s. This infatuation with the pass offense has only accelerated since the turn of the century. By 2016, the fraction of pass plays has reached almost 60%, as the NFL no longer has many blocking tight ends or full backs.

With all of these pass plays, one might expect the efficiency of these plays to decrease. The defense begins to take away the passing game, making it easier for the offense to run the ball.

However, the exact opposite has happened. Yards per pass attempt, a number that includes the negative yardage from sacks, increased drastically after the 1978 rule changes. This pass efficiency has continued to increase after the turn of the century despite an increasing fraction of pass plays.

In contrast, the NFL run game has seen only small gains in efficiency. While the NFL averaged about 4 yards per carry in the 1980s, ball carriers now squeak out 4.18 yards per carry from 2012 through 2016.

Not only has the pass game become more efficient, it has also become safer. In the 1970’s, more than 5% of pass attempts ended in the hands of the defense. But with the adoption of the short passing game, the interception rate has dropped in a linear fashion and reached a new low of 2.2% in 2016.

The passing game is the gift that keeps on giving in the NFL. The increases in efficiency and safety despite an increasing fraction of pass plays suggest that the pass game will continue to dominate the NFL.

So let’s take this idea to its logical extreme. Should NFL teams throw on every down?

The NFL’s most efficient play

To understand whether teams should throw on every down, let’s ask a question first proposed by Grantland writer Robert Mays in 2013: what is the NFL’s most efficient play?

To determine the answer, Mays consulted the ESPN analytics group. They perform calculations that determine the value of each play through the idea of expected points added (EPA). This concept puts the result of a play in context; a 2 yard gain means more on 3rd and 1 than 1st and 10.

To understand EPA, suppose a team has a 1st and 10 at their own 20 yard line. They could drive the length of the field for a touchdown for +7 points or kick a field goal for +3 points. In the worst case, an interception gets returned for a touchdown, netting -7 points for the offense.

Expected points added (EPA) is the expected points gained or lost from a play. For example, a team starts with 1st and 10 at its own 20 yard line, which has an expected points of 0.3. Let’s say they gain 20 yards on first down. The resulting 1st and 10 from the 40 has an expected points of 1.3. Hence, the 20 yard gain results in a EPA of 1.0.

Mays had the ESPN analytics group look at the EPA for different types of plays. As a baseline, they first looked at rush plays based on data from the 2009 through 2013 seasons. On average, a rush play earns -0.04 EPA.

In case you missed the negative sign in front of that number, the average run play in the NFL loses points for the offense. That might be the biggest indictment of running the ball in the NFL yet.

First, the efficiency of run game by yards per carry has stagnated since the late 1970’s. With a closer inspection of run plays based on EPA, the average run play hurts a team in its quest to score more points than the other team. During the 5 year period of the study, only 4 of 32 NFL teams averaged a positive EPA on run plays.

In contrast, the average pass played earned a team +0.04 EPA. Throwing the ball moves a team in the right direction to score more points than the opponent.

A certain type of pass play does much better than the typical. The play action pass nets the offense +0.17 EPA. The deception of play action, a play in which the quarterback fakes a hand off before throwing down field, leads to a staggering increase in the point value of the play.

Should a team throw the ball on every play? No, since the lack of run plays kills the deception necessary for play action. Throwing on every down makes the offense predictable.

However, the study also found that a team doesn’t need to run the ball effectively to make play action work. Remember those 4 teams that had a positive EPA on run plays over the 5 seasons of the study? Only one of them had a top 10 EPA for play action. This team (New England) had a quarterback named Tom Brady.

During this period, Adrian Peterson of the Minnesota Vikings was the most dominant running back in the NFL. However, Minnesota never excelled at play action. The Vikings ranked 21st in EPA on play action. In 2013, Buffalo led the NFL in rush attempts but ranked last in EPA on play action.

The analytics suggest that a team doesn’t have to run the ball well to excel at play action. Despite the inherent deception, play action is a pass play whose success relies on a accurate pass down the field. A team needs a good quarterback to make that pass. In the study, the top play action teams (New England, Green Bay, Pittsburgh) had elite quarterbacks (Tom Brady, Aaron Rodgers, Ben Roethlisberger, respectively).

Even in showing that teams need to run the ball, we run into the insignificance of running the ball well.

How to evaluate NFL statistics

In Super Bowl XXV, Bill Belichick’s plan to let Thurman Thomas rush for 100 yards worked, maybe too well. Against a small defense designed to slow down the pass, Thomas ran for 135 yards on 15 carries, a staggering 9 yards per carry. In the second half, he broke off a 31 yard run for a touchdown.

The game ended when Bills kicker Scott Norwood sent a field goal attempt wide right. The Giants won the Super Bowl 20-19.

The Giants did not win the game solely because of Belichick’s defensive plan. The offense generated two long scoring drives in the second half that took time off the clock. And I would bet my life savings Belichick did not want his defense to allow that 31 yard touchdown run to Thomas.

But, as Halberstam discusses in Education of a Coach, Belichick did want the Bills to pick up small gains on the ground if it meant keeping Jim Kelly from throwing the ball. He understood that rushing means little to winning in the NFL.

Each year, the significance of the passing game in the NFL becomes more apparent. Teams offer huge contracts to cornerbacks like Patrick Peterson while future Hall of Fame running backs like Adrian Peterson wait for a free agent deal.

Teams also now look to the NFL draft to fill needs in the running game. They can draft a running back good enough to crank out 4 yards per carry behind a decent offensive line. He doesn’t cost much on a rookie contract, leaving salary cap resources for the pass game on offense and defense.

The insignificance of the run game should also impact NFL handicapping. You have limited time to evaluate injuries, so focus on those that affect the passing game. The quarterback is an obvious adjustment, and an injury to a top receiver matters too. But cornerbacks do not get the same attention even though they might be the NFL’s most difficult players to replace.

Passing dominates the NFL. Rushing hardly matters.

How to make accurate football predictions with linear regression

By Dr. Ed Feng 9 Comments

As a smart football fan, you would like to identify overrated college football teams. This is a difficult task, as half of the top 5 teams in the preseason AP poll have made the College Football Playoff the past 4 seasons.

However, analytics has a nifty trick for identifying teams that don’t belong near the top of the preseason polls.

In addition, this trick lets you look at the statistics on any major media site and identify teams playing above their skill level. In a similar fashion, you can find teams that are better than their record.

The trick relies on regression to the mean.

When you hear the word regression, you probably think of how extreme performance during an earlier period most likely gets closer to average during a later period. It’s difficult to sustain an outlier performance.

This intuitive idea of reversion to the mean is based on linear regression, a simple yet powerful data science method. It powers my preseason college football model that has predicted almost 70% of game winners the past 3 seasons.

The regression model also powers my preseason analysis over on SB Nation. In the past 3 years, I haven’t been wrong about any of 9 overrated teams (7 correct, 2 pushes).

Linear regression might seem scary, as quants throw around terms like “R squared value,” not the most interesting conversation at cocktail parties. However, you can understand linear regression through pictures.

Let me explain.

1. The 4 minute data scientist

To understand the basics behind regression, consider a simple question: how does a quantity measured during an earlier period predict the same quantity measured during a later period?

In football, this quantity could measure team strength, the holy grail for computer team rankings. It could also be turnover margin or win percentage in one score games.

Again, consider this question:

How does a quantity in an earlier period predict the same quantity during a later period?

Some quantities persist from the early to later period, which makes a prediction possible. For other quantities, measurements during the earlier period have no relationship to the later period. You might as well guess the mean, which corresponds to our intuitive idea of regression.

To show this in pictures, let’s look at 3 data points from a football example. I plot the quantity during the 2016 season on the x-axis, while the quantity during the 2017 season appears as the y value.

If the quantity during the earlier period were a perfect predictor of the later period, the data points would lie along a line. The visual shows the diagonal line along which x and y values are equal.

In this example, the points do not line up along the diagonal line or any other line. There is an error in predicting the 2017 quantity by guessing the 2016 value. This error is the distance of the vertical line from a data point to the diagonal line.

For the error, it should not matter whether the point lies above or below the line. It makes sense to multiply the error by itself, or take the square of the error. This square is always a positive number, and its value is the area of the blue boxes in this next picture.

The area of the blue boxes is the mean squared error.

In the previous example, we looked at the mean squared error for guessing the early period as the perfect predictor of the later period. Now let’s look at the opposite extreme: the early period has zero predictive ability. For each data point, the later period is predicted by the mean of all values in the later period.

This prediction corresponds to a horizontal line with the y value at the mean. This visual shows the prediction, and the blue boxes correspond to the mean squared error.

The area of these boxes is a visual representation of the variance of the y values of the data points. Also, this horizontal line with its y value at the mean gives the minimum area of the boxes. You can show that every other choice of horizontal line would give three boxes with a larger total area.

Regression requires finding the line that minimizes the squared error, or the area of the boxes. This line is called the best fit line, and the next visual shows the best fit line along with the corresponding minimum mean squared error.

In trying to scare off normal people, the quants will thumb up their nose and say things like “the best fit line explains 70% of the variance.” Even worse, they might call this the “R squared” value.

You can understand this statement through the pictures. The best fit line explains 70% of the variance means that the total area of the red boxes is 70% less than the original blue boxes of the horizontal line.

In this example, the best fit line causes a significant reduction in the area of the box corresponding to the left data point. The box gets larger for the middle point (the blue box is obscured by the red box corresponding to the best fit line). But overall, the area of the red boxes are 70% less than the blue boxes.

You can also think about the error the best fit line doesn’t explain. The area of the red boxes is 30% of the area of the blue boxes. This 30% is the remaining variance after the best fit line removes 70% of the original variance.

The higher the R squared value (70% in the example above), the smaller the red boxes of the best fit line. The line explains the data very well.

In contrast, the lower the R squared value, the larger the red boxes of the best fit line, which will be more horizontal. It doesn’t do much better than the horizontal line of the average.

Get my best football predictions

The Power Rank specializes in making accurate football and March Madness predictions.

To get my college football win totals this preseason and a sample of football predictions usually reserved for paying members, enter your best email and click on “Sign up now!”

2. Persistence versus regression to the mean

The data above come from my team ratings in college football. To develop these numbers, I take margin of victory in games over a season and adjust for strength of schedule through my ranking algorithm. The rating gives an expected margin of victory against an average team on a neutral site.

For the 2014 through 2016 season, here is how the team rating for the season predicted the next season.

The data hug the best fit line, and ratings from the previous season explain more than half of the variance in ratings the following season (54.1%).

Compare this to the same plot with turnover margin, or take aways minus give aways.

For turnover margin, the best fit line is almost flat. Turnover margin in one season explains 2.6% of the variance in turnover margin the following season.

From these two plots, we can make two statements:

Team strength in college football as measured by adjusted margin of victory tends to persist from year to year. This will be useful in making a preseason college football model.
Turnover margin regresses to a mean of zero from year to year. This implies that turnover margin last season has almost zero ability to predict turnover margin this season.

When most people talk about regression, they usually mean the strong type we see in turnover margin. To see this from a different perspective, let’s consider the wins and losses of Coach Average. His results come from flipping of a coin with an equal chance for a win and loss.

For this visual, I generated this data once with 8 lines of Python code. In no way did I search for a sequence with streaks. However, streaks almost always appear in the sequence.

In this simple experiment, the flipping of any one coin has no impact on the outcome of the next coin. The code makes each game for Coach Average independent of all other games.

Regression to the mean implies that despite a hot 8-2 start for Coach Average, he should still expect to win half of the next 10 games. In fact, he wins 6 of the next 10 games.

Coach Average also expects to regress to .500 after 9 straight losses starting on game 19. However, he loses 6 of the next 8 games.

3. Skill versus luck

Phenomena in the real world are not as simple as this coin flipping experiment, and we need to be cautious in making statements about sports. When a quantity like turnover margin has no ability to predict future turnover margin, it doesn’t imply a lack of skill in preventing or forcing turnovers.

While the analytics community doesn’t have a complete picture of turnovers, a few key insights have started to emerge. The game situations matters in turnovers. Over the past decade in college football, teams in the lead have committed a third of all turnovers. For teams ahead by a touchdown or more, the rate drops to 14% of all turnovers.

This would help explain why Alabama has posted +68 turnover margin the past 8 seasons. The Crimson Tide failed to have a positive turnover margin only in 2014, when they gave away the ball 2 more times than they took it away.

The dependence of turnovers on game situation makes sense. Teams in the lead tend to run the ball, especially later in the game. Turnovers happen at a lower rate on running than passing plays. If a team faces a deficit, they need to throw the ball to get back into the game.

For another example of how regression doesn’t necessarily imply a lack of skill, let’s turn to college basketball. Ken Pomeroy wondered how much control teams have over three point shots. He asked how a team’s 3 point shooting percentage in the first half of the conference season predicted the second half.

The visual shows his results.

The left panel shows 3 point percentage on offense. The first half of the conference season has almost no ability to predict the same quantity later in the season. Does that mean shooting is not a skill? Tell that to Steph Curry.

The visual also shows the strong regression for 3 point field goal defense. This suggests a lack of skill in defending the 3. To confirm this, Pomeroy performed a more detailed study on 3 point defense for teams over a 5 year period. He concluded there is some skill, but randomness plays a bigger role than anyone expects.

When a quantity regresses to the mean like turnover margin or 3 point shooting percentage, it doesn’t necessarily imply a lack a skill. Fumbles may regress to the mean, but that doesn’t mean a running back isn’t fumble prone if he palms the ball with one hand while running through the line of scrimmage.

However, it is safe to say that randomness plays a large role in turnovers and 3 point shots.

4. How to make preseason college football predictions

USC had high expectations heading into 2017. Sam Darnold took over the starting QB job the previous season and led the Trojans to a 9-0 finish, which included a dramatic win over Penn State in the Rose Bowl.

At the start of the 2017 season, the pollsters put USC 4th in the preseason poll (both AP and Coaches). This made Clay Helton’s team the favorite to make the College Football Playoff out of the Pac-12 conference.

In contrast, no one knew quite what to expect from Georgia. Just like USC, 2017 was their second under a young coach, Kirby Smart. But in contrast to USC, they struggled in 2016.

Georgia started true freshman Jacob Eason at quarterback, who delivered a mediocre 55% completion rate. They ranked 81st in my adjusted yards per attempt.

Georgia went 8-5 in 2016, a record acceptable only for new coaches in Athens. To start 2017, they landed at 15th in the preseason AP poll.

So what does regression say about these two teams? Each year, I put together a preseason college football model that uses regression on many variables.

In college football, team performance tends to persist from year to year. Programs like Alabama have financial resources and traditions that Rice will never have. These teams will not swap places in the college football hierarchy.

Hence, my preseason model uses the past 4 years of team performance to predict the next season. This part of the model says that team is most likely to perform as some combination of their last 4 years, with recent years weighted more. This makes the model cautious about an outlier season or 9 games.

The preseason model also considers turnover margin. Turnovers can impact the scoreboard, as a key fumble halts a game winning drive, or an interception returned for a touchdown turns a tight game into a laugher.

However, turnover margin regresses to the mean of zero from year to year in college football. Hence, the model uses turnover margin in each of the past 4 seasons. This holds back the excitement for a team that made a huge jump in rating with a +25 in turnover margin.

Last, the model considers returning starters. More experience implies better performance for college football teams.

Over the past 3 season (2015-17), this regression model for college football has predicted 69.8% of game winners. This rate doesn’t include easier to predict cupcake games with FBS teams facing inferior FCS teams. The model only makes predictions for games with two FBS teams.

Heading into 2017, the preseason college football model had USC 16th. In the previous 4 seasons, USC had never finished the season higher than 14th in my college football rankings. Despite their impressive 9-0 finish in 2016, they only rose to 14th because of a poor start.

The model doesn’t distinguish between returning starters at different positions. The quarterback has an outsized impact on a football team, and Darnold’s status as a top NFL prospect could convince you of USC as higher than 16th. However, 4th in the AP poll seemed too optimistic.

In contrast, the regression model agreed with the AP poll on Georgia. The model had the Bulldogs at 18th while the polls had them at 15th. Analytics and polls agreed on Georgia as a solid top 25 team but not a playoff contender.

During the actual 2017 season, USC did not live up to their top 4 ranking. They dropped an early road game at Washington State. Then their playoff hopes ended when Notre Dame stomped their defense for 8.4 yards per carry in a 49-14 win.

Georgia’s season got off to a distressing start as Eason sustained an injury in the first game. However, this turned into a blessing, as true freshman Jake Fromm took over and had a brilliant season. Georgia’s pass offense finished 10th in my adjusted yards per attempt.

The defense also improved, as they jumped from a solid 28th in my adjusted yards per play in 2016 to an elite 3rd in 2017. This unit had only one bad game when they allowed 40 points at Auburn. However, they atoned for this blip by holding Auburn to 7 points in the SEC title game win a few weeks later.

Georgia made the College Football Playoff, and they took Alabama to overtime in the championship game before losing.

Here’s the take home message about college football preseason predictions: It’s much easier to predict regression for a team like USC than a sudden rise for Georgia.

In each of the past 3 seasons, I’ve written about 3 overrated college football teams in the preseason polls on Football Study Hall, an SB Nation site (2015, 2016, 2017). This analysis combines my regression model with knowledge of programs.

Looking back on these predictions, I’ve been right about 7 of 9 teams. The two teams, Penn State and Oklahoma State, finished lower in the final poll than the preseason poll in 2017. However, I’ll call these predictions a push as they both performed better than I predicted.

There’s a good reason I have never written a corresponding 3 underrated teams article. A regression model is unlikely to identify breakout teams like Georgia in 2017. Their true freshman quarterback worked out, and their defense made a leap.

There is a random element to teams that make a sudden rise. These predictions are more difficult than finding an overrated team by a regression model.

5. How to make accurate predictions with regression

I want you to take home two main points.

First, quantities in football like turnover margin show very little persistence from an earlier period to a later period. These quantities regress to the mean, as your best guess for the later period is the average.

To find football teams that might not be as strong as their record suggests, look for teams with large, positive turnover margin. In contrast, teams with large, negative turnover margin might be better than their record. Use these links for data on college football and the NFL.

As another example, consider an NFL team’s record in close games from year to year. Based on data from the 2012 through 2017 season, the visual shows that one year explains 1.3% of the variance the next year.

A team like Oakland, who went 9-2 in one score games in 2016, should not expect the same good fortune again. The Raiders went 4-3 in one score games in 2017. However, regression to the mean isn’t a perfect predictor, as Cleveland has a 2-16 record in one score games the past 3 seasons.

Second, some quantities like team strength in college football tends to persist from year to year. This allows for predictive models based on linear regression.

Even with this persistence, the models still predict regression for outlier performances, both good and poor. The 9-0 stretch for USC to end 2016 serves as an example. However, regression models can not predict teams that jump from ordinary to the outlier, like Georgia in 2017.

These ideas apply for both my preseason regression model at The Power Rank and Bill Connelly’s S&P+ numbers. Use these rankings as a guide to find overrated teams near the top of the polls.

The Football Analytics Resource Guide

By Dr. Ed Feng 1 Comment

With less than 4 weeks before 2018 football season, you’d like get up to speed on football analytics. You need the insight that makes you the smartest fan on your block or message board.

What do the numbers say about skill versus luck in various aspects of football? How can coaches make better decisions during games?

To help you on your journey, I’ve complied the 9 best articles on football analytics. Instead of searching the web for resources, check out my curated list, which will show you the following:

the huge impact plays that teams have little control over (no, it’s not turnovers)
the simple decision that increases a team’s win probability that no NFL coaches make
the paradoxical reason that luck plays an increasing factor in NFL quarterback performance
the mistake that almost all college football statistics sites make
why Belichick was correct in going for it on 4th and 2 from his own 28 yard line against the Colts

For each resource, I summarize the essential aspects and then provide a link for more details. It’s a long article, but you can consume the 9 parts independently.

To check out The Football Analytics Resource Guide, click here.