Evaluating a wisdom of crowds prediction for the 2017 NFL draft

Last season, I made a wisdom of crowds predictor for the 2017 NFL by combining the predictions of 15 different mock drafts.

If a mock draft predicted a player to go first, the player earned 32 points. If a mock draft predicted a player to go second, the player earned 31 points, and so on down to 1 point for the last pick of the first round.

The players got ranked by points to make the wisdom of crowds prediction. You can get the full list of predictions here.

How did the wisdom of crowds predictor do compared to the individual mock drafts?

Error for each NFL draft prediction

To evaluate each draft predictor, I evaluated the distance from predicted to actual draft spot for each player. For example, Mitch Trubisky went 2nd in the draft, so the wisdom of crowds prediction of 8th has a distance of 6. Jonathan Allen went 17th despite a prediction of 3rd, which gives a still positive distance of 14. The total distance for all 32 predicted players gives an error for a draft prediction.

I tracked each player compared to his actual draft slot for the first two rounds. In a few cases, a predicted player got drafted outside these top two rounds. To evaluate the distance, I assigned draft slot of 65 (first pick of the third round), plus a penalty of 3. Only 7 of the mock drafts had a prediction outside the first two rounds, and they are denoted by a * in the list below.

Here are how the 15 mock drafts stacked up compared to wisdom of crowds.

1. Sporting News, 222
2. The Power Rank Wisdom of Crowds, 239
3. Todd McShay, 249
4. CBS, 259*
5. Daniel Jeremiah, 263
6. Mel Kiper, 265
7. Drafttek, 266*
8. Lance Zierlein, 277
9. Charlie Campbell, 290
10. Sports Illustrated, 291*
11. SB Nation, 295
12. Fox, 306
13. Newsday, 310*
14. Cris Collinsworth, 344*
15. Walter Cherepinsky, 405*
16. Pro Football Focus, 431*

I took this data on April 19, 2017. The predictions may have changed for some of these mock drafts as authors made updates before the first round of the NFL draft on April 27, 2017.

The wisdom of crowds predictor did better than any constituent mock draft except for one: Vinnie Iyer at Sporting News.

In future years, I plan to use the full draft to evaluate the mock drafts, not only the first two rounds with an arbitrary penalty for picking a player outside of these rounds. In most cases, this better method would make the error larger for the mock drafts with a *.