Twenty-four percent of all NFL games are decided by three-points or less. If that happens this weekend at the 51st Super Bowl, all the glory (or the blame) will fall on Matt Bryant (placekicker, Atlanta Falcons) or Stephen Gostkowski (placekicker, New England Patriots). It seems reasonable to give them the credit, but in this case reason has it wrong. Giving Bryan or Gostkowski the MVP for making a crucial kick is like giving a gambler credit for the roulette wheel landing on red.In American football the team is generally a single unit, but the kicker is a unique position. Quarterbacks are the de facto leaders of the team, but a quarterback is only as good as his offensive line, receivers, and running backs. Unlike baseball or even basketball, measuring the performance of an individual player in football is notoriously difficult. Unless that player is the kicker. In that case, it’s easy.
A kicker does need teammates — a “long snapper” and a “holder” to get the ball in place and an offensive line to guard him from being blocked — but generally his job is solitary. He is given the well defined, self-contained task of kicking the ball from a set distance through the goal posts. The difficulty of his task is easy to measure: How far from the goal posts is he required to kick? And his success or failure is equally easy to measure: Did the ball go through the goal posts or not?Football placekicking is obviously a high-skilled occupation. But is luck involved too?The Falcons’ Bryant has successfully kicked 91.5% of his field goal attempts this season. The Patriots’ Gostkowski has only an 84.4% average this season, but has a better lifetime success rate than Bryant. This is Gostkowski’s first season since 2011 in which he did not lead the league kickers in scoring. On the surface, it looks like Gostkowski has the long-term edge — but Bryant is on a hot streak and might be the one to take the crown in the 2017 Super Bowl.We like to tell underdog/reigning champion stories, but what if those stories distract us from what’s really going on? What if there was no actual, measurable difference between Bryant and Gostkowski? What if I told you we could replace either of them with Roberto Aguayo of the Tampa Bay Buccaneers, who, at 71% success rate, has the worst performance in the league, and expect him to do equally well?What if who’s kicking on Sunday doesn’t matter at all?
THE HIGHEST PAID PERSON ON THE FIELD
Kickers, who earn $500,000 to $4 million per year, do not command the salaries of quarterbacks.
However, when you calculate pay in terms of amount earned per play (per “snap”), eight of the top 10 highest paid players in 2013 were kickers. Using “per-snap” metrics, the highest paid person in the league was Adam Vinatieri, kicker for the Indianapolis Colts, who earned a whopping $47,000 per kick. The highest non-kicker was Eli Manning at $21,000 per snap. Vinatieri’s coach, Tony Dungy, was known to mouth “money, money, money” when Vinatieri went out for an important kick.
No pressure, right?
Case in point: October 27, 2013, was a bad day for Shaun Suisham, then the starting kicker for the Pittsburgh Steelers. His team had just lost to the Oakland Raiders 21–18, and Suisham felt directly responsible. He had missed two field goals, which, had he made, would have given the Steelers a three-point victory.
Afterward, it was no wonder Suisham felt terrible. Other factors could have helped his team win, but nothing was as obvious as his two missed field goals. And any other errors could be shared by a lineup of guys. The missed field goals were all on Suisham’s shoulders.
In his post-game interview, Suisham looked dejected. He could barely make eye contact with the reporters and by the end he had a hard time getting words out clearly.
Suisham: “We would have won the game if I made the field goals. Tough one to swallow. No choice but to do it. I should have been better today and we lost because of it.”
Reporter: “What would you say it was?”
Suisham: “I just missed.”
Reporter: “How tough is this loss to take?”
Suisham: “We would have won the game today if I was better. You know it’s. Ah. It’s hard to take. But. You know. Swallow it. Get ready for next week.”
Watching the interview, it’s impossible not to feel empathy for Shaun. He’s a young man under an unimaginable amount of pressure and he messed up in front of a national audience that blames him (as he blames himself) for his team’s loss. Imagine if you were getting paid $40,000 to do one thing and you very clearly failed at that one thing and had no one else to blame. It was 100% your fault. Now imagine failing like that twice in a row in a way that costs your employer 10% of its annual goal, since it takes about 10 wins in a season for an NFL team to make it to the playoffs. From that perspective, Suisham took it about as well as could be expected.
Unlike playing the lottery, kicking is clearly a skill. There’s no doubt that some people — through a combination of inborn talent, temperament, and training — are better at kicking than others. If I was a kicker in the NFL, I would be the worst kicker the league had ever seen. There would be no way for me to “get lucky.” So when Suisham failed publicly in front of millions of people — after the long snapper and holder did their jobs right and set him up to succeed — he couldn’t chalk it up to luck or chance. Or could he?
WHAT IS LUCK IN A GAME OF SKILL?
How do we know how much of an activity or competition is luck versus skill? Sometimes it’s obvious: If we played a game where we each rolled dice and whoever rolled higher won, it is pretty clearly 100% luck. Perfect skill activities are harder to identify. In chess, success is close to 100% skill. All of the information is fully visible and, in theory, players can plan ahead for every possible future contingency with complete ability to make choices without outside complications. But in practice, chess players are making decisions with imperfect information. Two options may look equally attractive but a choice must be made. When a player gives up calculating and “just chooses” one of those two options, how else do you define the two different possible future results if not “luck”?
Most activities fall somewhere between high-skill chess and random-luck dice games. If it is played perfectly, blackjack is a game of luck — but the ability to play it perfectly is a skill. Poker is far more skill-based, but still involves a lot of luck in any individual hand. The hand game rock-paper-scissors seems to be pure chance — but championship RPS players (yes, there’s a world championship rock-paper-scissors contest) say there’s skill in using psychology to predict an opponent’s next choice.
It turns out, there is an objective way to measure the relative amount of luck in, say, poker versus rock-paper-scissors.
In a game of pure luck, one should expect no predictability in future performance based on previous performance. If I win four dice games in a row, I am no more likely to win the next round than you are (nor less likely — believing “luck will come around” is known as the Gambler’s Fallacy). If I beat you four times in a row in head-to-head poker tournaments, I am not guaranteed to win the next one, but an outside observer would be crazy not to give me more than a 50% chance.
This ability to predict future performance based on historical results is the key to measuring skill versus luck. If a history of success predicts future success (or a history of failure predicts future failure) then we can confidently say skill was involved. If the past does not predict the future, we can be comfortable calling the results “luck.” The better the future prediction the more skill we can claim.
HOW PAST DATA HELPS PREDICT THE FUTURE
Good data analysis is using data about the past to predict the future. If you do it right, you will have unique insight into what is going to happen: How many sales your company will make tomorrow; the chances your favorite sports team will win the championship; how much it will rain in NYC in July. If you do it wrong, your analysis may “predict” the past, but have no use in telling you what will happen tomorrow. But sometimes the best you can do is know that you can’t do it well enough to predict anything.
Breaking out skill from luck gets complicated when there are two people of similar skill competing against each other. If I were to play tennis against Roger Federer I would lose every game. One could say the results of Federer vs. Nevraumont matches were perfectly predictable and therefore the game is 100% skill. When Federer plays Andy Murray he will usually win, but sometimes (like in the 2012 Olympics) he will lose. That doesn’t mean the game isn’t highly skilled. Instead of looking at single-player matchup, we can look at all the games from all the players and use the data to predict the likelihood that a given player will win a given match against a given opponent based on skill ratings we give them modeled on their historical performance. It turns out that when you do this, you can predict tennis matches with a degree of accuracy when player ratings are far apart — but much less so when ratings are close. One could say that tennis is a very high-skill game, but when two players are evenly matched it comes down to luck.
How does this help us with Shaun Suisham’s kicking in the NFL?
It is relatively easy to model kicking performance. We can look at each kicker and know whether they succeeded or failed on each kick attempt. And we know the level of difficulty based on how many yards he had to kick. The NFL keeps detailed records of every play, so we can go back to see kicker performance over time.
The first thing you’ll see is that kickers have been getting better. The website FiveThirtyEight ran the numbers to look at field goal success rate in the NFL by year going back to 1960.
But this doesn’t dig into individual performance. For that we need to look at how well Suisham’s 2008 performance predicts his 2009 performance. More accurately, we should look at all the kickers in 2008 and see how correlated their field goal success rate was to that of the following year. To get an even better result, we could look at all the kickers from, say, 2008 to 2015, and compare their first appearance in that time period to their second appearance.
If we see a high correlation, we can assume the job has a high skill component and that we can be confident at predicting next year’s performance based on how they did in the past. If the correlation is low, that doesn’t mean the job is not skilled, but that when one gets to the level of the NFL, the skill difference between kickers is very small — like the tennis players who compete against one another.
The higher the correlation, the higher the relative skill gap between the individual kickers. The lower the correlation, the more we cannot differentiate which kicker is actually better than any other.
So what do we find?
Each dot represents an individual kicker. The x-axis is the percent of field goals the kicker successfully achieved in his first year as a kicker (from 2008–2015). The y-axis is the percent of field goals each kicker achieved in the second year as a kicker in that time period.If there was a high correlation, we would see all of the dots lined up close together from the bottom left to the top right. Instead what we see is a big blob.
Four kickers had 100% field goal success rates in their first year (mostly due to low numbers of attempts). Of those four, three were just average (79–84% success rate) in their second year and one was terrible (69%). Meanwhile, the worst-performing kicker who was asked to return had a 69% field goal success rate in his first year. But in his second year, he was almost in the top third of performers, at 86%.
This huge fluctuation is everywhere in the data. Ignoring the 100%-ers, the kickers with field goal rates of 90% or higher in year one had success rates in year two ranging from a consistent 94% to a league-low 63%. Meanwhile, the third worst kicker in his first season was the third best in his second season.
No one is claiming that kickers are not highly skilled. But what is obvious from this data is that we have no way of knowing which kickers in the NFL are better than the others. No matter how well or badly a kicker performs in a season, it doesn’t help us predict how well he will do in the following season.
In another interview, Suisham said, “With my job it’s very easy for anyone to evaluate how I did. Same for myself. Nothing to hide behind. Nothing to run from. It is what it is. And I own it.”
Suisham, and other NFL kickers, need to think everything is under their control. Without that core belief it is hard to imagine they would ever become the world-class athletes they are. But once at that exceptional level, things change.
If Bryant or Gostkowski make an amazing kick on Sunday they will be in high demand across the league and could commend tremendous salaries during their next negotiation. If they miss a clutch kick they will be the target of half the country’s scorn. They don’t deserve either outcome. Once you get to their level of ability, what we think of as skill turns into a weird sort of luck. And athletes that are at the pinnacle of their profession turn into commodities.
So if they miss a kick, don’t be too hard on them. And if they make a kick, don’t bother with the raise. I’m sure Roberto Aguayo would be willing to replace them at half the price. And he has a 50/50 shot of out kicking either one of them next season.
Want to learn more about how to predict the future by analyzing data from the past? Discover the predictive power of data science and transform your career. Learn on campus with General Assembly’s full-time or part-time programs, on campus or online.