The anatomy of a 4th-down decision
|
|
Patriots coach Bill Belichick is seen Wednesday at his team's facility in Foxborough, Mass. AP PHOTO |
Toolbox
By FRANK FRIGO The New York Times - Published: November 27, 2009
As New England Patriots coach Bill Belichick labored over his infamous fourth-and-2 decision against the Indianapolis Colts on Nov. 15, little did he know that he was about to set off one of the season's great debates.
Of course, if Belichick had succeeded — if the Patriots had gotten a first down from their 28-yard line with two minutes remaining while nursing a 6-point lead — the postgame discussion would be quite different. When a choice results in success, we attribute brilliance. When it fails? That is another matter altogether.
Zeus, a computer program capable of simulating hundreds of thousands of NFL outcomes in seconds, recently weighed in on this controversy and landed on the side of Belichick, as did several other statisticians and quantitative analysts.
After our findings were published by The New York Times, letters poured in, representing a broad spectrum of opinions. The onslaught of sports commentary on television and in the blogosphere is testimony to more than just the American public's obsession with football. It is also an extraordinary insight into how we evaluate complex decisions.
It quickly became apparent that there was a gross misunderstanding of decision theory, particularly among the most vocal opponents of Belichick's strategy. If we can leave emotion on the sideline for a moment, objectivity and computing power can get to the core problem.
It is important to understand the criteria for a correct decision: The best choice typically provides the highest expectation of achieving one's goal. In football the goal should be to win. This might sound obvious, but coaching decisions are often clouded by other factors. How often do we see a team punt on fourth down when trailing by a wide margin? In such situations, it seems that losing gracefully is more important than the slight increase in the chance of winning associated with trying to get a first down.
Expectation should not be confused with guarantee. In science, as in life, there is no such thing as a 100 percent certainty. Any physicist will tell you that all scientific facts are provisional. All reasonable people reject the notion that the earth is flat, but in theory there is not a 100 percent certainty. As the evolutionary theorist Stephen Jay Gould said, a fact can only mean something confirmed so thoroughly that "it would be perverse to withhold provisional assent." When choosing between alternatives, we can only ask which is more likely to be correct, and which supporting argument is more plausible.
In theory, there are only three ways to accurately assess a fourth-down decision:
Find a large number of historical cases that exactly match the problem.
Ask the opposing teams to line up and play the game to conclusion thousands of times, beginning with the play in question.
Precisely replicate the outcome in a virtual environment and set the number of simulations to a statistically significant level.
The first two options are absurd; the third can be realistically attempted.
Consider a situation in which a team is trailing by two points on the final play of the game at the opponent's 2-yard line. Every coach in the NFL would kick a field goal and would be absolutely correct to do so. An average team might expect to win 99 percent of the time when it kicks, and 45 to 50 percent when it goes for a touchdown.
Unfortunately, most fourth-down situations are not so easy to handle. There is more time on the clock, a different score, a different ball position and other considerations. This is where computing becomes so important. Zeus is not a black box with a mysterious opinion. It simply can process more data, more quickly than a human. Its criteria are no different than a coach's criteria for choosing the kick on the last play of the game. That decision tree had only two branches. When the tree has thousands of branches, humans become incapable and machines become reliable. The argument that a computer will never take into account every subtlety of the game is a valid one. No computer will ever perfectly predict the outcome of a game played by humans, but that is not what we are after. When looking at a difficult fourth-down decision, we are trying to come up with only the optimum choice, a choice that often only has two candidates. Zeus may say the Patriots' pass created a win probability of 77.3 percent and a punt would have garnered only 75.7 percent. The essential information revealed in those numbers is in the difference, not the absolute. The coach is only trying to choose the correct branch on the tree, the one that is most likely to get the most victories. When making a binary decision, some reliable information is better than none.
In his postgame news conference, Belichick defended his decision.
He did not know his exact probability of getting a first down. He could not have known how often Manning could lead the Colts to a touchdown after a good punt. But if pressed for an answer, Belichick could certainly provide a range of probabilities for each of those branches. Members of his staff could tell him what the team's historical average would be of converting fourth-and-2 or even third-and-2. They may not produce precise information for a specific opponent in an exact situation, but most assuredly they could provide a correlated reference point.
When Zeus analyzes this problem, it does so in a two-part process. First, we determine the relative expectations of the play choices with our best possible assumptions about the team's relative strengths and weaknesses. The number of simulations is set to a high enough level to dismiss any random variation in the results. Then the model, in a sense, argues with itself.
Skeptics may say the pass was not 1.6 percent better than the punt because we did not accurately account for the Patriots' offense or defense, or the Colts' offense or defense. In fact, we not only account for them, we account beyond them. The custom characteristics of the Patriots and the Colts can be adjusted to the point at which Zeus' recommendation is finally overturned. In other words, we alter the offensive and defensive inputs to such a level that the punt becomes the better choice.
In this situation, the required adjustment to overturn the decision was so far beyond the reasonable characteristics of the Patriots and the Colts, you could say it would be perverse to withhold provisional assent. Is it safe to say it is far more plausible that the pass was more correct than a punt would have been? Absolutely. Are we 100 percent certain it was correct to go for it?
Ask the physicist.
42