The Tennessee Titans were an indisputably mediocre football team in 2013 for myriad reasons, one of which may or may not have been third-year head coach Mike Munchak. After starting out 3-1 and then falling to 5-9 in mid-December, the Titans won their last two games but ended with their second consecutive losing season under Munchak. And that was enough to raise the strong possibility of him losing his job.
Six years earlier, in 2007, the Denver Broncos suffered through an equally mediocre season under coach Mike Shanahan. They lost two of their last three games to also finish 7-9, raising questions as to whether Shanahan's tenth season in Denver would be his last.
Two teams, facing a crossroads. Best as can be discerned by Skidmore College assistant mathematics professor Mike Lopez, each franchise, at that moment, could be assigned a roughly 25 percent chance of firing its coach.
Following that 2013 season, the Titans did just that, firing Munchak and opting for a fresh face and approach with Ken Whisenhunt. The Broncos, on the other hand, chose to keep Shanahan.
So did either move make a difference?
In this example—as is frequently true, according to Lopez's research—firing a coach didn't do much of anything. In this example, in fact, firing a coach actually made things worse than its comparative situation, in which the incumbent coach was retained. While the 2008 Broncos went 8-8 under Shanahan (admittedly leading to his firing after that season), the 2014 Titans under Whisenhunt went 2-14. And Whisenhunt was then fired seven games into the 2015 season, thereby perpetuating a carousel of hirings and firings that has come to define the NFL over the course of the first month of every new year.
This season, six franchises fired their coaches (and a seventh coach, Denver's Gary Kubiak, retired). What difference does it make? Quite often, Lopez has concluded, this annual show of power by owners and general managers doesn't really accomplish much of anything in the immediate future. Instead, it's more about appeasement than improvement.
According to Lopez's study of teams that have hired and fired coaches since 1983, teams that kept their coach rather than choosing to fire him had a 3.7 percent higher winning percentage the following season. That, according to Lopez's estimates, equals roughly 0.6 more wins per season for those teams.
"All these teams in the NFL are making franchise-altering decisions," Lopez told VICE Sports, "and it's not something that people have looked at extensively—whether these decisions aggregated across all teams are actually the right choice."
Drawing on data from a study by some student members of the Harvard Sports Analytics Club and their advisor, Lopez constructed a model that would attempt to measure whether teams might be overreacting. Using factors such as each team's past win percentage, divisional win percentage, coaches' experience, strength of schedule, and whether or not the team also changed general managers, among other things, Lopez was able to examine comparatively equal cases like the 2007 Broncos and the 2013 Titans—in which one team chose to fire their coach, and one team didn't.
What Lopez found was not entirely surprising, but it did reinforce what he and many other sports analytics types already presumed, which is that pro football so often tends to disregard hard data in favor of more esoteric factors. For instance: Munchak was fired in large part because he refused to fire several of his own assistant coaches.
"I think it's incredible," Lopez said. "You look at major tech companies like Amazon and Facebook and Yahoo, and every decision they make is based on data. And it just strikes me as insane that so few football decisions are made based on data. They're all sort of based on gut and feel and things that are little bit less technical and prone to human bias. I think this is one where general managers overreact. Things can go wrong and it's not necessarily the coach's fault."
Lopez is willing to admit that the factors involved are perhaps more complicated than his study could measure. As much as he tried to account for every possible variable, he could only measure out the impact of a coaching change (or a non-change) one year after the fact. The long-term impact is much more complex to assess; in addition, those outside factors that aren't quite measurable by numbers—what statisticians refer to as "unmeasured confounding," and whatever might lead countless message-board posters to call for a coach's demise whether it makes rational sense or not—are also difficult to factor in.
"Maybe there's something else they have organizationally that we can't pick up on—I'm just looking at the results on the field," Lopez says. "[We can't measure] the other results in terms of satisfying stakeholders and the people buying the box seats. If they're dissatisfied, maybe you have to suck it up and do it even if you know it's not going to help you."
But the general sense Lopez got is that maybe teams do resort to firings too often, and too reflexively. A study using a different method by ESPN's Brian Burke found essentially the same results, as did a study of English Premier League soccer coaches. There is an astounding cycle of risk aversion that guides so much of the decision-making in the NFL. Coaches don't go for it on fourth down or take risks as much as they should, according to the numbers, because they are afraid of getting fired. And general managers and owners seemingly fire their coaches far more often than they should, presuming Lopez's analysis is correct, because it's less of a risk to keep an unpopular coach than it is to make a change, thereby blaming the fired coach for any larger failures within the organization—or for declining to take the risks that may have saved his job.
There is a prime example of such an organization right now, of a team whose owner largely refuses to recognize that a series of coaching changes might not make a difference if the franchise is itself a mess. So maybe it doesn't matter who the San Francisco 49ers hire to be their fourth coach in four seasons. Maybe the lesson of Lopez's study and others like it is that change can often get you nowhere.
As Lopez said, "The 49ers are sort of figuring out when you fire a coach it doesn't necessarily mean you're going to improve."
Want to read more stories like this from VICE Sports? Subscribe to our daily newsletter.