Shut Up About 2016 and Trust the (Good) Polls

Want the best of VICE News straight to your inbox? Sign up here.

Every time a high-quality poll comes out, the internet erupts with cries that it must be wrong. “But 2016!” partisans of every stripe howl as they misinterpret how and why most reporters missed President Donald Trump’s come-from-behind win and blame it solely on bad polling.

It happened again on Thursday. Twitter trolls freaked out over a bevy of new polls from the New York Times that showed Joe Biden with a huge lead over Trump in the presidential race.

That’s largely because the same pundits who misinterpreted what was happening in 2016 also misinterpreted why they were wrong — and armchair pundits have dumbed down those explanations even further to fit their own needs.

A combination of Democratic PTSD and GOP schadenfreude at media failure has taken legitimate criticism about the quality of some polling from the last presidential election and distorted it into a myth that all pollsters missed that election by a wide margin and have no idea what they’re doing.

That’s simply incorrect. And it has created a false narrative that quality polls can never be trusted, undercutting voters’ (and even some lazier pundits’) understanding of the election while allowing politicians and partisans to dismiss any numbers that might prove inconvenient.

The “don’t trust the polls” conspiracy theory is based on a myth that high-quality pollsters missed the mark in 2016’s shocking win for Trump. But here’s what really happened in 2016: There wasn’t enough quality polling at the state level in the final days of a fast-moving race, and many pollsters failed to weight their samples to adjust for education, which skewed some results too Democratic because they were counting too many college-educated voters, who unlike in past elections were a lot more likely to vote Democratic. Most good pollsters have fixed this now.

This false narrative that pollsters badly missed the mark in 2016 and can’t correctly estimate what the 2020 election will look like is pervasive enough that before releasing its latest surveys, the New York Times laid out a detailed explanation of exactly how the polls were conducted. It’s worth the read, both as an explanation of why their polls are considered some of the most reliable and to understand how polling actually works.

Both skittish Democrats and desperate Republicans have fallen back on an “ignore the polls, they’re always wrong” mantra to dismiss the current state of the race. That includes members of Trump’s own team. His campaign threatened to sue CNN in early June for a poll showing Trump down 14 points nationally, and on Thursday senior Trump adviser Jason Miller tweeted a 2016 Siena College poll that showed Clinton leading Trump by 17 points, conveniently ignoring that it was only a poll of New York state, not a national poll. Clinton wound up winning New York by 22.

It’s an effort by Trump’s team to keep supporters from getting depressed that they’re losing, which could lead to a drop in fundraising. Democrats are using the same method to try to keep voters from feeling complacent right now. And while a “run like you’re 10 points behind” mantra is smart for campaigns — races can change fast — that’s different than campaigns pretending there’s a tied race when that’s not the reality.

Here’s why you should put faith in (good) polls — but know how to read them without taking them as undisputed facts.

If the election were held today, Joe Biden would win in a landslide. Basically every good poll shows that. But there’s a long time to the election, and plenty can change.

There is a lot of bad polling out there too, especially at the state level, where it matters more. Even perfect polls will be wrong one out of every 20 times — that’s what the 95% confidence interval on polling represents. And polling is hard to do well, especially as more and more people don’t answer their phones when unknown numbers pop up.

Even given all that, polling is useful and crucial to understand what’s actually going on in an election. The politicians know that — that’s why they spend so much money on high-quality private polls to let them know where they stand and what voters care about.

There are a few things to look for to determine whether a poll is high-quality. First, while there are some good online polls, there are a ton more bad ones. A general rule of thumb is to take a skeptical view toward internet-based polling unless you can check out the source and make sure it’s one of the companies trying to do innovative work in the space rather than churn out useless polls because they’re cheaper to conduct online.

Second, live-caller polls that don’t reach cellphone users are basically useless. Most people don’t have landlines anymore.

Third, sample sizes matter a ton. If a statewide poll has a sample size under 500 people, it’s probably not very reliable, and the larger the sample, the better.

The latest NYT poll had samples of over 600 for every state poll and more than 1,300 for their national survey. Other polls routinely have 750 to 1000 for state-level polls. The bare minimum for smaller areas, like congressional districts, is a 300-person sample size. But the smaller the sample, the larger the poll’s margin of error — so if a margin of error is +/- 3 points, if a politician is at 50% in a poll, in reality they could be as high as 53% or as low as 47%.

Generally, national news outlets don’t pay for bad polls. If you want a quick answer on whether a pollster is quality, FiveThirtyEight has a whole scorecard where you can look them up.

Even good polls can be wrong, however, so it’s a lot better to look at multiple surveys than just one number. The more data, the better, and poll averages aren’t a bad way to approach things — so long as you know they’re not just an average of multiple bad polls.

The 2016 race showed how fast things can change in a campaign, especially with the perfect storm of foreign interference, a surprise FBI announcement that it was reopening an investigation into Clinton and an active effort by Russia to boost Trump and hurt Clinton.

Quality polls can show a snapshot of where a race stands, and in a presidential election with as much volatility as the 2016 race had, they did a good job overall. The final RealClearPolitics national polling average showed Clinton leading the popular vote by 3.3 points; she won by 2.1 points.

There have been plenty of smart postmortems on “what went wrong” with polls in 2016. The short answer is that pollsters didn’t weigh their samples to control for education. Historically, there wasn’t much difference between how white voters with and without college degrees voted, so a lot of pollsters didn’t adjust their samples to make sure they were counting enough of each type of voter.

But that’s been changing, and the huge gap between the groups in 2016 has only gotten larger. Because white people with college degrees are more likely to respond when called by pollsters, and are now more likely to vote Democratic, that skewed a lot of polls a few points towards Clinton.

But the much bigger problem with the media narrative that Clinton was a lock to win in 2016 wasn’t the polls — it was bad interpretations of what they meant, both by handicappers and by pundits.

The Huffington Post was particularly off the mark with a model that predicted a 98% chance that Clinton would win the election, while the Times gave Clinton an 85% chance at victory. FiveThirtyEight’s model was closer to the mark but still gave Clinton a 2-1 chance of winning.

It turns out that the models were building in way too much certainty about what numbers were showing — and that there was a lack of high-quality state polling that ended up deciding the 2016 election. The race itself moved so fast that the snapshot pollsters took a few days before the election didn’t capture the final results. And there simply weren’t enough good polls in some places to see Trump’s upset coming.

In the final three weeks of the race, Clinton’s Michigan lead dropped from 10.7 points to 3.4 points in RealClearPolitics’ polling average; her Pennsylvania lead went from 8.7 points to 1.9. With those trendlines, it’s no shock that a few days later, Trump barely caught her in both states.

As one strategist described to me at the time, the final 2016 polls showed glimpses of Clinton’s airplane plunging nose-down at the ground. The question was whether that plane would hit the ground — fall below Trump — before the race ended. (I should have listened more closely at the time, I was too bullish on Clinton’s chances too — though I did warn that Trump could win the presidency despite losing the popular vote.)

There weren’t enough pictures to see that clearly in some key states, however. In Wisconsin, for instance, there wasn’t a single public poll of the election conducted in the race’s final week, just as the wheels came off Clinton’s campaign.

Be as skeptical as you want of pundits (myself included). We’re often wrong. But rigorous polls are actually pretty damn useful.

Cover: In this combination of file photos, former Vice President Joe Biden speaks in Wilmington, Del., on March 12, 2020, left, and President Donald Trump speaks at the White House in Washington on April 5, 2020. (AP Photo, File)