On Thursday, Business Insider published the results of a study it had conducted with Foursquare with the stated intention of revealing America's most popular fast food chains, state by state. Surveys of this sort are released pretty routinely, and this one seemed innocuous in intent: if anything, its findings would let people engage in some friendly tribalism around fast food.
But something went horribly wrong.
"For the study, we looked at which chains received the most visits on average per location in every state based on the total number of visits to each chain divided by the number of locations in that state," the Business Insider article accompanying the findings explained, which essentially explains the flaws in its own methodology. Measuring "popularity" through logged Foursquare visits distorts actual consumer preference, giving a bit too much weight to chains with a few high-volume locations in certain states, as the results show.
You'll notice that Chick-fil-A just about swallows most of America. According to the map, California's most popular chain isn't In-N-Out Burger, as one may imagine, but instead Raising Cane's, which has a grand total of eight locations within the state. Meanwhile, Texans
quite fathom that their state's nominally most "popular" fast food chain was In-N-Out, which has a mere 35 locations compared to Whataburger's 669.
The survey was pilloried as "almost inconceivably bad" in execution, and Business Insider's tweet publicizing the study was deleted within hours, a move that some attributed to the furor it generated. That wasn't quite the reason why.
"We deleted the tweet because it contained a draft version of the map that was different from the one actually in the post—our Twitter version showed Chick-Fil-A in Alaska when the map in the post showed McDonald's," Business Insider's Ashley Lutz, the story's editor, wrote MUNCHIES over email Friday morning. "We deleted it as not to confuse readers."
But Business Insider told MUNCHIES that it had no hand in the survey's methodology. Foursquare provided the company with data that Business Insider subsequently used for the map. "Yup, we goofed," Foursquare's Senior Director of Communications, Dorothy Jean Chang, admitted to MUNCHIES. "We used a methodology that highlighted the popular fast food chains per location in each state, which is a different thing than popularity overall."
Chang cited the example of California, where Raising Cane's was somehow christened the most popular within the state because of the frequency of visitors logging their visits on Foursquare. "On average, in California, any given Raising Cane's is likely to have longer lines than your average In-N-Out," Chang said. "Does that make it the most popular chain in the state of California? No. But it gets the most visits per location."
Foursquare has plans to pull another version of this list with an "adjusted methodology that looks at the most popular fast-food chains in each state overall," providing more clarity and thus aligning more clearly with people's expectations. The company did not confirm to MUNCHIES, as of writing, when that list would be finalized.
Excited to see what it looks like—and whether it will lead to far fewer confused Texans and Californians, among others.