Image courtesy John Bohannon
It's a rule that's as applicable to health news as it is to apartment listings: If it seems too good to be true, it usually is.
Last fall, a questionable study claiming that eating chocolate aids weight loss grabbed headlines around the world. This week, the truth came out: it was all a hoax. A hoax concocted by none other than John Bohannon, a science journalist who was hoping to expose the lax standards of some health journalists and publications for a documentary he participated in.
Bohannon really did conduct a study and it really did show that participants who ate chocolate every day lost more weight… but the study was extremely flawed, as Bohannon outlined in a confession blog post on io9. For one, the sample size was only 15 people. For another, the study was only conducted for three weeks. There were plenty of other reasons to question the study: the journal where it was published has been highly criticized, the institute promoting it is not even a real institute, "Johannes Bohannon" (Bohannon's pseudonym for the study) is not a real person. The list goes on.
Yet dozens of news outlets from the Daily Mail to Shape magazine ran with the study, most without even so much as reading the paper or contacting the authors. We caught up with Bohannon via email to chat about his stunt, why he did it, and why he thinks health and nutrition science is particularly riddled with junk science:
Motherboard: You talked about this a bit in your blog post, but what exactly were your motivations for doing this?
Bohannon: I'd say 1. an insanely enthusiastic and brilliant German film director who called me out of the blue with a fantastic plan, and 2. the diet science-media complex is corrupt.
Why did you want to do an actual (albeit flawed) study rather than just make up stats?
It's way more powerful to demonstrate the full arc, from study to headlines, to show how little quality control there is.
Were you at all surprised by how far it spread or which publications ran with it?
What, in your view, is the biggest problem facing science journalism: is it laziness? A lack of basic scientific knowledge? Both? Neither?
Full-on laziness. I mean, if you're not going to read the scientific paper at the heart of your story, at least contact an outside expert who will!
Although the story spread pretty widely, it didn't wind up in any publications that are well-known for quality science journalism. Is this really a problem facing science journalism in general or just the sort of sites that run a lot of questionable stories anyway?
I caught up the worst offenders, sure. But diet and nutrition are treated as second-class scientific topics by everyone.
Do you think the average reader knows the difference between the stuff they read on some random tumblr and a well-researched story in an actual publication? If not, does an experiment like this really do any good?
It's not that hard to see if a story (anywhere) reports the number of people in a study. Any reader can do that. They just need to know that this is important to look for.
What are a few simple ways average readers can scrutinize a science story (other than checking the number of people in a study)?
Basically, if a story seems to be giving you diet advice, don't trust it. Full stop. There is almost no scientific consensus yet. So nearly everything you read about diet and nutrition is misleading.
Why do you think diet and nutrition research are kind of disregarded? Is it an area particularly riddled with junk science?
This seems to be the worst. No idea why. Maybe because it's an area of science that *seems* so easy to fake as a reporter? I mean hey, it's just food! But really, the science is every bit as complicated as astrophysics.