How to successfully moderate user-generated content is one of the most labor-intensive and mind-bogglingly complex logistical problems Facebook has ever tried to solve. Its two billion users make billions of posts per day in more than a hundred languages, and Facebook’s human content moderators are asked to review more than 10 million potentially rule-breaking posts per week. Facebook aims to do this with an error rate of less than one percent, and seeks to review all user-reported content within 24 hours.Facebook is still making tens of thousands of moderation errors per day, based on its own targets. And while Facebook moderators apply the company's rules correctly the vast majority of the time, users, politicians, and governments rarely agree on the rules in the first place. The issue, experts say, is that Facebook's audience is now so large and so diverse that it's nearly impossible to govern every possible interaction on the site.
"If you say, ‘Why doesn’t it say use your judgment?’ We A/B tested that."
"There was basically this Word document with ‘Hitler is bad and so is not wearing pants.’”
According to several other former early Facebook employees, the discussions inside the company at the time centered on determining and protecting Facebook’s broader ideology, mission, and ethos rather than taking specific stands on, say, hate speech, harassment, or violence. Motherboard granted these former employees anonymity because they expressed concern about professional repercussions.
"We want to have one global set of policies, so that people can interact across borders"
A training questionnaire offers some examples: A photo of Taylor Swift with anus eyes: OK. A photo of Donald Trump with an anus mouth: OK. A photo of Kim Jong Un with an anus mouth and anal beads being removed from it: Not OK.
The process of refining policies to reflect humans organically developing memes or slurs may never end. Facebook is constantly updating its internal moderation guidelines, and has pushed some—but not all—of those changes to its public rules. Whenever Facebook identifies one edge case and adds extra caveats to its internal moderation guidelines, another new one appears and slips through the net.One hate speech presentation obtained by Motherboard has a list of all the recent changes to the slide deck, including additions, removals, and clarifications of certain topics. In some months, Facebook pushed changes to the hate speech training document several times within a window of just a few days. In all, Facebook tweaked the material over 20 times in a five month period. Some policy changes are slight enough to not require any sort of retraining, but other, more nuanced changes need moderators to be retrained on that point. Some individual presentations obtained by Motherboard stretch into the hundreds of slides, stepping through examples and bullet points on why particular pieces of content should be removed.
"What many of these firms fear is that their primary business could very easily shift to just being the security forces of the internet, 24/7"
If the reported content contains multiple violations—as it often will, with speech being as complicated as it is—then moderators have to follow a hierarchy that explains which policy to delete the content under, a second moderator explained. One of the sources said that this can slow down the removal process, as workers spend time trying to figure out the exact reason why the content should be deleted.
"We still don't know if [language processing in Burmese] is really going to work out, due to the language challenges"
And so Facebook continued to push into new territories, and it is still struggling with content moderation in some of them. Earlier this year, Facebook was blamed for helping to facilitate genocide in Myanmar because it has allowed rumors and hate speech to proliferate on the platform. As Reuters reported, this is in part because, until recently, Facebook had few moderators who spoke Burmese. Rosen, Facebook’s head of product, told Motherboard the company’s hate speech-detecting AI hasn’t yet figured out Burmese, which, because of Myanmar’s isolation, is encoded by computers differently than other languages. Facebook launched in Myanmar in 2011.“We still don't know if it's really going to work out, due to the language challenges,” Rosen said. “Burmese wasn't in Unicode for a long time, and so they developed their own local font, as they opened up, that is not compatible with Unicode. … We're working with local civil society to get their help flagging problematic content posts, and, current events also, so that we can go and take them down, but also understanding how we can advance, especially given our app is extremely popular, the state of Unicode in the country.”Osofsky says that his team, Bickert’s policy team, and the product team often does “incident reviews” in which they debrief on “painful mistakes” that were made in policy moderation and to determine how they can do better next time.“Was our policy wrong or right? If our policy was ok, was the mistake in how we enforced the policy?” Osofsky said. “Was the mistake due to people making mistakes, like reviewers getting it wrong, or was it due to the tools not being good enough, they couldn't surface it?"Everyone Motherboard spoke to at Facebook has internalized the fact that perfection is impossible, and that the job can often be heartbreaking.“I feel an immense responsibility to ensure that people are having a safe, good experience on Facebook,” Osofsky said. “And then, when you look into mistakes we've made, whether it's showing an image that violates our policies and should have been taken down, whether it's not responding sensitively enough in an intimate moment … these are moments that really matter to people's lives, and I think I and my team feel a responsibility to get it right every time. And when you don't, it is painful to reflect on why, and then to fix it.”The people inside Facebook’s everything machine will never be able to predict the “everything” that their fellow humans will put inside it. But the intractable problem is Facebook itself: If the mission remains to connect “everyone,” then Facebook will never solve its content moderation problem.Facebook’s public relations fires are a symptom, then, of an infrastructural nightmare that threatens an ever-increasing swath of humanity’s public expression. For Facebook, it is a problem that could consume the company itself.“What many of these firms fear is that their primary business could very easily shift to just being the security forces of the internet, 24/7,” Roberts, the UCLA professor, said. “That could be all Facebook really does.”And that’s why, though the people who were invited to Mark Zuckerberg’s house to talk about content moderation appreciated the gesture, many of them left feeling as though it was an act of self-preservation, not a genuine attempt to change Facebook’s goal of making one, global community on the platform. (Facebook confirmed that the dinners happened, and said that Rosen and Bickert were also in attendance to share the feedback with the product and policy teams.)Everyone in the academic world knows that Zuckerberg is talking to professors about the topic, but he hasn’t made clear that he’s actually doing anything with those conversations other than signaling that he’s talking to the right people.“Everyone I’ve talked to who comes out of it said ‘I don’t really think he was listening,’” a person familiar with the dinners told Motherboard. “It feels like by inviting all these people in, they’re trying to drive the problem away from them.”
"I feel an immense responsibility to ensure that people are having a safe, good experience on Facebook"
Got a tip? You can contact Joseph Cox securely on Signal on +44 20 8133 5190, OTR chat on email@example.com, or email firstname.lastname@example.org.