Valve's "Solution" to Review Bombing Ignores Steam's Longstanding Problems

Despite their influence over the PC gaming industry, Valve is not a company known for making knee jerk reactions, even under pressure. They move slowly, quietly, frustratingly. Staying in character, the company today announced a change to user reviews on Steam, in response to the now-popularized concept of "review bombing." Instead of tackling the platform's system problems of moderation and toxic, empowered users, Valve's solution is to rely on more data, by allowing review bombs to exist, brightly highlighted in a fancy graph.

Steam is an incredibly useful service that I use every single day, but as a company, Valve has made their philosophy crystal clear: 1) Assume the presence of More Data can fix human problems and 2) Assume consumers are so attentive and aware that they'll go digging for context, if only they had more data.

Marketplace consolidation has many consequences, including granting consumers the chance to weaponize those platforms. One of the most regular tactics is expressing displeasure at what a creator says or does outside that platform by writing negative reviews, aka "review bombing." By concentrating a blast of negative reviews at once, it's able to bring down the overall review score. If it succeeds, the review bomb drags the game from aggregate positive to aggregate negative, potentially influencing someone who stumbles upon the game and merely looks at the review score. This is common on storefronts like Amazon and Steam, and recently, Valve's portal was subject to two high-profile incidents.

When former Half-Life writer Marc Laidlaw secretly released a summary of what Half-Life 2: Episode 3 could have been, unhappy fans targeted Dota 2 with negative reviews. And when Firewatch developer Campo Santo issued a DMCA takedown request to PewDiePie's video of their game in response to the YouTube personality uttering the N-word on a stream, Firewatch was hit with thousands of reviews suddenly disgusted with a game they'd bought.

(On Steam, you cannot review a game without purchasing it and playing it for 20 minutes, but it is possible to buy it, play it for 20 minutes, write a review, and then request a refund.)

"We decided not to change the ways that players can review games," said Valve product designer Alden Kroll, "and instead focused on how potential purchasers can explore the review data."

Valve's solution is a new set of data for users to parse, a graph that tracks the history of user reviews for a game. In Valve's eyes, this will allow people to visualize whether a negative review score is an anomaly (a review bombing) or truly indicative of a game's quality.

"As a potential purchaser," said Kroll, "it's easy to spot temporary distortions in the reviews, to investigate why that distortion occurred, and decide for yourself whether it's something you care about. This approach has the advantage of never preventing anyone from submitting a review, but does require slightly more effort on the part of potential purchasers."

What the graph doesn't do, unfortunately, is explain why the review bomb occurred in the first place. For this, Valve expects users to do their own homework and research what caused a spike in one direction or another. (Valve claims that, in aggregate, their data shows review bombs for quality games disappear, with user reviews reverting to their original form.)

Look at Firewatch's review graph, for example:

For one, the graph doesn't show up on a game's Steam page by default. You need to consciously click "show graph," which means most people aren't ever going to see this.

Two, the idea that a Steam user will take time out of their day to become an amateur reporter and discover the mystery behind a spike in negative user reviews is horribly naive. Even as a trained reporter, I often find myself relying on the snap judgement usefulness of an average review score on Steam to help me understand if a game is worth checking out. Review bombing relies on casual laziness to be effective; at the time the review bomb sticks, some number of users will rely on snap judgement usefulness to walk away from a game.

As a platform holder, it's Valve's job to make their service more useful.

Even though a casual search of Firewatch on Google would instasntly produce a bunch of articles about what's happened in the past few weeks, there's nothing on Steam itself that points you in that direction. Did a patch break the game's balance, introduce frame rate issue? Was a recent piece of downloadable content confusing? Steam could add some form of usefulness to this graph by linking highly shared articles about the game to the graph, providing context.

The reaction from developers I've talked to so far hasn't been positive.

"Seems like a copout," said Double Fine community manager James Spafford on Twitter, "putting the onus onto the new customer to figure out if the mob are angrily spamming or not. They're basically saying, 'We have all this data and can detect review bombs, here it's yours now, enjoy!' Washing their hands of the issue and walking off into the sunset uncaringly."

Perhaps more worrying is how the visualization of a hate mob's successes and failures might incentivize others to engage in similar activities. While the removal of review bombs might not outright stop groups from trying again, by making such activities part of the historical record, it adds legitimacy that's otherwise robbed by erasing toxic attempts to muddy history.

"Not entirely sure about how 'give review bombs more exposure, a permanent record, and gamify it with a graph they can min/max' fixes things," added No Goblin (100 ft. Robot Golf) designer Dan Teasdale on Twitter. "After looking at it in-store, it feels even worse. Putting up a big yellow banner saying "CHECK OUT THIS REVIEW BOMB!" isn't positive."

"These tools will probably help review bombers understand more clearly how effective or ineffective their criticism is," said PC Gamer editor-in-chief Evan Lahti on Twitter. "It's better feedback. Think about recent uptick in players using concurrency stats as "evidence" - rightly or wrongly, we'll see graphs cropped + shared as ammo."

In Kroll's blog about Valve's decision, he explained decided against removing review scores entirely ("Demand for a summary of some kind is likely to still be there, even if players know it isn't always accurate"), temporarily locking review score submissions ("We didn't like the way this ultimately meant restricting the ability for players to voice their opinions"), or changing the range of time a review score is calculated ("Doing this would likely result in more fluctuation and potentially less accuracy for all games, not just review bombed ones").

Valve's conclusion is reasonable, perhaps, but fails to address a longstanding critique of the platform writ large: Valve's misguided perception that the solution to everything is more data.

Valve is a private company, and therefore doesn't disclose how much it makes every year, but estimates put Steam's yearly profits at nearly one billion per year. Despite this enormous surplus, Valve continues to operate Steam as though it's a thrifty mom 'n pop shop unable to invest substantive resources into improving quality of life on its service for developers and users alike. The solution to review bombing isn't a graph, it's a resource-rich moderation staff paying close attention to influxes of user reviews, addressing them on a case-by-case basis.

In writing about Artifact, Valve's new trading card game based on Dota 2, writer Will Partin summarized the company's disappointing but effective tradition of distancing from the consequences of the products it creates, instead putting that weight on the users themselves:

The graph isn't useless, but it's a half-hearted solution to a problem that underscores larger, system issues Steam has faced, as it's continued to get bigger. These are issues Valve has done very little to address over the years, and this latest "fix" only highlights the disconnect.

Maybe just hire some people who can tell when people are being assholes?

Follow Patrick on Twitter. If you have a tip or a story idea, drop him an email here.

Have thoughts? Swing by Waypoints forums to share them!