Fox Is Using Google's Machine Learning to Predict What Movies You'll Like

Data scientists at 20th Century Fox and Google Cloud have developed machine-learning software that can analyze movie trailers and predict how likely people are to see those movies in theaters.

A recent preprint research paper breaks down how the program, named Merlin, can now recognize objects and patterns in a trailer to understand movie scenes. Merlin can scan trailers and spot objects like "man with beard," "gun," "car," and decide whether the movie is an action flick or a crime drama based on the context in which those objects appear.

"A trailer with a long close-up shot of a character is more likely for a drama movie," the study's authors write, "whereas a trailer with quick but frequent shots is more likely for an action movie."

Merlin can use its knowledge of common tropes in trailers to understand how sequences of actions in trailers play into our expectations for genre films. For example, Merlin knows that cars going fast may break into a car chase "followed by a car flipping, and a car explosion." A car chase with explosions is very likely in an action movie, so Merlin can use that information to tag the movie with "action" and "car chase," and in turn use that to recommend other movies with car chases.

Merlin compares these tags to a large dataset that includes hundreds of movies and millions of attendance records. Fox and Google claim that the data is "fully anonymized" and "user privacy-complaint," but it’s not clear exactly what data is included or how it is gathered. According to the paper describing Merlin, the system pairs attendance records with “basic demographics information” at the individual level.

Merlin uses its method of categorizing movies and historical data of a moviegoer's preferences to predict whether they will buy a ticket for the next big summer action movie, Man With Beard And Gun And Car.

Beginning with 2017's The Greatest Showman, 20th Century Fox has been using Merlin's predictions to decide which movies to make and how best to market them, according to a Google blog post.

Here's the thing, though: art doesn't work like that. There are many factors that can make a movie successful that can't be identified by a computer, even one that can correctly identify a beard. Movies have silent performances, jokes, and harder-to-quantify intangibles that even real humans struggle to explain. This is why good movie criticism is compelling: there are infinite ways to analyze what movies do, how, and whether that is "good" or "bad."

To see Merlin’s limitations, we can look a its analysis of Logan, the 2017 superhero western from director James Mangold, which Google used as a test-case in its blog post. Merlin watched the trailer for Logan and tagged every object it recognized, like "vehicle," "car," "man," "facial_hair," and, most frequently, "tree."

According to Merlin, if you saw Logan, you'll most likely pay to see The Magnificent Seven, Jason Bourne, John Wick 2, and The Legend of Tarzan. It's easy to see how "man," "beard," and "gun" would draw recommendations for The Magnificent Seven and John Wick 2, but I suspect Tarzan was chosen mostly on the strength of "tree." Out of the top five movies that real audiences saw prior to Logan, Merlin only got one ( John Wick 2) correct. Jason Bourne and Tarzan weren’t even in the top 20.

I'm not sure that Merlin has really captured the essence—or even the appeal—of Hugh Jackman's haunting portrayal of old-man Logan, an aging but violent cowboy struggling to live in an increasingly sterilized frontier.

Looking at the results shows that Merlin predicted 11 of the top 20 likely movies, which certainly isn't a total miss. From 20th Century Fox's point of view, their software correctly predicted more than 50 percent of movies that Logan viewers also went to see. Having even a fuzzy frame of reference based on data would be a useful thing when it comes to deciding on advertising budgets or deciding how to market upcoming films.

But just because they're based on data doesn't mean that Merlin's predictions alone are worth betting money on. It would have been foolish of Fox to follow some of Merlin's suggestion and, for example, advertise Logan to fans of The Hunger Games.

These results also show how easy it is for humans to culturally understand things that machines can't (yet). The top-five most common movies for people who saw Logan were, naturally, comic-book movies like X-Men: Apocalypse, Doctor Strange, and Batman V Superman. But Merlin doesn't appear to recognize the cultural context around comic book movies or the X-Men characters. It also doesn't recognize audiences' exhaustion with a decades-long flood of superhero movies—exhaustion that made both Logan and Deadpool popular with the same long-suffering audience.

Merlin's accurate prediction of John Wick 2 is worth pointing out, though, as a sign of how accurate Merlin might become in the future. Unlike all of the other comic book movies that filled out the top five, John Wick isn't a famous character or a superhero. Still, Merlin was able to spot enough of a thematic similarity to predict that fans of John Wick 2 would be fans of Logan. As Google put it in its blog post, "This result is a win-win because the [John Wick] audience 'adds' to the core superhero audience, and can potentially be used to extend the reach of the movie beyond that core audience."

Whatever its potential, It's depressing to imagine that movie studios are already using tools like Merlin to perfect what many of them already do: trying to recreate the success of movies that are already successful. “Historically, movie studios have relied heavily on experience when deciding to invest in a particular script—but this can lead to huge risks, particularly when investing in new, original stories,” the Google blog explaining Merlin states.

And god help us if this approach ever becomes standard or becomes the sole metric that drives which movies get made. Avoiding big risks is how we end up with summers crammed full of sequels, reboots, and attempts to launch "franchises" or "universes" just because that's how Marvel and Disney are making millions of dollars. Relying solely on data produced by tools like Merlin is how you create an industry where every car gets into a chase, then flips over and explodes.

Merlin doesn't take into account all of the other things that can affect box office attendance, like parental groups leading a boycott; decades of Hollywood's racial baggage; politics; and accusations of sexual assault. Sure, it's cool that a computer program can recognize that "gun fighting sequences may have a high correlation with explosions," but that's doesn't guarantee a computer program can spot a successful movie, let alone a good one.