Technologists Are Using AI to ‘Expand’ Famous Works of Art

Artists have been creating all sorts of surreal compositions using OpenAI’s DALL-E, the AI tool that generates images from a few words of descriptive text. But some AI enthusiasts have been using it for a slightly different purpose: to “expand” classic works of art by using the machine learning model to imagine what they would look like if the canvas were larger or zoomed out.

“It's like your phone keyboard trying to guess what the next word you would like to input and suggests it,” Denis Shiryaev, a YouTuber and CEO of the AI company neural.love, told Motherboard. “The same idea works with the image ‘prompt’—AI tries to finalize the image based on the source pixels provided, and the text prompt helps to manipulate that generating process.”

Shiryaev says his experiment began during the initial rollout of DALL-E 2 as an attempt to recreate the unfinished painting of George Washington that famously sits on the dollar bill. With help from a Telegram user who has access to the closed test build of DALL-E, Shiryaev then began feeding the model “masks” derived from famous works of art—which tell the algorithm which areas of the image to expand—along with tightly-worded text prompts. 

The results mimic a camera zooming back on the subject, as the model fills in parts of the background that the artist never visualized. So far, the pair have expanded the canvas edges of classic paintings like the Mona Lisa, the medieval-era Christ Pantocrator of Sinai, and Vermeer’s “Girl with a Pearl Earring.” 

It’s not the first experiment using machine learning to play around with famous works of art. Last year, Amsterdam’s Rijksmuseum used AI to digitally restore a Rembrandt painting that was trimmed in 1715 to fit inside a doorway. Other researchers recovered a trio of paintings by Gustav Klimt that were lost to Nazi looting during World War 2. DALL-E has also been used to completely reimagine classic paintings by having the model reproduce them in different styles. 

“From the perspective of art, those results do not give us much, I believe,” said Shiryaev. “But for me personally, it was an enjoyable journey to experience the same familiar art, but from unexpected angles.”