The best television characters have already been immortalized on screen, given reruns and our ability to stream our favorite episodes on demand. Now early research at University of Leeds is attempting to give these characters new life, by building digital recreations that can appear in new episodes or even take over for Siri on your smartphone.
Their first “immortal” character: Joey from Friends.
The Leeds team is building a nearly-automated pipeline for capturing the digital essence of a TV character. A series of algorithms mutes the laugh track on a TV show, tracks the faces of each character, and analyzes their voices and body language, as well as the way their mouths move as they speak. The machine learning tool also scours the show’s script in order to learn dialogue and understand how each character strings together sentences.
TV is a natural fit for this kind of research, because there’s so much video from which to learn. The Leeds algorithm was trained on more than 200 episodes of Friends—about 4,400 hours of video and thousands of pages of script. But even with all that data, the proof-of-concept they’ve developed is far from perfect. In short video clips generated by the algorithm, Joey’s mouth looks garbled, and his words sound mismatched. The team is working to improve their proof-of-concept, so that newly-generated video is indistinguishable from the original.
The individual AI techniques for tracking the characters, analyzing the script, and new generating mouth movements aren’t new. Earlier this year, a Stanford project called Face2Face showed that AI could be used to flawlessly manipulate a person’s mouth on video. The Stanford team showed off their tech by literally putting words in the mouth of political figures like Donald Trump, George W. Bush, and Vladimir Putin. However, this work from Leeds isn’t just meant to manipulate a character, but create a digital model of their personality and likeness.
The researchers write that Joey from Friends might be able to take over for iPhone’s Siri or Amazon Echo’s Alexa. Even better, new episodes of our favorite shows like Friends could be procedurally generated and we could binge forever. Ultimately, that would be up to the TV networks that own the likeness of all these characters, and since the technology is still in development, we probably won’t be getting any new episodes soon. But, down the line, as long as Chandler and Monica stay together, I’m in.
[H/T Jack Clark and Prosthetic Knowledge]