​​VR, AI & Machine Learning: How Sundance filmmakers are experimenting with new formats

At The Power of Story—a 2019 Sundance Film Festival panel sponsored by Dropbox—five filmmakers gathered to share their wild visions for the future.

John Gaeta—the man behind “bullet time” from The Matrix series—says we’ve only seen the first few stages of virtual reality in film. Machines, he says, will increasingly play a bigger role, but humans will remain at the center, for now. “Stories inherently are created by people for people,” he says. “We’re moving into an era where we’re teaching machines our stories, and machines are doing some form of processing.” 

The way John sees it, machines will start to create their own narratives, building on what humans originally taught them. “Is [cinema] going to separate between human stories and machine stories?” It’s a future that’s both exciting and perhaps a little scary, as creators will face a tradeoff between surprise and control.

In the meantime, John’s company (Magic Leap) is helping creators like Sarah Ellis of the Royal Shakespeare Company. Using virtual reality tools from Magic Leap, Sarah can take a 400-year-old play and add live, digital avatars performing on stage with human actors in real time. As director of digital development at the RSC, Sarah wants to bring together the old and new in a bold, unexpected way.

Sometimes, Sarah’s critics will question why we’d want to bring VR into a production as sacred as Shakespeare. “Theater’s always been reinventing itself,” she says. “It’s reinventing itself alongside what’s been going on in society, what’s going on with technology. Technology and theater have always had a relationship.” Sarah’s work serves as a good reminder that Shakespeare remains as vital and truthful today as it was in 1619.

While John and Sarah still spend much of their time with a screen or stage, artist Stephanie Dinkins wants to experiment with narrative in a completely different medium. “I’m using deep learning algorithms and oral history to try to make a memoir of my family,” she says. In practice, this means creating a sculpture anyone can walk up to and talk to. Superimposed with the faces of various family members, the sculpture has gradually learned to speak and think for itself. Unlike a contained play or film, the sculpture’s narrative continues over time, and no two interactions are the same.

Like Sarah, musician and comedian Reggie Watts wants to rethink narrative from the ground up. “What I’m trying to do is become improvisational inside of very complex mediums,” he says. His Sundance hit from two years ago combined music, VR, and comedy to create Waves, a project that defies simple categorization. Bursting with color and hypnotic in its use of shapes and sound, the piece dares viewers to think about all the tools we haven’t yet used or haven’t thought to explore.

Today, Reggie obsesses over time travel. “I get excited about championing the idea of other forms of time travel. I’m a huge time travel fan, and I’m trying to set up an artificial time travel network using VR, immersive theater, all sorts of mediums.” Even if he can’t actually bring viewers back in time today, he’s convinced he can do the next best thing with technology and creativity. He can’t describe exactly how he’ll do it yet, but his past work gives fans confidence. “Because I’m an improviser, all writing happens in real time. It’s chaos organizing itself as it wishes to be organized.”

Finally, there’s Terence Nance, the creative mind behind the new HBO series, Random Acts of Flyness. As moderator Julia Kaganskiy put it, Terence’s work involves the “mixing of surreal and kaleidoscopic forms of expression.” But to simply watch his work and note its bizarre nature would miss the deep, philosophical thinking that underlies all of what Terence does—and his passion for questioning what we even mean by “narrative.”

“What is a way of not being reactive to this Western idea of ‘narrative?’” he asks. So many of the stories we hear—whether in books, television or film—are influenced by a standard structure, a hero and villain, a beginning, middle, and end. Even the Power of Story panel itself was based on a series of accepted principles.

“We’re having this conversation in English, for instance. Even just the reality of what [a story] could have been—what could we be in terms of how we express our shared experience, given that reactivity to culture?”

Terence wants to keep questioning why we tell the stories the way we do, and if possible, to change those assumptions entirely. And if there was one thing every panelist could agree on, it’s that even the five of them are just getting started.