The current world of fantasy filmmaking just wouldn’t be the same without Weta Digital director Joe Letteri, who’s responsible for the visual effects in everything from James Cameron’s The Abyss to Peter Jackson’s Lord of the Rings trilogy to last year’s Dawn of the Planet of the Apes, which netted him an Academy Award nomination. I sat down with Letteri, who’s now working on Avatar 2 and 3 and Batman v Superman: Dawn of Justice, at this year’s Santa Barbara International Film Festival, where he explained his recent accomplishments and the tasks that lie ahead…

NERDIST: What accounts for the improvement in visual effects that we’ve seen just between Rise of the Planet of the Apes and Dawn of the Planet of the Apes?

JOE LETTERI: There’s a lot of technical issues that underpin all this. That’s an important part of it. We got better at doing motion-capture, with being able to take it out on set. Those things help you to craft the movie when you’re shooting it, to be able to be in the moment and have the technology kind of step away. That was great. That was a large part of giving [director] Matt Reeves the space that he needed to really focus on building the film and not worry about the technology.

But behind the scenes we do a lot more work, just as time goes on, about understanding characters and what makes them realistic. It spans the realm — it’s how characters move, how they emote, how the nuances help them do that… What are the subtle details in the eye that cause you to engage? What are the realistic attributes you need for hair and water? Things like that. When you look at something for real there’s a million little clues that you take for granted and it just takes time to get all those right. With Apes we had the advantage of very well-proven technology and we could stretch out where we needed to. But we also had the advantage of having worked with the characters on the first film. So we weren’t so much finding the characters this time as kind of growing with them.

There were subtle changes we did to make them a little more humanistic, to prepare them for their speaking lines. To just bring them more present in the film itself because they were central to the film. In the first film they were just being introduced as central characters. It’s hard to put your finger on all those different factors that allow us to push the bounds of realism but more importantly to engage audience with something they haven’t seen before.

N: Can you say what the challenges you’ll face in Avatar 2 and 3 might be?

JL: In visual effects, so many things are moving at so many different levels right now. In the past, it used to be that we’d try to craft one thing at a time as being the next thing that you might need to tell the story. When Jim [Cameron] did Terminator 2, you were just at the point where doing reflection-mapped organic characters was possible. He pushed it. But it was just the one thing. And as you start moving more and more to these bigger films… I think Avatar really was the breakthrough, where we had to explore the development of a world on all fronts — from trees and plants to water to characters to spaceships. That meant that things are happening now on all fronts. That’s our motto, just from a research and development point of view — we’re looking at anything that we don’t know how to do well or don’t fully understand, to try to understand it more and more. Then the specifics of what we focus on, well, we’ll know that as soon as we see the script.

N: Having worked on Peter Jackson’s films back to back, does the task of working on Avatar 2 and 3 become easier? Did that help prepare you?

JL: It did, in the same way that the first Lord of the Rings trilogy and King Kong prepared us for doing the first Avatar. In turn, Avatar has prepared us for the films we’ve been doing subsequently. What we did to Avatar has directly led to what we’ve been able to do with the facial capture technology we’ve used on Rise and Dawn. That has served as a foundation for so much of what we’ve done, but also we’ve built on that, with the Hobbit trilogy for example. So these things have a way of feeding on themselves and coming back full circle. That’s what we’re looking forward to on Avatar.

N: Since there was such amazing rendering in the first Avatar, has that level been enough for the sequels? And if that’s the case, if you’re principal alien race is already set, will you now focus on the rendering of other alien life forms?

JL: We’re hoping we’ll learn more, in the same way we learned how to do realistic chimps when we did Rise. But we still had to add some fantastic elements to them. There were elements to true design of the main characters that pointed to their evolution. Caesar was born with slightly more human characteristics just from his design. What was great about having a second film to work on was a lot of the design choices we had to make, a lot of the really subtle choices, we spent so much time on the first film that we got a lot of that out of the way. We were able to improve in other areas that were maybe not so obvious, but they still make the character that much more realistic. So rather than obsess over the length of the eyelashes or what the nails were like, or all the details that we worked out on the first film… Once you have that established on the second, there’s a lot more focus on the nuance of character, on really tuning the eyes, on really looking at what’s happening on the skin. There’s a lot of scientific and technical details — you keep learning more and more of what you know, but it also exposes what you don’t know. That’s what we’re hoping for — to continue that trend that we saw between Rise and Dawn, and that that will apply to the Avatar films.

N: You’re also working on Batman v Superman: Dawn of Justice. Is there anything you saw in Man of Steel that you’d like to improve?

JL: For us on the first film, we did the opening sequence, the Krypton sequence. That was our body of work primarily. Zack Snyder threw that right in our comfort zone. We like creating whole worlds. We didn’t deal so much with what happens in the human world, in the city of Metropolis. We’ve done that on Marvel films, so we had experience with that. But the more that you can keep bringing those fantasy realms into it, the more I like doing those parts of it.

N: Is this film therefore a bigger challenge for you, since it apparently will be set primarily in our here and now?

JL: It doesn’t make it trickier. We know how to do it, and we always like to push the boundaries of what’s real whenever we can. That’s just interesting to me.

N: Thank you very much, Joe.

JL: My pleasure!

Letteri was nominated for best special effects at last night’s Academy Awards, but lost to the team behind Interstellar. You can see the rest of the Oscars results here on Nerdist.com.