All that glitters is gold indeed!

The 89th Scientific and Technical Awards, hosted by the Academy of Motion Picture Arts and Sciences’ — a precursor to the Academy Awards later this month — was a proud evening for our country, as two Indians scooped up honours! While Parag Havaldar was recognised for the development of expression-based facial performance-capture technology at Sony Pictures Imageworks, Kiran Bhat was among the team that won for the design and development of the ILM facial performance-capture solving system — both in the Technical Achievement Awards.

After a glittering ceremony at Beverly Hills, Parag and Kiran talk to us on the merits of their big wins, the lowdown behind their impressive array of work and on the possibility of future collaborations with Indian cinema.

Q Many congrats on the Sci-Tech Oscar! Were you expecting this?

Parag: I was interviewed by the Academy a number of times last year for this technology. Another big body of work of mine was nominated a few years ago, but unfortunately the Academy decided that it was premature at that time.

Kiran: We had been working on this tech for almost six years now and have seen the technology mature from a science project to a fully production quality system.

I can’t say that we were expecting the award, we were cautiously optimistic!

Q Could you give us a little background about yourself?

Parag: As a kid, I always loved math, physics and programming. I was happy to be able to pursue a degree in computer science at IIT Kharagpur, after which I furthered my interest to do a PhD in computer graphics, at the University of Southern California in Los Angeles.

With Los Angeles being vibrant in the entertainment area of movies and games, I started working for the movie industry — from the technology side. There are many projects that I am proud of where realistic and stylised (cartoony) facial animation play a formidable part in the emotional connection with audience — the Spiderman series, Monster House, Alice in Wonderland, Green Lantern, Hotel Transylvania, Smurfs to mention some.

Kiran: I studied high school at Stanes in Coimbatore and went to BITS Pilani for my engineering. I did my PhD in Robotics from Carnegie Mellon University in Pittsburgh and worked at ILM on several projects related to computer vision before focussing on facial performance capture.

In late 2015, I reconnected with a batch mate from BITS Pilani, Mahesh Ramasubramanian, who is a visual effects supervisor at DreamWorks Animation. We started discussing ideas for bringing our experience of building emotionally engaging characters to the consumer market. This resulted in us starting Loom.ai in early 2016, with the aim of building 3D digital avatars. You’ll hopefully see our new technology in action in the later half of this year!

Q Do you keep abreast of tech and animation trends in India? For instance, films like Baahubali or the upcoming 2.0 starring Rajinikanth are touted to be breakthroughs. What is your take on the scene overall?

Parag: Of course, and with a matter of great pride! When Baahubali came out last year, we (my colleagues and I) went and celebrated it — for its visuals, its story and the technology. None of us (including myself) understood Tamil, but it was a visual treat. We look forward to the sequel. Today, the film industry is a big melting pot where people — artists, animators, directors, engineers — all move around from place to place. So great projects will be created in all corners of the world, it is just a matter of time.

Kiran: The startup trend in India is sky-rocketing, I see a lot of venture capital investment in bigger Indian cities. Not all that surprising, considering the amount of highly skilled engineering talent that we produce.

In the film world, many new movies, including the ones that you pointed out, have high production value and overall quality, including VFX. India could benefit from having a few good sci-fi movies which can push the boundaries of imagination.

Q What’s next for yourself after this award? A sneak-peek perhaps into what we can expect over the course of the next 2-3 years?

Parag: I have consciously left the movie industry, but have not left the world of using technology for entertaining people. I now work for a premier game company called Activision-Blizzard that has created  some of the biggest games such as World of Warcraft, Call of Duty, and we recently released Overwatch.

Stories and movies have always excited us, but our world has changed  over the last few decades to define novel paradigms of digital entertainment. As part of the faculty of computer science at the University of Southern California over  the last decade, I have seen much research/technological progress made in various fields of computer science that relate to entertainment. Today I see many mature areas:

Natural language processing and speech synthesis — where a computer can analyse what we speak, and digital characters can be made to synthesise speech.

Artificial intelligence and machine learning — where a computer can reproduce intelligent behaviour. Interactive computer gaming that involves characters — where people can interact with characters.

Faster computational graphical processing units — enabling interacting with images. With the confluence of all these fields we might start to see a transformation in entertainment from a passive ‘storytelling’ experience as we witness with movies, to a possibly active and even interactive ‘story defining’ experience.

That is one reason why I have moved to the game industry. I think games of the future will be more about ‘active, emotionally engaging’ roleplaying with digital characters. Passive watching as we welcome in movies today will never go away but active engagement with digital characters in games and virtual reality worlds will definitely prove exciting and entertaining.

Kiran: We are super excited about the brand new AI technology that we are building at Loom.ai. We feel that it is truly transformative, we hope to digitise five billion faces using this tech. Anyone with a smartphone can use Loom.ai’s technology to create their 3D digital identities, which can be trained and personalised. We envision that 3D avatars would be the enabler for modern virtual communication in mobile space and especially in virtual reality.

Q Can we ever see you working on an Indian film/project?

Parag: I would love to share experiences and if allowed to, even mentor. It will be a privilege if I can, in any small way, help advance the art of filmmaking in India. India has a rich culture with vivid stories, and many talented directors, artists to make these stories come to life. Additionally, it also has the rare distinction of nurturing bright technical minds that can contribute to this art. I foresee a bright future for the Indian industry.   

Kiran: I am currently focussed on getting Loom.ai’s technology off the ground, and our early focus will be on VR and messaging. But, perhaps in a year or so, our technology will be accessible to a wide range of creative artists, including Indian filmmakers and VFX houses. At that point, it’d be really cool to work with SS Rajamouli (or other VFX savvy directors) who is creatively pushing the state of the art in Indian movies, if the opportunity arises.

Q A word of advice to aspiring animation designers in India, on how to innovate?
 
Parag: The advice depends on what role you play in the world of filmmaking. Filmmaking today is a holistic process that is dominated by the artistic vision, as it rightfully should be, of creative people — directors, visual effects supervisors, animators etc. Technology plays an important role to help that creative vision, but it is only a means towards attaining that vision. The artistic people can craft their art to produce innovative and effective visual effects when they understand how technology works, and how it can be harnessed for their own creative means.

The technology experts need to understand the ultimate artful vision of the artistic people, so as to implement useful solutions that can be controllably used by the creative artists. It is in the prosperity of such synergy between art and technology that great projects triumph.

Kiran: Innovation is hard, takes a long time to materialise, and is constantly evolving. Pick a problem that you really enjoy working on, and pour all your energy on it, and get feedback from other creative people and experts, and always be self-critical. With enough iterations and good ideas, things will
crystallise into something amazing.