IAN BLOOM | Founder, CEO | MANIFOLD VALLEY
With deep roots in film-making, Ian Bloom, Founder and CEO of Manifold Valley in Los Angeles discusses his company’s development in next-gen performance capture technology that reconstructs real-world actors to identical in-game characters
Blending Cinematography and Programming
Your background spans cinematography and programming. How did you merge these creative and technical interests?
I started programming at age seven. That happened because my older brother went to computer camp and came home and taught me to make simple games in BASIC. My goal as a kid was to make games like Castle Wolfenstein and Quake and to program like John Carmack, the Co-Founder of id Software. Reading up on 3D graphics magazines and Renderman (the Pixar rendering engine) at the University of Maryland Library back-stacks shaped my early years and in high school. I developed my own algorithm for fast occlusion and rendering of textured polygons that was fueled by my passion for games.
Later, I received a scholarship from Deep Springs College where I became one of the subjects of a documentary. I befriended the Sound man and after I graduated, travelled to Ireland to work on his film. This was my first time on a film set and encounter with a real cinematographer who shot actual 35mm film.
You have an interesting upbringing where family legacy of invention clearly influenced your path. Can you tell us about that?
Absolutely. My parents ran an engineering startup from our house, Dad performed the technical work and Mom secured the contracts. Mom’s brother was always over for dinner and their entire conversations were about start-up ideas. As kids we had access to CAD workstations and all types of fabrication tools. For instance, my brother entered a competition to design an experiment for use on the Space Shuttle. His project tested expanding foam as a building material in orbit. It won the national prize which he received a scholarship, but the reason he had knowledge on expanding foam was because Dad fabricated prototypes using the material in our garage. Innovation is literally in the genes with the Bloom family!
Pioneering Film Technology and Cinematography
You have transitioned from cinematography to developing machine learning for digital humans. How do you manage to work in such different fields?
After earning a Film Production degree in North Carolina, I moved to New York where I immersed myself in various roles from Gaffer to Digital Imaging Technician, and eventually as Director of Photography on many projects. Initially I was mostly using film, then worked extensively with the early digital cinema cameras, in particular with the RED camera. The projects I shot would rarely have the resources to properly utilise the raw footage for finishing, and this affected the visual quality of my work. In the early years in NYC, it wasn’t unusual to have little work between November and February, so I used that down-time to write some software. I created a tool called Crimson Workflow™, which was a bridge between editing programs and high quality delayering of 4K footage. I sold the software online and it went viral for about two years with a large customer base reaching users like Weta Digital in New Zealand. Basically, for those utilising a RED camera and seeking cost-effective post-production, Crimson was an essential bridge for a certain segment. That experience allowed me to invest in several brick and mortar film-adjacent companies, including camera rentals and a post-production facility in Williamsburg.
Programming was still intrinsic, I was a regular at the Haskell meetups in Manhattan and wrote another rendering engine in Haskell and OpenCL. I also read a bunch of textbooks about machine learning, one stand-out read was Deep Learning by Ian Goodfellow, who’s a renowned computer scientist and engineer. The main algorithm for Manifold Valley just occurred to me one day – next I was running a start-up trying to build that.
How did your experience shape your approach to developing advanced technologies?
One invaluable lesson I learned from my extensive film experience is that no amount of cinematography tricks or technology can make a movie exceptional. I remember a specific moment where I was filming an amazing actress named, Breeda Wool. I had this complicated lighting, I’m sitting on a big dolly on tracks and through the eyepiece I’m watching a take.
“Her [Breeda Wool] portrayal of this character was so raw and in-the-moment that I had this epiphany where I realised my job is to just get out of this person’s way. She’s lighting the film for me in a way and my work is just in service of her work.”
So, it may sound like a crazy leap to transfer this understanding into machine learning-based performance capture, but once we appreciate that actors inhabiting their characters means everything, the industry will produce movies or games with ‘lines around the block’. People in the film business understand that franchises are anchored to characters which people fall in love with. The impact of the technical process on the talent is not a secondary consideration; it’s the primary focus. When people fall in love with your characters, they’ll watch the sequels, buy the toys and read the books. The process for human talent is critical and is why the Manifold Valley process will be the inflection point.
Revolutionising Acting in Video Games with Manifold Valley Technology
Manifold Valley ‘performance reproduction’ with machine learning: actors’ facial expressions retargeted to 3D character in real-time.
How does Manifold Valley enhance actors’ natural performance during live shoots?
Traditional motion capture methods often hinder actors, forcing them into restrictive suits and equipment that disrupt their creative process. In particular, when a performer has to wear a helmet strapped with a witness camera directly in front of their eyes is from my perspective, totally unacceptable. Their intricate facial expressions captured are not usually a direct translation of their true live performance on-set, to what’s seen on their game characters face. My impression is actors know this, which affects their performance and desire to accept game acting projects.
“There’s so many additional steps in the process and quality loss which means what’s taking place in the studio is not guaranteed to be transmitted into the game. The actor is aware that the rendition which makes it in the video game is very different than their actual performance. That’s true even in the most modern games today.“
Our process eliminates these barriers by harnessing machine learning to make the game character asset produce the same image we captured of the actor through the game engine. We are considering coining the term ‘performance reproduction‘ instead of capture, as capture implies only the first step. We need to have an end-to-end process the Director trusts, just like they trust a close-up on a film camera. The cameras themselves can be extremely far away from the talent. In fact, so far, they may be unaware of the cameras. The user experience for them needs to be exactly as comfortable as a rehearsal day on a theatre production.
In essence, you’re replicating a theatre stage, an actors’ natural environment to perform, while Directors and crew utilise the technology, allowing events unfold before them. Is what you’re achieving akin to ‘invisible technology’?
Exactly right. When Emmanuel Wazar, our Animation Specialist joined Manifold Valley, he was the first to harness the tech in an artistic manner. We’d been deep in foundational experiments for two years and when he produced an animation with data from one of our networks, it immediately clicked. It was overwhelming to witness!
When people recognise the creative possibilities unlocked by Manifold and see how our pipeline is revolutionising their approach, it’s incredibly gratifying. Our approach enhances realism in rendering facial expressions and body movements, revolutionising how stories are told in interactive media.
What inspired your shift into developing this groundbreaking technology?
Computing has always been something I would return to especially since it occupied hours of my free time. Don’t get me wrong, film-making took me all over the world to incredible places like the Amazon, Antarctica and underwater, deep down I knew I wanted to use this other talent.
How do you see Manifold Valley shaping the future of video games?
A lot of what we see in games today is shaped by the economics of including digital human characters. In Days Gone (a Bend Studio and Sony Interactive Entertainment open-world action-adventure game) the characters often talk to each other on radios, often while the characters are driving or have their back to camera. I just think it’s important to make it clear that what we mean here is that there are large parts of the story exposition that you only ever hear about in passing audio, because only a precious few cutscenes are produced with facial animation. From shooting movies, you learn that capturing close-ups of the actors’ face is crucial to create an emotional connection to the characters and story points for the audience. We want to give this same ability to game developers.
When we change the economics of putting complete human performances in games, means many artists will prioritise human interaction over traditional game mechanics, like say first person shooting. In the future we’ll see entire video games where all game mechanics are purely based on human interaction. This shift will elevate the cultural significance of video games, making it a compelling medium for storytelling and immersive experiences, comparable to the impact cinema has had on modern culture. We have an opportunity for video games to be the centre and source of culture. But deeper human performances are paramount to achieve that.
Future of Digital Characters in Gaming and Beyond
How does Manifold Valley enhance storytelling in games?
We find it brings deeper storytelling by creating characters which emotionally connect with gamers, in the same way iconic figures in film and TV franchises like Star Wars and Breaking Bad has. We have these characters people love and gravitate towards. Companies like Disney spin off new shows based on small characters.
In contrast, video games haven’t as many examples of people falling in love with game characters. They primarily anchor franchises on gameplay and game worlds. Gameplay will always be an important anchor point, but I think gaming studios need technology like ours to deliver acute realism efficiently because they need characters that gamers identify and fall in love with. This emotional engagement boosts game longevity, player immersion and further commerce.
Can the tech adapt to different character styles beyond humanoids?
Yes, the same basic technique is applied to any style of character you want to capture, from fantasy creatures to realistic humanoids. Real-world props are also very easy for us to capture. The rest of the team tease me because I’m always talking about how we can apply this tech for horse animations.
“Our technique for facial rigs works well for retargeting making it much easier and we deliver superior quality.”
Where do you see the future impact of AI-driven performance capture and what role will Manifold Valley have in shaping this future?
AI-driven technologies like Manifold will revolutionise digital human creation, offering game studios, filmmakers and content creators unprecedented control and efficiency in producing compelling narratives and life-like characters. The flexibility of our system extends its application beyond gaming, impacting sports broadcasts and other digital entertainment formats. Our system supports different cameras, reaching 200 frames per second, everything is in real-time. We have the potential to produce quality slow motion for sports replay. I call it ‘sports reimagining‘ because it’s a reconstruction through an engine. When we produce great quality renditions and the audience is emotionally captivated by what they’re viewing, it doesn’t matter. Therefore, when we place people into the eyes of an athlete, then we’re able to ‘transport’ them in the game in ways that will never be possible with traditional photography. In my opinion this has huge potential for sports.
In summary, what is your vision for Manifold Valley in shaping this future?
Most companies in the artificial intelligence and machine learning space are building or using huge one-size-fits-all models. So their entire product line consists of only a few models available to all customers. I think that comes with some big trade-offs in computing costs, but mainly in control.
What differentiates us is the fact we efficiently specialise in smaller models which does exactly what you need as a creative tool. For performance capture in a few hours we’re able to produce a specific network for a specific actor and animation rig, which will run an entire production. That’s extremely powerful for creating compelling products where human creativity is the main input and machine vision is just the translation step.
I believe there will be a place in the market for both approaches – large generative models and targeted specialisation. We need both to produce something which people are prepared to pay to see.
I’m so excited about the creative potential in ways artists will use our technology for! That’s what motivates me to push through the difficult times running a start-up.
Thank you Ian for contributing to TrailblaXR, keep inspiring and trailblazing the performance capture, or should we say ‘performance reproduction‘ and games industries!
Image Credits: Manifold Valley, Inc.
Return to TrailblaXR Magazine
Become a future TrailblaXR of the multiverse and shine under the starlight with Instinctively Real Media.
Let Instinctively Real Media promote your brand, contact Enquiries@InstinctivelyReal.com for a chat.
Immerse yourself in the latest articles and news. Don’t worry, we don’t believe in spamming.
Discover more from Instinctively Real Media | Global Creative Agency
Subscribe to get the latest posts sent to your email.
