This realistic face-to-face interview is hosted not in the real world

 

A few years back, Facebook Reality Labs (FRL) Research unveiled Codec Avatars, a groundbreaking project at the intersection of Computer Vision and Machine Learning. These avatars offer uncannily realistic digital representations, mirroring people's real-life appearances and movements in real-time. The potential applications of Codec Avatars span the realms of gaming, social interaction, and work. In this article, we'll delve into the latest strides made in this transformative venture.

 

 

Codec Avatars: Bridging Geographical Divides

Imagine being able to instantly connect and interact with someone across the globe, almost as if you were physically in the same place. This is the promise that Codec Avatars hold, revolutionizing how we bridge geographical distances and interact with one another. But how do we make this technology more accessible and inclusive?

The latest phase of the Codec Avatar project is the Codec Avatar Inclusive Dataset Research Project. Previously, creating a high-quality Codec Avatar required hours of meticulous image and audio capture in specialized labs. The collected data was then processed over several weeks to generate the final avatar. To scale this technology for broader adoption, the focus is shifting towards enabling instant Codec Avatars, accessible via mobile phones. However, this transition demands a more comprehensive and diverse dataset, encompassing gender expressions, body types, hairstyles, skin tones, and more.

To gather such diversity, FRL is embarking on a unique journey to take their lab equipment to Meta campuses. They aim to involve a broader community, both internal and external, in this voluntary research project. Participants will engage in activities mirroring those at the Pittsburgh lab, such as mimicking expressions, reading sentences, and engaging in conversations. This data will pave the way for a future where anyone can create their Codec Avatar from home without specialized equipment.

The ultimate goal is to democratize the creation of these avatars and make them more representative of the diversity that exists in the real world. While challenges like depicting avatars with accessories, such as glasses, still exist, the commitment to enhancing Codec Avatars remains unwavering.

Applications in Gaming, Social Interaction, and Work

Yaser Sheikh, Director of Research at FRL Pittsburgh, underscores the project's core mission: to bring people closer together, even when they are oceans apart. The potential applications of Codec Avatars span various domains, including gaming, social interaction, and the workplace.

In the realm of gaming, Codec Avatars offer the prospect of highly precise virtual representations. These avatars go beyond advanced graphics and motion tracking; they aim to replicate the ease and authenticity of interacting with someone right in front of you. It's about achieving genuine social presence within virtual gaming environments.

Social presence, according to Sheikh, can be gauged by two simple tests: the "ego test" and the "mother test." You have to love your avatar, and your mother should recognize and love it too before virtual interactions feel as natural as real-world interactions. The goal is to make interactions in simulated environments indistinguishable from physical presence.

FRL Pittsburgh has made significant strides in this regard, continually refining avatar details, improving expressions, and ensuring realistic eye contact between avatars. These efforts are pivotal to achieving social presence in virtual spaces.

In the context of the workplace, Codec Avatars could redefine remote work and collaboration. By enabling individuals to project lifelike avatars of themselves, this technology has the potential to create a sense of physical presence during virtual meetings and conferences. This can significantly enhance remote team dynamics and collaboration, bringing the "workplace" closer, regardless of geographical distances.

The Journey of Codec Avatars: From Vision to Reality

The journey of Codec Avatars began with the ambition to erase physical distances between people. To achieve this, FRL Pittsburgh was founded in 2015, with Yaser Sheikh at the helm. This multidisciplinary team set out to create the future of social presence by building highly realistic avatars.

With offices across the United States, FRL is dedicated to pushing the boundaries of AR and VR technology. Researchers and engineers in the Pittsburgh lab, like Danielle Belko, Technical Program Manager, were drawn to this mission, embarking on a journey of cutting-edge research. They work in fields spanning machine learning, materials science, optics, and haptics, leaving no stone unturned in their pursuit.

The creation of lifelike avatars hinges on capturing intricate physical details, such as subtle eye movements. This level of detail is essential to constructing authentic avatars. It involves complex processes such as measurement, as Tomas Simon, an FRL Research Scientist, emphasizes.

The creation of Codec Avatars involves two primary components: an encoder and a decoder. The encoder uses cameras and microphones on a headset to capture the subject's actions and surroundings. This data is then compiled into a unique code, which represents the person's bodily state and environment. The decoder interprets this code, generating audio and visual signals to create a lifelike representation of the sender's likeness and expressions.

However, creating avatars that perfectly replicate every individual's uniqueness is an ongoing challenge. The team at FRL Pittsburgh is relentlessly working to improve this technology, developing data-driven solutions for photorealistic avatars.

Animating Avatars and Ensuring Trust

One of the challenges in bringing Codec Avatars to lightweight headsets is animating them realistically. The Pittsburgh team has devised prototype Head Mounted Capture systems (HMCs) to capture a broad range of human expressions. These HMCs integrate cameras, accelerometers, gyroscopes, magnetometers, and microphones to achieve this. The challenge is to seamlessly integrate these sensors into comfortable headsets while maintaining the user's ability to gesture and express themselves freely.

Software is another critical aspect, providing flexible control over the capture system and enabling precise image capture for various research needs. This suite of tools helps researchers and developers enhance the avatar creation process.

Trust is a fundamental element in virtual reality, much like real-life interactions. To build this trust, avatars must capture subtle expressions and movements accurately, ensuring there is no doubt about the virtual persona's identity. Safeguarding user data is also paramount, with rigorous authentication, device verification, and hardware encryption measures in place.

Deepfake technology poses a unique challenge. The ability to fabricate scenarios convincingly using AI can undermine the authenticity of virtual interactions. FRL Pittsburgh is actively working on safeguards to protect avatar data, including potential identity verification solutions for future devices.

Connecting People Anywhere

Codec Avatars hold the potential to redefine how we connect and interact with others. Imagine putting on a headset and instantly being transported to attend a class, collaborate at work, or celebrate with family and friends, regardless of geographical distances. You'd look, move, and sound just like your real self.

This technology aims to eliminate the constraints of physical distance, making virtual connections as seamless and authentic as real-world interactions. The journey toward this future is ongoing, but Facebook Reality Labs is wholeheartedly committed to realizing it.

Yaser Sheikh envisions a world where you can go to bed with a head-mounted display and wake up at work. Families can gather for dinner, even if they are thousands of miles apart. Codec Avatars have the potential to make this vision a reality.

Codec Avatars represent a remarkable fusion of cutting-edge technology and human connection. As this technology continues to advance, the dream of transcending physical distances in our interconnected world draws ever nearer.