Avatars

6.12 Avatars

Photo from Wikipedia.

The talking avatar, an animated speaking virtual character with vivid human-like appearance and real or synthetic speech, has gradually shown its potential in applications involving human-computer intelligent interactions. The talking avatar has rich communication abilities to deliver verbal and nonverbal information by voice, tones, eye-contact, head motion and facial expressions, etc. Avatars are increasingly being used on a variety of electronic devices, such as computers, smart phones, pads, kiosks and game consoles.

Avatars also can be found across many domains, such as technical support and customer service, communication aids, speech therapy, virtual reality, film special effects, education and training [6]. Specific applications may include a virtual storyteller for children, a virtual guider or presenter for personal or commercial website, a representative of user in computer games and a funny puppetry for computer-mediated human communications. It is clearly promising that talking avatars will become an expressive multimodal interface in human computer interaction.

Despite many years of efforts, current research has not yet advanced to the stage where it is possible to synthesize an affective intelligent talking avatar who can express their feelings and emotions through vocal and facial behaviors as natural as our human beings. To this end, recent researches aim to understand complicated human behaviors and generate lifelike speech and facial expressions according to the emotional content to enhance the expressivity of talking avatar.

https://thenextweb.com/artificial-intelligence/2019/05/23/samsungs-new-ai-can-create-talking-avatars-with-a-single-photo/ (from real face to avatar)

Research Notes:

Avatar Personalities – Driven By Artificial Intelligence

Published on May 12, 2020

Markus Gross

Swiss Computer Scientist, VP of Research At Disney, Expert in Graphics, Vision and AI - Investor - Two Academy Awards

5 articlesFollow

I spoke recently about our journey of creating realistic **digital humans**, that are visually indistinguishable from the real thing. We also discussed the challenge of bringing such models to life instantly – in realtime. The assumption is that instant digital humans will represent us as believable avatars in the upcoming Cyberworld (more about it **here**), and such avatars will be driven and controlled by their real-world counterpart – a human being like you or me. This is a very important assumption in the development of applications in social media and the telepresence industry.

Today, I want to lay out a scenario where a digital avatar – human-like or non-human, realistic or stylized – is not driven by you or me, but by artificial intelligence. Potential industry applications of these AI avatars are countless and span a huge spectrum as diverse as personal digital assistants and advisors, virtual liftboys, greeters in physical and online retail stores, personalized school teachers, fitness trainers, or virtual characters in the interactive stories we are going to consume in the future.

Thanks to machine learning, there has already been much progress in AI-driven digital assistants. Alexa, Siri, Cortana, etc, are good examples of products that work specifically well in a query-type context; for instance, when we ask them: “What’s the weather in Zurich today?” or “Can you play me a few Beatles songs?” With a variety of existing software products, it is pretty easy to build a templated chatbot that, for instance, can help you to book a table at your favorite restaurant, without any additional effort on your part.

Smarter versions of these products also learn from your queries and improve the quality of their answers over time. However, we have to admit that the current state of research is still pretty far from an AI system that truly understands and comprehends human language, emotion, personality, story and context. Additionally, most of the existing systems are ‘faceless’, much like HAL 9000, and don’t have an embodiment that lets us connect and bond emotionally.

Let me summarize here several key technical challenges we face if we want to take a personality out of a story, an institution or a brand and bring it to life. Let’s call our fictitious personality Charlie:

• Personality integrity: the human brain has the ability to construct a model of Charlie’s personality from very sparse data, such as a few pages of dialogue in a book or a few minutes of presence in a movie. (Deep) machine learning algorithms need huge amounts of data to do the task, if at all. Building a believable AI-driven avatar of Charlie must ensure that the integrity of its personality is not compromised by erroneous or unexpected answers.
• Notion of memory: we still do not have a commonly agreed concept of how to represent past dialogue between Charlie and you; i.e. how the actual conversation can be best informed from past conversations and what Charlie has learned about you from your previous chats.
• Knowledge representation: besides the actual state, Charlie must have access to sufficient knowledge about our human world in general. For instance, it should be able to handle questions such as: “Can you give me a summary of Mahayana Buddhism?” Or “Who is the current president of the United States?”
• Conversational natural language processing: even general chit-chat conversations are still a significant challenge for most chatbot systems currently available. It is impossible for them to pass, for instance, the Turing **test**. However, one of my ETH PhD students recently ran a few experiments on a short-spanned chatbot dialogue and 60% of the user study participants erroneously guessed the AI was human. At least, there’s hope!
• Distinct personality: humans are more likely to bond with avatars that have a distinct personality. The distinction is an invariant that lets Charlie behave and react in a certain way. Therefore, a believable AI system must synthesize logical and meaningful responses in a dialogue flavored with an anticipated personality of the avatar. Currently, we do not really know how to model the personality of the avatar other than feeding tons of data into a neural network or using simplistic hard-wired models. None of these options is the best way of modeling Charlie’s personality, particularly when the data is not readily available.
• Emotion: in addition to personality and partly as a function of it, Charlie’s character needs to display emotion. This is what makes it human and is a function of its own internal state and the history of its dialogue with you. It should be able to portray happiness, anger, sadness, compassion, fear, disgust – ideally, all the emotional gamut implying human interaction. Similarly, the AI has to read the emotion of the human counterpart (you) and adapt both Charlie’s answers and the tone of its voice to it.
• Body language and facial animation: a significant portion of human communication occurs through non-verbal elements, such as body language. An intelligent AI-driven animation system will have to modulate the responses generated by AI through the personality model and the underlying emotional state, and translate them into articulation and facial performance consistent with our mental model of Charlie.
• Speech recognition and synthesis:automatic recognition of speech and translation into text is certainly the most developed technology in this space, and some commercial speech recognizers even beat human performance. Some promising products from startup companies also generate realistic sounding speech from human samples. What’s much more challenging is to adapt the prosody, speed and vibe of Charlie’s voice as a function of its emotional state and response.
• Planning, goals, beliefs and intentions: finally, a believable avatar must be proactive in its intentions, and we expect Charlie to plan its actions and dialogue as a function of its goals and beliefs. Existing research on agent-based modeling provides promising results.

Let me give you a (incomplete) list of examples of work in the wider space of AI-driven avatars:

• Some of the true pioneers who saw the potential in combining graphics, digital human technology and agents include Prof. **Demetri Terzopoulos** from UCLA, Prof. **Norman Badler** from the University of Pennsylvania, and Prof. **Nadia Thalmann** from the University of Geneva, all of whom have done a body of excellent work in the space.
• Among younger scientists, I appreciate the work of Prof. **Mubbasir Kapadia** on agent-based modeling and animation
• The Institute of Creative Technologies at UNC is doing very interesting work on **behavioral animation**
• Prof. **Robert Sumner** and his team at **ETH-GTC** together with Disney Research have built a very compelling hololens **demo** showing a virtual character driven by NLP that interacts with the real world in a creative play scenario. They use some of Microsoft’s excellent services on **language technology**
• **Cleverbot** is a very good example of a believable conversational chatbot. It scored 60% in the Turing test in some of our own test scenarios
• Another example of a commercial chatbot service is Microsoft’s **bot framework**
• Google has recently published **Meena**, a conversational chatbot - at its heart is a 2.6B-parameter deep learning architecture
• In the field of social robots, a good example is **Pepper**
• Another one is **Jibo**
• The startup company **Animatico**, founded by very talented former PhD students from ETH, is building a framework to bring Charlie-type characters to life
• **CereProc** has a fairly compelling product for voice cloning

So much more is going on this this very vibrant field and I apologize for the many omissions.

Overall, it sounds as though a very long and exhaustive list of technologies have to be developed, with a way to go until we get to a realistic digital avatar driven purely by AI. But my few examples indicate that the current AI horsepower of numerous corporate research labs, as well as the progress in academia and the driving forces of deep tech startups, will bring the snippets and pieces into place.

I also envision a bunch of near-term milestones that before too long will be turned into commercially very successful products.

I believe that a truly emotional conversational avatar is a very realistic moonshot to strive for – as part of achieving of what is called Artificial General Intelligence (AGI), which will change the world forever.