THE POWER OF A NOD AND A GLANCE: ENVELOPE VS. EMOTIONAL FEEDBACK IN ANIMATED CONVERSATIONAL AGENTS

In this article we describe results froman experiment of user interaction with autonomous , human - like ( humanoid ) conversational agents . We hypothesize that for embodied conversational agents , nonverbal behaviors related to the process of conversation , what we call envelope feedback, is much more important than other feedback , such as emotional expression . We test this hypothesis by having subjects interact with three autonomous agents , all capable of full - duplex multimodal interaction: able to generate and recognize speech , intonation , facial displays , and gesture . Each agent , however , gave a different kind of feedback: ( 1 ) content - related only , ( 2 ) content + envelope feedback , and ( 3 ) content + emotional . Content-related feedback includes answering questions and executing commands; envelope feedback includes behaviors such as gaze , manual beat gesture , and head movements; emotional feedback includes smiles and looks of puzzlement . Subjects' evaluations of the systemwere collected with a questionnaire , and videotapes of their speech patterns and behaviors were scored according to how often the users repeated themselves , how often they hesitated , and how often they got frustrated . The results confirmour hypothesis that envelope feedback is more important in interaction than emotional feedback and that envelope feedback plays a crucial role in supporting the process of dialog . A secondary result fromthis study shows that users give our multimodal conversational humanoids very high ratings of lifelikeness and fluidity of interaction when the agents are capable of giving such feedback .