Assessing the Usability of a Dialogue Management System Designed in the Framework of a Rapid Dialogue Prototyping Methodology

The goal of this contribution is to present how the notion of dialogue management evaluation was integrated in the rapid dialogue prototyping methodology (RDPM), designed and experimented by the authors in the framework of the InfoVox project. We first describe the proposed RDPM. The general idea of this methodology is to produce, for any given application, a quickly deployable dialogue-driven interface and to improve this interface through an iterative process based on Wizard-of-Oz experiments (i.e. dialogue simulations). More precisely, the methodology is decomposed into five consecutive steps, namely: (1) producing a task model for the targeted application; (2) automatically deriving an initial dialogue model (and the associated dialogue-driven interface) from the produced task model; (3) using the generated interface to carry out Wizard-of-Oz experiments to improve the initial dialogue model; (4) carrying out an internal field test to further refine the dialogue model; and (5) carrying out an external field test to evaluate the final dialogue model and dialogue-driven interface. In a second part, we more specifically focus on evaluation. We describe the integration of various evaluation aspects in the design steps corresponding to the internal and external field tests. The internal field test involved 20 "internal" users (typically the designers of the system, other members of the involved laboratories, willing relatives, etc.) and was used to further adapt the prototype. The various modifications made were validated during a second internal field test with a reduced group of seven users. The external field test involved a larger population of 50 "external" users (i.e. users that did not have any a priori knowledge of the system), randomly selected in all French speaking Swiss cantons. Its main goal was the final evaluation of the produced dialogue model, dialogue-driven interface, and evaluation procedure. Both the internal and external field tests were carried out over the phone. Finally, we concentrate on the assessment results obtained during the external field test for the InfoVox prototype. We first present the retrospective predominant modality and correlation analyses that were integrated in the external field test and then show how the obtained evaluation data can also be used in a more prospective way to derive useful hints for a better dialogue design (prospective correlation analysis). All the presented concepts are systematically illustrated by examples taken from the experiments that were carried out in the InfoVox project.