This repo contains my final thesis my M.S. in Computer Science degree at Colorado State University (Fort Collins, CO).
ABSTRACT
Embodied virtual agents offer the potential to interact with a computer in a more natural manner, similar to how we interact with other people. To reach this potential requires multimodal interaction, including both speech and gesture. This project builds on earlier work at Colorado State University and Brandeis University on just such a multimodal system, referred to as Diana. I designed and developed a new software architecture to directly address some of the difficulties of the earlier system, particularly with regard to asynchronous communication, e.g., interrupting the agent after it has begun to act. Various other enhancements were made to the agent systems, including the model itself, as well as speech recognition, speech synthesis, motor control, and gaze control. Further refactoring and new code were developed to achieve software engineering goals that are not outwardly visible, but no less important: decoupling, testability, improved networking, and independence from a particular agent model. This work, combined with the effort of others in the lab, has produced a “version 2” Diana system that is well positioned to serve the lab’s research needs in the future. In addition, in order to pursue new research opportunities related to developmental and intervention science, a “Faelyn Fox” agent was developed. This is a different model, with a simplified cognitive architecture, and a system for defining an experimental protocol (for example, a toy-sorting task) based on Unity’s visual state machine editor. This version too lays a solid foundation for future research.
Four avatars functional with the same cognitive architecture. (a) he original Diana model. (b) The new Diana model. (c) A humanoid robot. (d) An industrial robot arm.
Internal state displays, enabled for debugging or to aid with research objectives.
"Faelyn Fox" avatar, used in child development studies.