Engaging spect-actors with multimodal digital puppetry

We present Euclide, a multimodal system for live animation of a virtual puppet that is composed of a data glove, MIDI music board, keyboard, and mouse. The paper reports on a field study in which Euclide was used in a science museum to animate visitors as they passed by five different stations. Quantitative and qualitative analysis of several hours of videos served investigation of how the various features of the multimodal system were used by different puppeteers in the unfolding of the sessions. We found that the puppetry was truly multimodal, utilizing several input modalities simultaneously; the structure of sessions followed performative strategies; and the engagement of spectators was co-constructed. The puppeteer uses nonverbal resources (effects) and we examined how they are instrumental to talk as nonverbal turns, verbal accompaniment, and virtual gesturing. These findings allow describing digital puppetry as an emerging promising field of application for HCI that acts as a source of insights applicable in a range of multimodal performative interactive systems.