TESI: A Toolkit for Tangible and Embodied Sound Interaction

Sound offers unique affordances for interaction designers, enabling immediate, embodied, and expressive experiences. Yet sound is often treated as peripheral, limited to simple notifications rather than being explored as a central design material. Designing meaningful auditory feedback typically requires low-level programming, audio-specific expertise, and fragmented toolchains that hinder early-stage experimentation. We present TESI (Tangible and Embodied Sound Interaction), a modular toolkit that reduces these barriers by combining sensor-integrated hardware, open-source middleware, and a visual programming environment for real-time sensor-to-sound mapping. TESI enables interaction designers to prototype adaptive soundscapes across networked tangible interfaces with minimal setup. We describe the toolkit’s architecture, illustrate its use through examples, and discuss future directions for embodied sound design in distributed systems.

@inproceedings{DiMaggio2026a,
  title = {TESI: A Toolkit for Tangible and Embodied Sound Interaction},
  author = {Maggio, Francesco Di and Hengeveld, Bart and Funk, Mathias and Fiebrink, Rebecca},
  booktitle = {TEI '26: Twentieth International Conference on Tangible, Embedded, and Embodied Interaction, Chicago, IL, USA},
  publisher = {ACM},
  year = {2026},
  doi = {10.1145/3731459.3779334}
}