A Phygital Game Prototype to Support Speech-Therapy-in-the-Loop AI in Pediatric Rehabilitation

Speech perception is one of the fundamental prerequisites for correct language development in childhood. This process may be compromised in children with Speech Sound Disorders (SSD), for whom early speech therapy intervention is essential. In this context, play-storytelling may constitute privileged intervention tools, given it can simultaneously activate physical, cognitive, and socioemotional dimensions in child development, making it a speech therapist’s ally in repetitive activities of rehabilitative practices. This work-in-progress paper presents a phygital game designed to improve phoneme recognition in children with SSD. Grounded in the principles of Self-determination Theory (SDT) combined with TRACE Speech Perception Theory, the game aims to assist the speech perception activities using a physical component, consisting of composable phoneme cards, and an Embodied Conversational Agent (ECA) acting as mediator, storyteller, and motivator. A conceptual framework on the use of an Embodied Conversational Agent (ECA) in a phygital environment associating Autonomy, Competence and Relatedness (SDT theory) to categorical perception in speech rehabilitation is presented, involving a triadic relationship among child, game-based ECA phygital intervention and the therapist. This contribution include a conceptual model framework on the use of ECAs in a phygital environment and a prototype system aimed at recognizing and distinguishing phonemes during speech-language rehabilitation sessions.