Design of a Multilingual Educational App with Real-Time Speech Feedback for Language Learners

Abstract

This article reviews the design and assessment of a multilingual speech educational application which targets specific
skills like pronunciation and speaking through real-time speech feedback. Language learning tools have to be more
engaging and individualized owing to the explosion of digital learning resources that have stemmed from
globalization. The app in question is designed using concepts of automatic speech recognition (ASR) and natural
language processing (NLP) to evaluate users’ speech, provide corrective feedback, and refine content relevant to their
proficiency level and progress. The system enables multiple languages and a variety of skill levels which users can
select based on their preferences and needs, ensuring a broad user base. It describes the design process of a modular
system architecture based on a multilingual ASR phoneme error detection that includes a learner model for
individualized lessons. 120 English, Spanish, and Mandarin learners evaluated over the course of the study reported
improved engagement with the app, but more importantly, pronounced improvements in pronunciation accuracy over
traditional apps. The introduction suggests that the user's experience is enhanced by the implementation of real-time
feedback systems on mobile language applications, thereby enabling more effective language learning. This paper
addresses a gap in research in educational technology by exploring the application of processing strategies in the
development of personalized and adaptive systems for language instruction regardless of cultural and language
differences.