Authors - Mussa Turdalyuly, Aigerim Yerimbetova, Bakzhan Sakenov, Ulmeken Berzhanova, Orken Mamyrbaev, Duman Telman Abstract - In this paper, we examine the effectiveness of word- and phonemelevel automatic speech recognition (ASR) models for translating speech into sign language in the presence of limited language resources. A speech-to-sign system requires not only accurate transcription but also reliable preservation of semantic information for the correct selection of gestures. We focus on the Kazakh language and compare two ASR approaches based on the XLS-R architecture, namely, a word-level model and a phoneme-level model. Both models were studied and evaluated using the same Kazakh voice dataset and experimental settings. Its performance was assessed using traditional recognition metrics such as word error rate (WER) and phoneme error rate (PER), as well as gesture token accuracy (GTA), a follow-up evaluation index that measures the accuracy of gesture selection. The word-level ASR model achieved the best recognition performance with a WER of 0.360, while the phoneme-level model achieved a PER of 0.573. The phoneme-level model demonstrated excellent robustness in follow-up tasks despite low transcription accuracy and achieved high gesture identification accuracy of 0.673 versus 0.686 for the word-level model. These results suggest that phoneme-based expressions are suitable for supporting communication systems where preserving meaning is more important than accurate transcription. This study highlights the importance of evaluating speech recognition systems using the following target metrics in addition to traditional recognition metrics.
