Authors - Aidana Karibayeva, Oleg Myssov Abstract - This paper presents a systematic experimental evaluation of the SeamlessM4T speech-to-text translation system on four Turkic languages — Kazakh, Tatar, Turkish, and Uzbek — across nine language pairs: kaz-tur, kaz-uzb, tat-kaz, tat-tur, tat-uzb, tur-kaz, tur-uzb, uzb-kaz, and uzb-tur. The study compared the base model (v1) and its fine-tuned version (v2). A closed parallel corpus of 1000 sentences per language pair was used. For evaluation, the BLEU, chrF, and WER metrics were used. The results showed a moderate but uneven improvement in scores after fine-tuning. The best results were observed in pairs where Uzbek was used as the source language (uzb–kaz: +0.78 BLEU; uzb–tur: +0.95 BLEU). However, in some Turkic language pairs, the scores decreased, which may be related to the model forgetting its prior knowledge during multi-task learning. When Tatar was the source language, both models performed poorly. This is explained by the scarcity of data and the phonological differences between languages. Furthermore, it was found that the model's architecture is limited when Tatar is used as the target language. The chrF metric was found to be more effective than BLEU for morphologically complex agglutinative languages. Therefore, it is recommended for use as the primary metric in the S2TT evaluation for Turkic languages.
