Fine-Tuning a Large Multilingual Text-to-Speech Model for Nepali on Consumer Hardware: A Teacher Model for Knowledge Distillation with Speaker Identity Verification

Engineering notes

Engineering notes will be added by the aipentium editorial team.

Chinese explanation / 中文解读

中文解读待补充：本站会优先为大语言模型、生成式AI、ChatGPT相关技术、计算机视觉、深度学习等高价值论文补充中文说明。

Original abstract

Nepali is a low resource language for speech technology and there is very little open text-to-speech support for it. Most high quality neural TTS models are too large to run in real time on the low end machines that are common in Nepal, and the usual answer to that problem is knowledge distillation, where a large teacher model generates training speech for a small student model. That approach only works if the teacher is itself correct, because every error the teacher makes is copied into the student. This paper reports the construction and verification of such a teacher. A 937M parameter multilingual model, Indic Parler-TTS, was fine-tuned to a single Nepali female speaker identity using 4-bit quantization with a DoRA and RS-LoRA adapter of rank 32 applied only to the decoder, on a single laptop GPU with 6 GB of VRAM. The training data was 2,006 clips, which is 2.62 hours of licensed Nepali speech from 18 speakers, of which the target speaker contributed 496 clips or 35.8 minutes. The complete fine-tune used 2.48 GB of VRAM and 2,500 training steps. The fine-tune on these 2,006 clips succeeded. High frequency energy in the generated speech measures 0.3215 percent against the real speaker's 0.326 percent, so the output is spectrally matched to her recordings. A threshold-free blend prediction test shows the model favours the target speaker rather than averaging the corpus: the generated centroid scores 0.853 against her, while a constructed 18-way average of the corpus scores only 0.784, and a nearest-centroid assignment places 500 of 500 generated clips with the target speaker against a chance rate of 5.6 percent. A blind twenty clip listening comparison confirmed that the output is heard as one consistent woman. An earlier fine-tune, trained on a differently constructed dataset, had failed completely, and that failure is also reported because it is instructive: two data defects produced a voice nine times more muffled than the real speaker while character error rate stayed near 0.10 throughout, so every metric then in use stayed healthy through a total failure. The paper further reports that selecting a checkpoint by validation loss gives a worse voice than the final checkpoint, because validation loss over a speaker mixture is best for the average rather than best for the target, and that the teacher renders 100 percent of consonant conjuncts present in its fine-tuning data against 78 percent of those absent, which quantifies a generalization limit usually assumed away.

Links and sources

Comments