While based in Istanbul, we had the privilege of meeting people from the Laz Institute, an organization founded in 2013 dedicated to preserving, developing, and revitalizing the Laz language. They do extensive work: producing textbooks, organizing language courses (Laz is now an elective in some Turkish schools and universities), publishing books through the Lazika Yayın Kollektifi, and crucially for our purposes, contributing voices to Mozilla Common Voice.

The Institute also operates the LazuriTV YouTube channel, which hosts roughly 105 hours of spoken Laz content — about 4× the data available on Common Voice (28 hours). However, most of these videos lack transcriptions or subtitles.

Our previous involvement with speech recognition was a 2021 paper on low-resource ASR for Turkic languages. Out of three motivations — sharpening our own skills, learning the latest developments in the speech-to-text field, and wanting to help the Laz Institute eventually transcribe their YouTube archive to feed back into Common Voice — we decided to fine-tune a Whisper-small ASR model on the Common Voice Laz data.

This post documents how we did it. The resulting model is available on the Hugging Face Hub at Taruen/whisper-small-laz. (We've also built a browser-based Laz transliterator that converts Georgian Mkhedruli script into customizable Latin orthographies, with options for exactly the ejective and affricate consonants the model has to get right below.)

Why another guide?​

Great fine-tuning guides already exist:

1. HuggingFace's fine-tune Whisper post
2. Mozilla Data Collective's fine-tuning guide

So why write another? Besides showcasing our own work, we wanted to:

1. Show how to fine-tune Whisper on a language it didn't support at release — Laz isn't in OpenAI's original language list
2. Cover the workflow end-to-end with Mozilla Data Collective as the data provider, with all fine-tuning code in a single file you can tweak
3. Show how to run the pipeline on a proper GPU cloud when Colab/Kaggle hit their limits — which they will

The data: Common Voice Laz via Mozilla Data Collective​

Common Voice Laz has 34,909 recorded clips totaling around 28 hours. We accessed it via the Mozilla Data Collective Python SDK, which returns a pandas DataFrame with audio file paths and transcriptions plus a split column with Mozilla's official train / dev / test / validated assignments.

A crucial detail we initially missed: Mozilla speaker-separates these splits. We verified this explicitly:

# Verify zero speaker overlap across official splits
train_speakers = set(df[df['split']=='train']['speaker_id'].dropna())
test_speakers  = set(df[df['split']=='test']['speaker_id'].dropna())
dev_speakers   = set(df[df['split']=='dev']['speaker_id'].dropna())

print(f"train vs test overlap: {len(train_speakers & test_speakers)}")
print(f"train vs dev overlap:  {len(train_speakers & dev_speakers)}")
print(f"dev vs test overlap:   {len(dev_speakers & test_speakers)}")
# All zero — Mozilla's speaker separation holds

For Laz, the official splits are: train (5,051 rows, 7 speakers), dev (3,551 rows, 12 speakers), test (3,495 rows, 93 speakers), and validated (21,151 rows, 112 speakers — quality-checked but not assigned to a specific split). We used train + validated for training, dev for evaluation during training, and held test out entirely for final evaluation.

Adding Laz to Whisper's vocabulary​

Whisper doesn't know about Laz. The model uses special tokens like <|en|>, <|tr|>, <|ka|> to mark the target language. To add Laz, we added a custom <|lzz|> token and resized the model's embedding layer:

new_lang_token = "<|lzz|>"
processor.tokenizer.add_tokens([new_lang_token], special_tokens=True)
model.resize_token_embeddings(len(processor.tokenizer))

We then forced the decoder to start every transcription with our custom language token:

lang_id          = processor.tokenizer.convert_tokens_to_ids(new_lang_token)
task_id          = processor.tokenizer.convert_tokens_to_ids("<|transcribe|>")
no_timestamps_id = processor.tokenizer.convert_tokens_to_ids("<|notimestamps|>")

manual_forced_ids = [(1, lang_id), (2, task_id), (3, no_timestamps_id)]
model.config.forced_decoder_ids            = manual_forced_ids
model.generation_config.forced_decoder_ids = manual_forced_ids

The Colab debacle​

Our first attempt was on Google Colab Pro. The data preprocessing pipeline — which loads audio files, computes mel spectrograms, and tokenizes labels — hung indefinitely at 0% on A100 and A100 High-RAM runtimes. Same code worked on T4 instances. We initially blamed everything from PyArrow to HuggingFace datasets chunking before identifying the actual cause:

OpenMP thread pool deadlock in PyTorch's STFT operation. PyTorch's audio feature extraction uses OpenMP for parallelism, and on high-CPU instances the thread pool grew large enough to deadlock during fork operations. T4 instances escaped it because they have fewer cores and thus a smaller thread pool. The fix is one environment variable:

export OMP_NUM_THREADS=1

This is a known issue (PyTorch #17199) but not well documented in the context of Whisper feature extraction. It's also a good lesson: if your training script "just hangs" with no error on a beefier machine, suspect threading before suspecting your code.

After the deadlock fix, Colab still had issues — runtime disconnections overnight, lost checkpoints because we forgot to save to Google Drive. We pivoted to a proper GPU cloud.

Running on RunPod​

We moved to RunPod. An A100 80GB SXM instance at $1.49/hour ran the full pipeline in about 5 hours, for a total of ~$8 per training run. Compared to the gymnastics required to keep Colab alive overnight, an SSH-accessible VM with tmux and proper checkpoint persistence is a different world.

Setup:

# After SSHing into the RunPod instance:
pip install transformers accelerate evaluate jiwer tensorboard \
            datasets[audio] datacollective librosa pandas pyarrow

# Set the API key
export MDC_API_KEY=your_key_here
export OMP_NUM_THREADS=1

# Run inside tmux so a dropped SSH connection doesn't kill the job
tmux new -s laz
python3 train_whisper_laz.py
# Ctrl+B then D to detach

The full training script is included at the end of this post.

First run: catastrophic overfitting​

Our naive first run used a random 90/10 train/test split and no text normalization, weight decay, or audio augmentation. By step 1500, the training loss was approaching zero while validation WER had plateaued at 42%. Classic memorization.

Worse, the random 90/10 split meant the same speaker could appear in both train and test, contaminating the evaluation. The 42% number wasn't even an honest representation of generalization.

Second run: methodology improvements​

For the second run we applied several improvements:

1. Official Mozilla splits: Train on train + validated, evaluate on dev, hold out test entirely. Speaker-separated by construction.

2. Text normalization: Whisper outputs are cased and punctuated. If your dataset has inconsistent punctuation, the model wastes capacity (and gets penalized in WER) for guessing punctuation correctly. We strip punctuation and lowercase before training and evaluation:

def normalize_text(text: str) -> str:
    text = text.lower()
    text = re.sub(r"[^\w\s']", "", text)   # keep word chars, whitespace, and apostrophes (for Laz ejectives like k')
    text = re.sub(r"\s+", " ", text).strip()
    return text

3. Regularization: Added weight_decay=0.01 and turned on Whisper's dropout (off by default):

model.config.attention_dropout  = 0.1
model.config.activation_dropout = 0.1

4. Speed perturbation augmentation: Roughly half the time, randomly stretch audio to 0.9× or 1.1× speed. Free data multiplication, forces the model to learn speaker-rate-invariant features:

SPEED_RATES = [0.9, 1.0, 1.0, 1.1]   # 50% no augment

rate = SPEED_RATES[np.random.randint(len(SPEED_RATES))]
audio_array, sr = librosa.load(audio_path, sr=None, mono=True)
if rate != 1.0:
    audio_array = librosa.effects.time_stretch(audio_array, rate=rate)
audio_array = librosa.resample(audio_array, orig_sr=sr, target_sr=16000)

5. Earlier stopping: Capped at 2000 steps instead of 4000, since the first run made it clear nothing useful happens after ~step 1750.

Results​

The improved methodology produced these numbers on the held-out Mozilla test set (3495 examples, never seen during training):

The dev/test gap of about 2.4 points is normal and indicates the model isn't overfit to the dev set either. We were honestly surprised the model worked this well for a language Whisper had never seen — a testament to Whisper's strong multilingual phonetic priors.

Looking at actual predictions on the test set, most errors are diacritic confusions or single-character swaps:

ğormotik gamaǩçǩvidan        | ngormotik gamamçkvidan      ← 2 char errors
dido ini on                  | dido ini on                 ← perfect
aya lemşik va duçvinasinon   | aya lemşik va duçvinasinon  ← perfect
nana baba do bere isa renan  | nana baba do bere isa renan ← perfect
pucis tzǩuni uğun            | puciş tzǩuni uğun           ← single char swap

The character error rate would likely be much lower than the word error rate — maybe ~10-12% — since most errors are partial-word.

Trying it on real-world audio​

We ran the model on a 6-minute Laz story from the LazuriTV YouTube channel — out-of-domain conversational speech, not isolated Common Voice sentences. The output was fluent Laz throughout, with consistent diacritic usage and natural word boundaries:

heva ǩa xalaşi mtsxuli metasi çiruğai va moxtaşa opute mtsxuli do oşkuri mogorum badi şuri çirdumtu...

A native speaker would still find errors, but the model produces something that's recognizably and consistently Laz — not English-tinged gibberish or Turkish-leaking output.

What's next​

A few things we explicitly did not solve here:

Punctuation and case restoration. Since we normalized everything to lowercase without punctuation during training, this system is more accurately called an ASR (Automatic Speech Recognition) than a full STT (Speech-to-Text). True STT requires restoring sentence boundaries, capitalization, and punctuation — usually as a separate post-processing step using a small language model. We'll cover this in a future post.

Efficient deployment. Having a fine-tuned model is one thing; serving it efficiently for inference is another. Whisper-small runs comfortably on CPU but a production system would want batching, streaming, and possibly quantization. We'll cover that separately too.

Closing the data loop. The most impactful next step isn't a better model — it's more data. With a working (if imperfect) Laz ASR model in hand, the natural next step is to transcribe the LazuriTV YouTube archive and have Laz speakers correct the output. Correcting a 28% WER transcription is dramatically faster than transcribing from blank audio, so this human-in-the-loop workflow could realistically double or triple the training set. The Laz Institute is the right partner for this — they have both the linguistic expertise and the community connections to make it happen.

If you're a Laz speaker who would like to help, or an organization working with under-resourced languages and looking for similar work, get in touch.

The complete training script​

The full train_whisper_laz.py is available on our GitHub. The key sections:

# train_whisper_laz.py
import os
os.environ["OMP_NUM_THREADS"] = "1"   # critical: prevents PyTorch STFT deadlock

# ... [data loading, model setup, audio extraction, collator, metrics] ...

training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-small-laz",
    per_device_train_batch_size=64,
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=2000,
    fp16=True,
    eval_strategy="steps",
    eval_steps=250,
    save_steps=250,
    load_best_model_at_end=True,
    metric_for_best_model="wer",
    greater_is_better=False,
    weight_decay=0.01,
)

trainer = Seq2SeqTrainer(
    args=training_args,
    model=model,
    train_dataset=dataset["train"],
    eval_dataset=dataset["dev"],
    data_collator=data_collator,
    compute_metrics=compute_metrics,
    processing_class=processor.feature_extractor,
)

trainer.train()

Google Assistant немесе Yandex Alisa қазақ тілінде де сөйлесе, керемет болмас па еді?

Мұндай сөйлесетін ассистенттердің қажетті компоненті болып сөйлеуді тану (ағылшыншасы “speech recognition”) немесе сөйлеуді мәтінге (ағылшыншасы “speech-to-text”) деп аталатын жүйе табылады.

Машина оқыту методтарымен бір тіл үшін жақсы сапалы сөйлеу тану жүйесін болдыру үшін, ол тілде даустық деректердің аса үлкен көлемі (мыңдаған сағат, төменде қараңыз) болуы қажет. Бұдан басқа, ол деректер әртүрлі адамдар тарапынан жазылған болуы және транскрипцияланған болуы керек. Осы уақытқа дейін, тиісті лицензиялі аудиодеректердің болмауы себепті, көпшілік тілдер үшін ерікті / ашық бастапқы кодты сөйлеуді тану жүйесі қол жетімсіз болып қала береді.

Бақытымызға орай, бұл жағдайды жақсы жаққа өзгертуге болады. 2018 жылы Mozilla, Firefox веб-браузері және көптеген басқа бағдарламалардың артындағы компания, Common Voice (“Халық дауысы”) жобасын іске қосты.

Common Voice деген не?​

Міне, ол жобаның About бетінен бір үзінді:

Common Voice — бүкіл әлемдегі ерікті үлес қосушылардың дауыстарынан құралған, жалпыға қолжетімді дауыс деректер жиынтығы. Дауыстық бағдарламалар жасағысы келетіндер бұл деректер жиынтығын машина оқыту модельдерін үйрету үшін қолдана алады.

Сіз қалай көмектесе аласыз?​

Біз Taruen ретінде қазақ тіліндегі Common Voice-ты іске қосқымыз келеді, және бұл мақсатқа қарай алғашқы қадамдарды жасадық. Бірақ Common Voice-та қазақ тілі де іске қосылсын үшін, қазақ тілін жақсы білушілердің көмегі керек. Егер қазақ тілі ана тіліңіз болса, мына қалай көмектесе аласыз:

1. Common Voice сайты Pontoon атты Mozilla-дың локализация құралында дұрыс аударылған ма жоқ па екенін тексеріңіз. Аудармалардың шамамен үштен бірін біз — яғни қазақ тілі ана тілі болмаған адамдар — үстедік, сондықтан олар қателі болуға мүмкін.

2. Біз Common Voice-тың сөйлем жинау құралына жіберген сөйлемдерді тексеріңіз. Бір тіл Common Voice-та іске қосылсын үшін, ол тілде ең азы 5000 тексерілген сөйлем болу қажет.

Бұл сөйлемдерді біз М. Әуезов атындағы Әдебиет және өнер институты жариялаған “Бабалар сөзі” деп аталатын, қазақ халық шығармашылығы туындыларын өз ішіне алған мықты 100 томдықтың 65 және 68 томдарынан алдық (65-68 томдарда мақалдар жиналған). Қазақстан Республикасының Авторлық құқық туралы заңының 8-нші бабына сәйкес, фольклор шығармалары авторлық құқықтан азат және осылайша қоғамдық байлық (public domain) болып тұр. Демек, оларды Common Voice-та қолдануға болады. Сонымен, 65-68-нші томдар ішіндегі мақал-мәтелдер Common Voice жобасының басқа критерийлеріне де сәйкес келеді — оларда сандар, шетелдік әріптер және рұқсат етілмеген басқа белгілер жоқ, олар әдетте қысқа, педагогикалық / қызықты және осылайша оқу үшін көңілді.

Common Voice талап ететін орфография мен грамматиканың дұрыстығын тексеруден басқа, біз сізден ол сөйлемдер кімді де болса ренжітпейтінін тексеруіңізді өтінеміз. Басқаша айтқанда, ол сөйлемдерде ерлерге, әйелдерге, ата-аналарға, балаларға, діни адамдарға, діни емес адамдарға, Оңтүстік Қазақстандықтарға, Солтүстік Қазақстандықтарға, Батыс Қазақстандықтарға, Шығыс Қазақстандықтарға, Қытайдың қазақтарына… түсінесіз шығар — тіл тигізетін заттардың болмауы қайырлы. Ол сөйлемдерден көз жүгірткенде ондайларын көрмедік, бірақ бұл сұраққа да қазақша жақсы білетін адамдар ғана жауап бере алады.

Бірге, қалаған әр адам да қазақ тілін түсінуші сөйлеу тану жүйесін болдыра аларлық көлемде аудиодеректер жинай аламыз деген үміттеміз. Жобаның жиі қойылатын сұрақтар бетінде мозиллашылар жақсы сапалы сөйлеу тану жүйесін нөлден үйрету үшін шамамен 10000 сағат тексерілген аудиожазбалар қажет деп жазады. Демек бұл — мақсат. 10000 сағат қолжетпестік көп сияқты, бірақ 10 мың сағатты қазақша білушілердің ең консерватив санына бөлсек те, кісі басына шамамен 4 секунд аудио тура келеді. Ал 4 секунд — ол әлдеқайда азырақ қорқынышты сан.

(Жоғарыдағы Жаңартуда айтылғандай, Whisper сияқты алдын ала үйретілген модельді файн-тюнинг жасауға әлдеқайда аз дерек керек — бірақ дерек көп болғаны еш зиян етпейді, әрі үлкен, ашық лицензиялы қазақ дауыс деректер жиынтығы модельдер қалай үйретілсе де құнды болып қала береді.)

Wouldn’t it be great, if Google Assistant or Yandex Alisa spoke Kazakh?

One necessary component of such speech-enabled digital assistants is a so-called automatic speech recognition (ASR) or speech-to-text (STT) system.

Large amounts of audio data (thousands of hours, see below), from many different people, along with transcriptions, are needed to train a good speech-to-text system using machine learning methods. So far, due to lack of appropriately licensed, freely available audio data in them, building a high-accuracy free/libre/open-source software (FLOSS) speech recognition system is out of reach for most languages.

Fortunately, there is a way to change this for the better. In 2018, Mozilla, the company behind the Firefox web-browser and many other programs, launched the Common Voice project.

What is Mozilla Common Voice?​

Here is a quote from the project's About page:

Common Voice is a publicly available voice dataset, powered by the voices of volunteer contributors around the world. People who want to build voice applications can use the dataset to train machine learning models.

How you can help​

We at Taruen want to launch Common Voice in Kazakh, and took the first steps towards that goal. But for that to happen, we need help from native speakers of Kazakh. If you are one, here is how you can help:

1. Review whether the Common Voice website has been correctly translated into Kazakh on Pontoon, Mozilla’s localization tool. About one-third of the translations were authored by us — non-native Kazakh speakers — and thus might be incorrect.

2. Review Kazakh sentences we’ve submitted to the Common Voice Sentence Collector tool. At least 5000 reviewed sentences are needed to “launch” a language on Common Voice.

These sentences were taken from Volumes 65 and 68 of the mighty 100-volume set with works of Kazakh folklore, called “Бабалар сөзі” and published by M. Auezov Institute of Literature and Art. By Article 8 of Kazakhstani Copyright Law, works of folklore are exempt from copyright and are thus in the public domain and suitable for submitting to Common Voice. Moreover, proverbs from volumes 65 and 68 match other criteria of the Common Voice project as well — they don’t contain digits, foreign letters and other symbols not allowed in the dataset, they are mostly short, arguably pedagogical/entertaining and thus fun to read.

Besides usual checks for correct spelling and grammaticality required by Common Voice, when reviewing sentences, we also ask you to make sure that none of the sentences are remotely offensive to men, women, parents, children, religious people, non-religious people, Southern Kazakhstanis, Northern Kazakhstanis, Western Kazakhstanis, Eastern Kazakhstanis, Kazakhs of China, Kazakhs of … You get the idea. A glimpse over the sentences did not suggest that they would contain anything like that, but again, that’s something for native speakers to judge.

We hope that together we can assemble enough data so that anyone who wishes to do so can train a speech-to-text system for Kazakh. On the FAQ page of the project, Mozillians mention 10000 hours as an approximate number of validated hours needed to train a production speech-to-text system from scratch, so that’s something to strive for. It does sound like a lot, but when divided among even a conservative number of Kazakh speakers, it requires each person to record sentences for about 4 seconds. That is a much less scary number.

(As the Update above notes, fine-tuning a pre-trained model like Whisper needs far less data — but more data never hurts, and a large, openly-licensed Kazakh voice dataset is valuable regardless of how models are trained.)