Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI

Fine-tune open source TTS model with Red Hat OpenShift AI

September 9, 2026
Dmytro Hryshchenko Abhijeet Dhumal
Related topics:
Artificial intelligence
Related products:
Red Hat OpenShift AI

Can't Read, Won't Buy. That is the title CSA Research gave its survey of 8,709 consumers across 29 countries, and the numbers justify it: 76% prefer to buy in their own language, and 40% will never buy in another. The same rule governs what your product says out loud.

Voice has become a core interface for customer service, accessibility, and digital products, but most open text-to-speech (TTS) models are trained primarily on English. Ask one to read Turkish and it often substitutes the nearest English sounds for letters such as ç, ğ, ı, ö, ş, and ü, producing speech that sounds fluent but is difficult for native speakers to understand.

In this post, we'll fine-tune the open source Orpheus-3B speech model for Turkish using Red Hat OpenShift AI and Kubeflow Trainer. We'll show how to package distributed training as a single TrainJob, and scale it across multiple nodes and GPUs. LoRA keeps the memory footprint under 16 GB, so any GPU with 24 GB or more of VRAM works comfortably, but the more GPUs you have the faster the job (we used 2 nodes with 2 A100 GPUs each and fine tuning completed in about 6 hours). The resulting model cuts speech errors by over 90% compared with the base model, and the same workflow can be applied to other languages simply by changing the training data.

How we taught the model to speak Turkish

20,000 Turkish sentences, each paired with a recording of a person speaking it, gradually pulled an English-trained model toward Turkish. On Red Hat OpenShift AI, that entire lesson is 1 job description: A TrainJob, a short file that tells the cluster what to train, on which data, and with how many machines. The steps below walk through the run in order - and at each step, the decision we made and why, so you can make different ones.

Step 1: Choose the base model

We fine-tuned unsloth/orpheus-3b-0.1-pretrained. Orpheus-3B writes audio the way a chatbot writes text: As a stream of tokens, just audio tokens instead of words. That's why we picked it. A model that trains like a chatbot can be trained with standard, well-established tools, with nothing custom to build. Canopy Labs also publishes a reference fine-tuning recipe, which our training script extends.

We also chose the pretrained checkpoint over the fine-tuned English one. It's less committed to English, which leaves more room for a new language. And 3 billion parameters is the sweet spot - big enough for natural speech, small enough to distribute across 2 nodes with 2 GPUs each. For your use case, any open TTS model that works with standard Hugging Face training tools fits this pipeline.

Step 2: Choose the dataset

We used afkfatih/turkish-tts-combined-raw, a public Hugging Face dataset of about 81,000 Turkish text-and-audio pairs used for training. Quality mattered more than quantity: A TTS model learns pronunciation from what it hears, so noisy audio or sloppy transcripts teach it the wrong lesson.

Picking a dataset for your language? Look for 4 things: Hours of speech, transcript accuracy, recording quality, and a license that permits your use. Training on 80,000 samples was our cost-quality trade, enough to more than halve the error rate and small enough for a single run across two nodes. More data pushes quality further, it just costs more GPU-hours.

Step 3: Prepare the data

The model cannot digest text and sound directly, so each pair is converted into tokens, the units it actually learns from. The recording becomes audio tokens (using a codec called SNAC), the sentence becomes text tokens, and the 2 are stitched into 1 sequence. This is illustrated in figure 1.

The generation path, from input text to playable audio.
Figure 1: The generation path, from input text to playable audio.

Here's what that looks like in the training function:

# Split the Turkish text into subword tokens using the Llama-3 tokenizer
t_ids = tokenizer.encode(text, add_special_tokens=False)
# Resample the audio to 24 kHz and compress it into three layers
# of SNAC codec codes (coarse -> fine detail)
l0, l1, l2 = _encode(wav, sr)
# Stitch text tokens and audio tokens into one training sequence.
# TOK_SOH/EOH wrap the text ("start/end of human"),
# TOK_SOA marks the start of the model's reply,
# TOK_SOS marks where audio begins.
seq = ([TOK_SOH] + t_ids + [TOK_EOT, TOK_EOH, TOK_SOA, TOK_SOS]
       + _interleave(l0, l1, l2) + [TOK_EOA])
# Mask the text portion so the model is scored only on the audio it produces
sos_idx = seq.index(TOK_SOS)
labels = [-100] * (sos_idx + 1) + seq[sos_idx + 1:]

What makes it all work is the masking - telling the trainer which tokens to grade and which to skip. During training, only the audio part is scored (the labels = -100 line above sets every text token to "do not grade"). The model learns exactly 1 skill: Given this Turkish text, produce this Turkish audio.

A detail to check before you launch: Your tokenizer, the component that splits text into tokens, must match your model. Mismatch it and nothing will crash - training runs, the loss falls, and the model still produces audio. It is confident gibberish, a data bug that looks exactly like a model failure. Preprocessing runs on the cluster as the first act of the job itself. When it finishes, the shared storage volume holds complete ready-to-train examples.

Step 4: Decide how much of the model to train

Full fine-tuning would update all 3 billion of the model's internal weights, and that needs far more GPU memory than 2 nodes offer. So we went one step cheaper, with a technique called low-rank adaptation (LoRA): Freeze the original model, and train a small set of add-on weights - roughly 3% of it - that steer the frozen rest. That cuts GPU memory and training time dramatically while reaching comparable quality.

lora_cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=lora_r,              # rank: capacity of the add-on weights (set to 64 in the TrainJob)
    lora_alpha=lora_alpha,  # scaling factor (set to 64 in the TrainJob)
    lora_dropout=lora_dropout,
    bias="none",
    # Attention + feed-forward layers - the parts that learn
    # which sounds map to which text
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora_cfg)

The training script logs exactly how small the trainable slice is:

LoRA: 97.3M trainable / 3.21B total (2.61% of params)

In business terms: A task that would need a rack of GPUs fits on any GPU with 24 GB or more of VRAM. We trained for 8 passes over the data. Every exact setting is in the repository.

For your use case: LoRA at this size is a solid default for teaching a model a new language. Give it more capacity - a higher rank learns more of the language but costs more GPU memory and risks overfitting on small datasets - or go to a full fine-tune, when you have more data and a higher quality bar. The same job runs either way.

Step 5: Describe the run as a TrainJob

Every choice so far - model, dataset, LoRA settings - lands in 1 file. You don't program the run, you state what you want, and Kubeflow Trainer makes it happen. The parts that matter:

apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainJob
metadata:
  name: orpheus-turkish-tts-v2
spec:
  runtimeRef:
    kind: ClusterTrainingRuntime
    name: torch-distributed
  trainer:
    numNodes: 2
    numProcPerNode: "gpu"
    command: ...
    podTemplateOverrides: ...
    env:
      - name: BASE_MODEL
        value: "unsloth/orpheus-3b-0.1-pretrained"
      - name: HF_DATASET
        value: "afkfatih/turkish-tts-combined-raw"
      - name: NUM_EPOCHS
        value: "8"
      - name: LORA_R
        value: "64"

A few things worth noting:

  • runtimeRef points at a ready-made Red Hat AI supported runtime image with PyTorch and CUDA.
  • numNodes: 2 tells Kubeflow Trainer to run across 2 machines and wire them together automatically.
  • Every tuning knob is an environment variable. Changing the language, dataset, or training length is a text edit, not a code change.
  • The training code starts from Canopy Labs' reference Orpheus fine-tuning recipe. We kept the LoRA rank and merge step, and made it run distributed.

As illustrated in figure 2, the 2 GPU pods share 1 storage volume and coordinate over the network:

2-node, 4-GPU distributed training on Red Hat OpenShift AI with Kubeflow Trainer v2.
Figure 2: 2-node, 4-GPU distributed training on Red Hat OpenShift AI with Kubeflow Trainer v2.

Step 6: Submit, and watch it learn

Submit the job, and the cluster does the rest:

oc apply -f manifests/trainjob-orpheus.yaml -n <namespace>

Watch the pods come up:

oc get trainjob orpheus-turkish-tts-v2 -n <namespace> -w

After a minute or 2, you see both nodes running:

NAME                      STATUS
orpheus-turkish-tts-v2    Running

Review the logs of the first pod, and notice that the opening lines confirm the model loaded and preprocessing completed:

oc logs orpheus-turkish-tts-v2-node-0-0 -n <namespace> -f

The output:

Loading model: unsloth/orpheus-3b-0.1-pretrained
LoRA: 97.3M trainable / 3.21B total (2.61% of params)
Train: 19000  Eval: 1000  Dropped: 42
Starting training ...

MLflow is the experiment tracker that ships with Red Hat OpenShift AI. Think of it as the training run's flight recorder: Every measurement lands there, live. How do you know it's working? 3 signals, in rising order of satisfaction. The loss curves (see figure 3) - the model's running error scores - fall steadily.

Figure 3: Training and evaluation loss falling over the run, tracked in MLflow.
Figure 3: Training and evaluation loss falling over the run, tracked in MLflow.

The pronunciation scores trend down. And best of all, every few hundred steps, the script records the model speaking 4 fixed Turkish announcements. You can hear the improvements (see figure 4).

Browsing per-step audio in the MLflow UI. Each checkpoint's Turkish output can be played and compared as training progresses.
Figure 4: Browsing per-step audio in the MLflow UI. Each checkpoint's Turkish output can be played and compared as training progresses.

 Step 7: Merge, then put it to the test

Training saves its progress as checkpoints. We take the best one and fold its LoRA add-on weights back into the base model, producing one ordinary, shareable model - nothing extra to carry around at serving time.

# Load the original 3B model
base = AutoModelForCausalLM.from_pretrained(
    args.base_model,
    cache_dir=args.hf_cache,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
# Load the LoRA add-on weights from the best checkpoint, merge them
# into the base model, and discard the adapter wrapper
model = PeftModel.from_pretrained(base, str(ckpt))
model = model.merge_and_unload()
# Save the result as a single standalone model
model.save_pretrained(str(out), safe_serialization=True)
tokenizer.save_pretrained(str(out))

Then the exam, in the form of a dictation test. The evaluation step loads both the original and the fine-tuned model and runs each through 10 Turkish sentences covering different registers - flight announcements, directions, news, weather, and more. 10 sentences is a small sample, so treat the numbers below as directional rather than precise, but the gap between baseline and fine-tuned is wide enough to be meaningful even at this size.

A listener transcribes what it hears. The listener is the Whisper open speech-recognition model, trained on 680,000 hours of audio, Turkish included - a consistent, impartial judge. The core of the test:

# Feed the Turkish text to the model and decode the audio tokens it produces
wav, elapsed = generate_audio(
    model, tokenizer, snac_model, text, device,
    baseline=(label == "baseline"),
)
# Whisper transcribes the generated audio back to text,
# then jiwer compares that transcript to the original sentence
wer, cer, transcript = compute_wer_cer(whisper_model, wav, text)

Whisper's mistakes become 2 scores: Word error rate (WER) measures the fraction of wrong words, and character error rate (CER) the fraction of wrong characters. Lower is better, and both can exceed 1.0 when the model inserts extra sounds that were never in the original text. We chose an automatic test deliberately: it is repeatable and cheap, where human listening panels are neither.

Testing the results with Whisper speech-recognition software.
Figure 5: Testing the results with Whisper speech-recognition software.

The results your stakeholders care about

Fine-tuning more than halved both error rates in the dictation test:

MetricBaseline (English model)Fine-tunedReduction
Word error rate (mean)2.440.3586%
Character error rate (mean)1.590.0994%
Evaluation loss9.503.9758%

WER and CER come from the 10-sentence dictation test. Evaluation loss comes from the 1,000-sample held-out split used during training.

What you can try next

We stopped at a fine-tuned Turkish model, but the pipeline is language-agnostic by design: Point HF_DATASET at a dataset in your language, optionally swap BASE_MODEL, and submit the same job again. Regional dialects or accents follow the same steps, though a dialect that differs significantly from the base model's training data may need a larger dataset or a different base model to get comparable results.

Beyond new languages, the same TrainJob is the starting point for several things we did not cover here:

  • A specific speaker's voice: With a few minutes of one person's audio, and the proper licensing and consent, the same job can adapt the model toward that voice.
  • The other direction: Speech to text. Whisper already played the listener in our dictation test (step 7). Fine-tuning a speech-to-text model - for a new language or a local accent - follows the same pattern: A single pipeline, both directions of the voice loop.
  • Serving the result: This project ends at a merged model on Hugging Face. The natural next step is deploying it behind an endpoint on Red Hat OpenShift AI - the Deploying models guide covers how.

Get started

Clone the repository, point the TrainJob at a dataset in the language you care about, and submit it. On OpenShift AI with Kubeflow Trainer, giving your product a new voice is a configuration change, not an infrastructure project.