Even as prompt engineering and model orchestration get more sophisticated, there are tasks where fine-tuning a smaller, specialized large language model (LLM) is the more practical path. It's how you teach a model to handle your specific data, your domain, or your format. Fine-tuning works, until it quietly breaks something you didn't test.
You train a model on your data, try a few prompts, and everything looks great. Then you deploy it and discover it's forgotten how to do basic math. Or it stopped following safety guidelines. Or it lost a language it used to handle fluently. The new skill landed, but something old disappeared.
This is not a fluke, and it's not something you can fix by tweaking your learning rate. It's a fundamental problem with how Supervised Fine-Tuning (SFT) and Low-Rank Adaptation (LoRA) work. Orthogonal Subspace Fine-Tuning (OSFT) addresses it.
The problem with SFT and LoRA
When you fine-tune a model with SFT or LoRA, the training process changes the model's internals to fit your data. The issue is it has no way of knowing which parts are responsible for capabilities you care about keeping. It overwrites whatever it needs to learn the new task.
This creates a tradeoff hard to manage: The better the model gets at your task, the more of its original abilities it risks losing. Researchers call this catastrophic forgetting, and it's not an edge case.
In the OSFT paper, Sculpting Subspaces (published at the International Conference on Learning Representations (ICLR) 2026), researchers fine-tuned a model that could solve 29% of grade-school math problems on eight new tasks. Afterward, its math score dropped to 2%. The fine-tuning succeeded on every new task and destroyed a skill the model already had.
LoRA reduces this somewhat because it changes less of the model overall, but it doesn't solve the core problem. It still has no mechanism for protecting what the model already knows.
What OSFT does differently to fix catastrophic forgetting
Orthogonal Subspace Fine-Tuning adds one thing missing in SFT and LoRA: protection for existing capabilities.
Before training starts, OSFT figures out which parts of the model are responsible for its current skills and locks them. Training then proceeds normally, but it's only allowed to update the parts of the model that won't interfere with what's already there.
A simple analogy: Imagine the model's knowledge is written on a whiteboard. SFT and LoRA erase parts of the board to make room for new notes. OSFT finds blank space on the board and writes there, leaving everything else untouched.
The OSFT paper tested this on TRACE, a benchmark chaining eight diverse tasks together: domain-specific question and answer (Q\&A), multilingual text, code, and arithmetic. As shown in Figure 1, Standard fine-tuning left the model at around 23% average accuracy and made it worse on earlier tasks. OSFT, using the same model and data, reached 48.4% and slightly improved earlier tasks. It didn't forget less; it learned more overall because it wasn't destroying its own previous work.
When you should use OSFT instead of SFT or LoRA
OSFT is most valuable when you're fine-tuning a model already doing useful things and you can't afford to lose those abilities:
- You're adding domain knowledge (medical, legal, financial) to a general-purpose model and need it to keep its conversational and reasoning abilities.
- You're training on multiple datasets in sequence, and each round of training undoes some of what came before. For example, you add a new language, then domain-specific Q\&A, and then a custom output format.
- Your users depend on a base model's existing skills across many tasks, not only the one you're fine-tuning for.
If you're fine-tuning for a single narrow task and don't need the model's other abilities, SFT or LoRA might still be the simpler choice. OSFT's advantage is specifically in preserving what the model already knows.
How to fine-tune models with OSFT in Training Hub
Training Hub is an open source, algorithm-centered Python library for LLM post-training, fully supported and available for enterprise customers within Red Hat OpenShift AI. If you already use Training Hub for SFT, switching to OSFT is straightforward. The API is the same: You call osft instead of sft and set one additional parameter:
from training_hub import osft
osft(
model_path="meta-llama/Llama-3.1-8B-Instruct",
data_path="./my_data.jsonl",
ckpt_output_dir="./checkpoints",
unfreeze_rank_ratio=0.3,
)
The unfreeze_rank_ratio parameter controls how much of the model is open to training. At 0.3, roughly 70% of the model is reserved for existing capabilities and 30% is available for learning your new data. This is a good starting point: enough room to learn your task without putting too much of the original model at risk. Your data format, your configs, your infrastructure: all of it stays the same.
OSFT supports the same model families you already fine-tune: Qwen, Llama, Granite, Gemma, gpt-oss, Phi, and others.
Because Training Hub is also included in Red Hat OpenShift AI, you get it preinstalled and ready to run inside your workbench images alongside the rest of your machine learning (ML) stack. When you're ready to move beyond local experiments, this guide walks through scaling Training Hub from a notebook to distributed, production-grade training jobs.
Get started
Ready to try it yourself?
- Run through our OSFT example notebook to fine-tune Llama-3.1-8B-Instruct on your own data.
- Dive into our comprehensive tutorial for a step-by-step look at how to tune your parameters.
