Atos, a global Red Hat Partner, recently completed a groundbreaking proof of value (PoV) for a German public sector customer, and what they found challenges conventional AI wisdom: Bigger isn't always better. Using Red Hat OpenShift AI and lean infrastructure, a fine-tuned 1-billion parameter model outperformed GPT-4.1 on a complex real-world task, proving that specialized small language models (SLMs) can act as domain experts.
When frontier models aren't the answer
The customer faced a unique set of requirements that made traditional AI solutions impractical:
Data sovereignty requirements
As a German public sector organization, they needed self-hosted models only. Cloud-based frontier models were not an option.
Multilingual complexity
The data consisted of complex German public social media comments with nuanced language, slang, and spelling variations and a focus on hate speech.
Beyond RAG's capabilities
The problem couldn't be solved by simply providing more context through retrieval augmented generation (RAG). The models needed to truly understand and extract structured information.
Cost and latency sensitivity
At scale, using frontier models would have been prohibitively expensive and slow.
Meeting the challenge
The core challenge is performing structured analysis on vast amounts of unstructured German public social media data with 2 critical goals:
Goal 1: Structured feature extraction
Automatically extracting 14 distinct features (entity extractions like persons and locations or linguistic features like slurs, tonality, vulgarity, linguistic markers, spelling errors) into strictly valid JSON format.
Goal 2: Violence detection
Accurately identifying calls to action for violence and specific triggers.
Solution: Synthetic data generation and specialized fine-tuning on OpenShift AI
The solution leveraged Red Hat's AI framework combining synthetic data generation with targeted fine-tuning:
Architecture flow
Here's the logic used for the workflow, with each stage explained in detail below:
Raw public social media data → SDG structured extraction flow → synthetic dataset → training hub → KFTrainer job on Red Hat OpenShift AI → tuned small models
- SDG hub: Used an out-of-the-box flow for generating structured insights data. A "teacher" model (GPT-4.1 accessed by API) generated high-quality synthetic training data from the public social media dataset, with specialized prompts for each component.
- Training hub: Applied supervised fine tuning (SFT) and orthogonal subspace fine tuning (OSFT) algorithms.
- Model customization: 3 student models were fine-tuned using synthetic data from Phi-4 Mini 3.8B, Qwen 2.5 1.5B, and Qwen 2.5 7B.
- Development environment: OpenShift AI with Workbenches and Jupyter notebooks served as the core IDE for both data generation and model training. Training jobs were launched using KFTrainer and Kubeflow SDK.
Lean infrastructure: High performance without massive hardware
A major win was proving that high-performance AI doesn't require massive resources.
- No GPU used for data generation: Teacher LLM accessed by API
- Training compute: Only 3 NVIDIA L40 GPUs
- Platform: OpenShift AI for resource scalability and model experimentation
A custom rubric for rigorous assessment
A key innovation of the project was developing a custom rubric system specifically designed to measure performance on this complex multilingual task. Rather than relying on generic benchmarks, the team created a domain-specific evaluation framework.
Custom evaluation data
The team generated custom eval data that accurately reflected the real-world challenge: German social media comments with all their complexity — slang, spelling variations, contextual nuance, and violent rhetoric. This ensured that model performance metrics would translate directly to production use.
The 34-point scoring system
Each model was evaluated on a comprehensive custom rubric totaling 34 points:
- Information Extraction (28 points): 2 points for each of the 14 correctly extracted features — slurs, tonality, vulgarity, violence triggers, spelling errors, groups, locations, and more. This granular breakdown allowed precise measurement of where models excelled or struggled. Here, 2 points are granted for a completely correct answer, while 1 point was given for a partially correct answer and 0 points for a wrong answer.
- JSON correctness (3 points): Ensuring output is parsable and all required fields are present — critical for downstream processing. This is binary, so a model output either received 3 points for a correct JSON or 0 for an invalid JSON.
- Violence detection (3 points): Classification accuracy against a human-validated gold standard. Here, also either 3 or 0 points were given.
This custom rubric captured what mattered for production: not just accuracy on isolated tasks, but reliable performance across the entire structured extraction pipeline.
Comprehensive testing scope
The evaluation covered an impressive range of models and configurations:
- 27 fine-tuned model variations (3 student models × 9 variations including SFT and OSFT across epochs)
- 12 base/reference baselines (including NuExtract 2B, 8B, GPT-4.1 and the un-tuned student models — each with different prompting strategies)
- Total: 39 evaluation runs
Small models, expert performance
Before fine-tuning, the base student models struggled with basic structural requirements. The Phi-4 Mini base model produced valid JSON only 1% of the time. After just 1 epoch of fine-tuning, JSON validity jumped to 92% and reached ~99% with further training.
Outperforming GPT-4.1 on certain tasks
The results were significant:
- JSON validity: While GPT-4.1 achieved 96% JSON validity, fine-tuned Phi-4 Mini and Qwen 2.5 7B both reached 99% (see figure 1).
- Domain-specific tasks: GPT-4.1 scored lower on nuanced German language tasks like adjectives (1.68) and spelling errors (1.86), whereas the customized models matched or exceeded these scores.
Overall performance on custom rubric
In summary, evaluation scores (figure 2) included:
- GPT-4.1 with short prompt: ~29 points
- Fine-tuned Phi-4 Mini 3.8B: ~30 points (5% improvement over GPT-4.1 baseline)
- GPT-4.1 with detailed prompt: 30.5 points
- Fine-tuned Qwen 2.5 7B: 30.8 points (exceeding even the detailed GPT-4.1)
- All SFT/OSFT variations: Consistent 29-30.8 point range
Hard skills mastery
All fine-tuned models, including the tiny 1.5B-class QWEN 2.5, achieved near-perfect mastery in:
- JSON formatting
- Entity extraction
- Violence detection
Task difficulty: Extraction vs. understanding
The custom evaluation data revealed important insights about different types of AI tasks:
- Interpretative tasks are hard: Across all models, interpretative tasks like determining tonality and generating a summary were significantly more challenging than entity extraction. This highlights a fundamental difference between recognizing concrete information and understanding contextual nuance.
- Extraction vs. understanding gap: NuExtract, a model specifically designed for extraction, demonstrated this pattern clearly — performing strongly on concrete entities (companies, persons, locations) but dropping significantly on contextual tasks like summary and tonality. Fine-tuned models, however, excelled at both types of tasks, demonstrating true domain understanding rather than pattern matching.
Why fine-tuning changes everything
There were some surprises that nobody expected in the findings.
From 1% to 99%: The power of domain-specific training
Before fine-tuning, the Phi-4 Mini base model produced valid JSON only 1% of the time. After tuning, it reached 99% validity. This dramatic improvement demonstrates how fine-tuning embeds desired behaviors directly into model weights, creating reliable, production-ready systems.
Custom evaluation frameworks matter
Generic benchmarks don't capture domain-specific requirements. By creating a custom rubric system and custom eval data, the team could accurately measure what mattered for production: multilingual extraction accuracy, structured output reliability, and contextual understanding. This evaluation approach revealed performance differences that standard benchmarks would miss.
Lean infrastructure requirements enable production deployment
Tuning small models required only modest infrastructure — 3 GPUs maximum. This isn't just about cost savings during training. It means:
- Rapid iteration cycles
- Accessible experimentation for domain experts
- Practical on-premises deployment
- Lower barriers to customization
Compare this to the massive compute required for training or even running frontier models at scale. Especially when running at scale, the small models excel as they require even less infrastructure, once tuned.
The agentic use case
These insights extend directly to agentic use cases, where the advantages become even more pronounced:
Agents demand consistent, schema-aligned behavior, not general intelligence. In multi-agent systems, reliability trumps versatility. An agent that produces valid JSON 99% of the time instead of 96% of the time is the difference between a production system and a prototype. An agentic workflow requires predictable, structured output that can be reliably parsed and processed by downstream components.
Agents are heavily orchestrated around narrow tasks. Most agentic use cases involve specialized roles—data extraction, classification, validation, transformation. In these orchestrated workflows, most of a large model's general capability remains unused. You're paying for billions of parameters when you only need millions, fine-tuned for your specific task.
As AI systems evolve toward multi-agent architectures, the winning approach won't be deploying GPT-4 clones for every task. It will be orchestrating ensembles of small, fine-tuned models — each an expert in its narrow domain, running efficiently on modest hardware, producing reliable structured outputs. The agentic future is small and specialized.
This PoV demonstrated exactly that: Specialized models acting as expert agents, producing consistent structured data, deployed on lean infrastructure, with better domain performance than general-purpose frontier models.
Strategic impact
This project demonstrates several critical advantages for organizations with similar requirements:
Data sovereignty and security
By using specialized SLMs that can be deployed on-premises with OpenShift AI, organizations maintain complete control over sensitive data — essential for public sector and regulated industries. In these areas it is often simply not allowed to use models hosted publicly with an API. Here, specialized SLMs open a door that was previously completely shut.
Efficiency and cost
AI costs are rising, but this method utilizes reduced resource consumption to enable local deployment, and a significantly lower inference costs at scale. The result? Faster response times compared to API-based frontier models.
Robustness and performance
There are specific advantages you can put to use to optimize your AI workflows:
- Domain-specific fine-tuning increases model robustness
- Enables complex, agent-based solutions
- More reliable structured output for downstream processing
Competitive advantage
For resource-constrained, on-premises environments, specialized SLMs are far more valuable than large generalist models. They provide expert-level performance in specific domains without the overhead.
Key takeaways
- Custom evaluation frameworks reveal true performance: Domain-specific rubrics and eval data capture what generic benchmarks miss.
- Customization beats scale for specific use cases: A 1B parameter model, when properly fine-tuned, can outperform 100B+ parameter frontier models in domain-specific tasks.
- Modest hardware requirements: High-performance model training on OpenShift AI required only 3 GPUs — making customization accessible and practical. For SLM-inference, the requirements are even lower.
- Synthetic data generation is powerful: Using a teacher model to generate high-quality training data from public social media datasets enables rapid customization without extensive manual annotation.
- Data sovereignty is achievable: Organizations with strict data residency requirements can achieve state-of-the-art results with self-hosted solutions.
- The agentic future is specialized: Multi-agent systems benefit more from small, reliable, task-specific models than general-purpose frontier models.
What's next?
This successful PoV opens the door for production deployment and expansion to additional use cases where data sovereignty, multilingual complexity, and cost efficiency are paramount. The principles demonstrated here—fine-tuning small models on Red Hat OpenShift AI for specific tasks—provide a blueprint for building reliable, efficient agentic systems.
Interested in learning more about the specific methodology, datasets, or reproducing these results? Reach out to discuss how specialized small language models might solve your organization's unique AI challenges.
