Run LoRA fine-tuning on Red Hat OpenShift AI with Ray

Ray has emerged as a leading open source distributed compute framework for scaling enterprise AI workloads. After its donation to the PyTorch Foundation in 2025, adoption accelerated across the industry, and Red Hat OpenShift AI includes prebuilt, fully supported Ray cluster runtime images.

Training Hub is a Python package providing a standardized set of fine-tuning algorithms for large language models (LLMs). It supports supervised fine-tuning (SFT), offline supervised fine-tuning (OSFT), Low-Rank adaptation (LoRA), and group relative policy optimization (GRPO). Starting with Red Hat OpenShift AI 3.5, Training Hub is preinstalled in a Ray CUDA runtime image, meaning you can run any of these algorithms on a Ray cluster without pip installs, without dependency management, and in air-gapped environments.

This article walks you through running a LoRA fine-tuning job on Ray using Training Hub and the CodeFlare software development kit (SDK). You'll submit a training job fine-tuning Qwen2.5-1.5B-Instruct for SQL generation and evaluate the fine-tuned model, all from a Jupyter notebook.

You can find complete step-by-step notebooks for all 4 algorithms in the Red Hat AI examples repository. Clone the repo and follow along.

Why Ray for fine-tuning?

You might already be using the Kubeflow Training backend for SFT, OSFT, and LoRA fine-tuning on Red Hat OpenShift AI. Ray provides a second backend with its own strengths:

• Elastic scaling. Ray clusters scale dynamically. If your job needs 4 GPUs, Ray provisions 4 workers; when it finishes, those resources are released.
• Unified runtime. Data preprocessing, training, and serving can all run on the same Ray cluster, simplifying multi-stage pipelines.
• verl integration. OpenShift AI includes verl, an open source library for distributed reinforcement learning. It coordinates the rollout phase (where the model generates trial answers) and the training phase across your Ray cluster automatically.

Both backends consume the same algorithms (SFT, OSFT, LoRA, and GRPO) from Training Hub, Red Hat's unified fine-tuning interface, and produce the same fine-tuned models. Choose based on your infrastructure:

The fine-tuned models are interchangeable. A LoRA adapter trained on Ray works identically to one trained on Kubeflow Trainer. If you already have Ray clusters for other workloads, fine-tuning fits in naturally.

What is available

Red Hat OpenShift AI 3.5 ships 4 fine-tuning algorithms on Ray:

Each algorithm has a complete working example in the Red Hat AI examples repository, including data preparation, cluster configuration, training, and evaluation.

Prerequisites

Before starting, ensure your environment meets the following infrastructure and developer requirements.

Cluster administrator requirements

You need access to a Red Hat OpenShift cluster with the following:

• Red Hat OpenShift AI (version 3.5 or later) with the dashboard, workbenches, and Ray components enabled.
• An OpenShift storage provisioner supporting dynamic provisioning of persistent volume claims (PVCs) with ReadWriteMany (RWX) access mode.
• A worker node with an NVIDIA GPU (A100 or H100 recommended). For multi-GPU distributed training, you need a node with multiple GPUs or multiple GPU nodes.

Developer requirements

• Access to an OpenShift AI data science project.
• Permission to launch workbenches and create PVCs within your namespace.

Set up your environment

Prepare your Red Hat OpenShift AI environment by launching a workbench and configuring shared persistent storage.

Create a workbench

Access the Red Hat OpenShift AI dashboard from the top navigation bar of the OpenShift web console. Navigate to Data Science Projects and create a new project (Figure 1).

Select Create a workbench, then choose a standard data science notebook image (such as Standard Data Science or any image including the CodeFlare SDK). The workbench is your control plane. You use it to submit and monitor Ray jobs, not to run the training itself (Figure 2).

Create shared persistent storage with RWX access mode (50 GiB recommended) for the model, dataset, and training checkpoints (Figure 3).

Once OpenShift provisions your workbench, select Open to launch JupyterLab.

Clone the example notebook

From the workbench, clone the examples repository:

git clone https://github.com/red-hat-data-services/red-hat-ai-examples.git

Navigate to examples/fine-tuning/lora_ray/ and open the LoRA notebook. (For other algorithms, navigate to the corresponding sft_ray/, osft_ray/, or grpo_ray/ directory.)

Prepare the model and data

The notebook begins by configuring paths and hyperparameters:

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
MODEL_PATH = f"{PVC_MOUNT_PATH}/{MODEL_ID}"
DATA_PATH = f"{PVC_MOUNT_PATH}/lora_text_sql_output/train_data.jsonl"
CKPT_DIR = f"{PVC_MOUNT_PATH}/checkpoints/lora"

LORA_R = 16
LORA_ALPHA = 32
MAX_SEQ_LEN = 512
LEARNING_RATE = 1e-4
NUM_EPOCHS = 1

It then downloads the base model to the shared PVC:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id=MODEL_ID,
    local_dir=f"/opt/app-root/src/{PVC_PATH}/{MODEL_ID}",
    token=HF_TOKEN or True,
    resume_download=True,
    local_dir_use_symlinks=False,
)

We'll use the public b-mc2/sql-create-context dataset, which pairs natural language questions with valid SQL queries. The notebook formats these pairs into conversational prompts and saves them as JSONL files to your shared storage.

Configure the Ray cluster

The CodeFlare SDK provides a Python API for creating Ray clusters on Red Hat OpenShift AI. The notebook uses ManagedClusterConfig to define a single-GPU cluster. LoRA is parameter-efficient and runs well on 1 GPU:

from codeflare_sdk import ManagedClusterConfig, RayJob
from kubernetes.client import (
    V1PersistentVolumeClaimVolumeSource,
    V1Volume,
    V1VolumeMount,
)

pvc_volume = V1Volume(
    name="training-data",
    persistent_volume_claim=V1PersistentVolumeClaimVolumeSource(claim_name=PVC_NAME),
)
pvc_mount = V1VolumeMount(name="training-data", mount_path=PVC_MOUNT_PATH)

cluster_config = ManagedClusterConfig(
    image=IMAGE,
    num_workers=0,
    head_cpu_requests=4,
    head_cpu_limits=8,
    head_memory_requests=64,
    head_memory_limits=64,
    head_accelerators={"nvidia.com/gpu": 1},
    volumes=[pvc_volume],
    volume_mounts=[pvc_mount],
    envs=env_vars,
)

ManagedClusterConfig provisions a single-node cluster mounting your shared storage. Because the Ray CUDA container preinstalls the Training Hub runtime, all 4 fine-tuning algorithms run without manual dependency installation.

Build the entrypoint and submit the job

The training entrypoint calls training_hub.lora_sft() directly with the parameters configured earlier:

entrypoint = f'''python -c "
from training_hub import lora_sft

lora_sft(
    model_path='{MODEL_PATH}',
    data_path='{DATA_PATH}',
    ckpt_output_dir='{CKPT_DIR}',
    num_epochs={NUM_EPOCHS},
    max_seq_len={MAX_SEQ_LEN},
    learning_rate={LEARNING_RATE},
    lora_r={LORA_R},
    lora_alpha={LORA_ALPHA},
)
"'''

Submit the job using the CodeFlare SDK's RayJob API:

job = RayJob(
    job_name="lora-sft-training-hub",
    entrypoint=entrypoint,
    cluster_config=cluster_config,
    namespace=NAMESPACE,
    ttl_seconds_after_finished=600,
)

job.submit()

The SDK creates a RayJob custom resource with an embedded RayCluster spec. KubeRay spins up the cluster, runs the entrypoint on the head node, and tears everything down when the job finishes.

Monitor training progress

You can check job status directly from the notebook:

job.status()

The status might show as PENDING while the RayCluster is spinning up and pulling the image. LoRA training is fast. With default parameters on a single L40S (48 GB), expect approximately 5–15 minutes depending on dataset size. You should see the loss decrease steadily over the course of training.

Evaluate the fine-tuned model

After training completes, the job saves the LoRA adapter to the shared PVC. The notebook loads the adapter, merges it with the base model, and tests SQL generation with a set of example prompts. To compare model responses before and after fine-tuning, you can first test the unmodified base model on the same prompts. Add the following cell to the notebook before section 8 ("Test the Trained Model"):

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    dtype=torch.float16,
    device_map="auto",
)
base_model.eval()
base_tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)

def generate_sql_base(question: str, schema: str, max_tokens: int = 64) -> str:
    messages = [
        {
            "role": "user",
            "content": f"Given the following database schema:\n\n{schema}\n\n"
                       f"Write a SQL query to answer this question: {question}",
        }
    ]
    prompt = base_tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    inputs = base_tokenizer(prompt, return_tensors="pt").to(base_model.device)
    outputs = base_model.generate(
        **inputs,
        max_new_tokens=max_tokens,
        temperature=0.1,
        do_sample=True,
        pad_token_id=base_tokenizer.eos_token_id,
    )
    response = base_tokenizer.decode(
        outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    )
    return response.strip()

test_examples = [
    {
        "schema": "CREATE TABLE employees (id INT, name VARCHAR, department VARCHAR, salary DECIMAL, hire_date DATE)",
        "question": "What is the average salary of employees in the engineering department?",
    },
    {
        "schema": "CREATE TABLE orders (order_id INT, customer_id INT, product_name VARCHAR, quantity INT, order_date DATE)",
        "question": "How many orders were placed in the last 30 days?",
    },
    {
        "schema": "CREATE TABLE students (student_id INT, name VARCHAR, grade INT, subject VARCHAR, score DECIMAL)",
        "question": "Find the top 5 students with the highest average score across all subjects.",
    },
]

for i, example in enumerate(test_examples, 1):
    print(f"\nExample {i}:")
    print(f"Question: {example['question']}")
    sql = generate_sql_base(example["question"], example["schema"])
    print(f"Generated SQL: {sql}")
    print("-" * 60)

Run this cell and check the output. Then continue with section 8 of the notebook, which loads the LoRA adapter, merges it with the base model, and runs the same test prompts:

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, CHECKPOINTS_PATH)
model = model.merge_and_unload()
model.eval()

Comparing the results

The fine-tuned model outputs clean, bare SQL across all 3 test prompts:

If you are feeding model outputs directly into a database API, conversational preamble breaks your application. Fine-tuning teaches the model to return raw, executable SQL, saving you from writing fragile parsing logic. The difference is clearest when comparing raw outputs for the same prompt. The base model wraps the correct SQL in explanation and Markdown formatting:

To find the average salary of employees in the engineering department, you can
use the following SQL query:

SELECT AVG(salary) AS average_salary
FROM employees
WHERE department = 'engineering';

This query selects the average (AVG) salary from the employees table where the
department...

The fine-tuned model outputs the query directly:

SELECT AVG(salary) FROM employees WHERE department = "Engineering"

The base model already knows SQL. It produces correct queries. What LoRA fine-tuning taught was the format: output the query directly, without preamble or explanation. This is exactly the pattern in the sql-create-context training data, where each example pairs a question with a bare SQL query.

Clean up

The RayCluster is automatically torn down when the job finishes. The RayJob custom resource is cleaned up after ttl_seconds_after_finished (600 seconds by default). Use job.delete() for immediate cleanup if needed:

job.delete()

The 4 examples at a glance

Because all 4 algorithms share identical cluster and storage setups, switching from LoRA to GRPO or SFT requires changing only your dataset and function calls—not your underlying pipeline. Each of the 4 Ray examples in the Red Hat AI examples repository (sft_ray, osft_ray, lora_ray, and grpo_ray) follows the same core workflow:

All 4 use the same cluster setup and PVC configuration. The only differences are the Training Hub algorithm flag and the evaluation criteria.

Learn more

Clone the Red Hat AI examples repository and run your first fine-tuning job in under 20 minutes.

To learn more about the platform components used in this guide, review the Red Hat OpenShift AI product documentation, the Training Hub repository, and the CodeFlare SDK documentation.