Bringing custom knowledge to agents with AutoRAG

The large language models (LLMs) that power generative AI work by drawing upon the patterns and information present in their training data. Without access to the right data, LLMs struggle to comprehend context—like our internal corporate vocabulary—and as a result, start to hallucinate. This can become a major issue for domain-specific use cases at an enterprise level.

To get reliable results, it's vital to ground these models in our own company documentation. This is a process known as retrieval-augmented generation (RAG). Typically, building a functional RAG pipeline requires a lot of manual work, including chunking documents, selecting embedding models, and configuring retrieval strategies.

AutoRAG can help speed up this process by automating the optimization of RAG pipelines. Introduced as a Technology Preview in Red Hat OpenShift AI 3.5, AutoRAG helps improve the accuracy of LLM responses by grounding them in an organization's domain-specific data. Instead of relying on trial and error, AutoRAG systematically processes documents, tests various pipeline configurations against an evaluation dataset, and identifies the optimal RAG setup that performs best for our specific business context.

This guide shows how to set up and integrate AutoRAG for a fictional bank use case. We'll also extend the system's knowledge using a Model Context Protocol (MCP) server to fetch customer data from a database. The objective is to create an agent that understands the company context using the best RAG configuration and MCP servers. Apart from AutoRAG, you will use the MCP catalog of Red Hat OpenShift AI to deploy the MCP server to interact with internal databases.

Prerequisites

Make sure you satisfy the requirements before beginning:

• Red Hat OpenShift cluster (4.20 or later).
• Red Hat OpenShift AI 3.5 operator installed.
• This demo requires a GPU to deploy models in Red Hat OpenShift AI. If you don't have one, you can bring your preferred model provider endpoint.

During the demo, you will deploy YAML files directly into OpenShift. Clone the repository to execute the commands from your terminal.

git clone https://github.com/dialvare/autorag-demo.git
cd autorag-demo

1. Set up Red Hat OpenShift AI

An efficient AI development process requires a platform that lets teams build, test, and deploy intelligent workflows collaboratively. Our platform of choice is Red Hat OpenShift AI, a flexible and comprehensive suite of tools that simplifies the lifecycle of AI/ML models.

First, we enable the AutoRAG and MCP catalog interfaces. Next, we instruct Red Hat OpenShift AI to handle the deployment of Open GenAI Stack (OGX) (previously known as Llama Stack).

To get this process started:

• Navigate to Home > Search in your Red Hat OpenShift web console.
• Look for the DataScienceCluster object that defines your Red Hat OpenShift AI deployment.
• Edit the manifest to set the ogx spec to Managed. Once the configuration is saved, Red Hat OpenShift AI will take care of automatically activating the OGX operator needed for AutoRAG.

apiVersion: datasciencecluster.opendatahub.io/v2
kind: DataScienceCluster
metadata:
 name: default-dsc
 labels:
   app.kubernetes.io/name: datasciencecluster
spec:
 components:
   /* rest of the components */
   trainer:
     managementState: Removed
   ogx:
     managementState: Managed
   /* rest of the YAML */

Now, let's enable the MCP catalog and AutoRAG interfaces.

1. Go to Home > Search and look for the OdhDashboardConfig resource that defines your Red Hat OpenShift AI dashboard in the redhat-ods-applications namespace.

2. Set the genAiStudio, autorag, and mcpCatalog specs to true to enable the Gen AI Studio, MCP servers, and AutoRAG interfaces.

   apiVersion: opendatahub.io/v1alpha
   kind: OdhDashboardConfig
   metadata:
    name: odh-dashboard-config
    namespace: redhat-ods-applications
    labels:
      app: rhods-dashboard
      app.kubernetes.io/part-of: rhods-dashboard
      app.opendatahub.io/rhods-dashboard: 'true'
      platform.opendatahub.io/part-of: dashboard
   spec:
    dashboardConfig:
      genAiStudio: true
      autorag: true
      mcpCatalog: true
      disableTracking: false
      /* rest of the YAML */

Following OpenShift best practices, we'll group all OGX-related resources inside a single dedicated ogx namespace.

Create a project named ogx. Run the following command:

oc new-project ogx && oc label namespace ogx opendatahub.io/dashboard=true

2. Set up the storage stack

To establish a reliable data ingestion pipeline for RAG, we'll deploy 3 storage components to separate structured data, vector embeddings, and artifacts:

• PostgreSQL: The database OGX uses for managing its operational configuration.
• Milvus: A high-performance vector database that stores and indexes RAG embeddings to support efficient similarity searches.
• MinIO: An Amazon S3-compatible object storage solution that will serve as a central repository for our documents and artifacts.

To simplify the deployment of these components, apply the following manifests. If you want a detailed look at the YAML configuration, check out this article's GitHub repository.

1. Deploy the PostgreSQL database to store OGX's metadata:

   oc apply -f 2-storage/postgresql-setup.yaml -n ogx

2. Deploy Milvus as a remote vector database provider for OGX:

   oc apply -f 2-storage/milvus-setup.yaml -n ogx

3. Deploy the MinIO object storage in the cluster. The username is minioadmin and its password is minioadmin123:

   oc apply -f 2-storage/minio-setup.yaml

4. Apply this Job to automatically create the pipeline-artifacts and autorag-data MinIO buckets. It also uploads the internal knowledge data we want the agent to understand.

   oc apply -f 2-storage/minio-job.yaml

Let's verify that the MinIO buckets exist and that the custom data was uploaded successfully.

1. Change the web console project to minio.
2. Go to Networking > Routes and select the minio-console link (Figure 1).

1. Log in with the minioadmin user and minioadmin123 password.
2. Navigate to the autorag-data bucket.
3. Verify the structure is the one shown in Figure 2.

3. Deploy AI models

Our bank agent relies on data that includes sensitive information like clients' personal data and account balances. Depending on compliance requirements, sending this data to a cloud provider may not be an option.

Fortunately, Red Hat OpenShift AI provides a curated model catalog, allowing us to pick and choose which models best fit our needs. If you're still having trouble deciding on a model, AutoRAG can evaluate model candidates and suggest the best option.

Let's test this out by deploying a couple AI models for AutoRAG to evaluate.

In your Red Hat OpenShift AI console, navigate to AI Hub > Models > Catalog and search for the model you want to evaluate. In this example, our LLM will be Qwen3-8B, as shown in Figure 3. To keep a low response time, we'll use a GPU to run it.

Deploy the model using the following configuration parameters (see Figure 4):

• Project: ogx
• Hardware profile: gpu-profile
• Deployment resource*:* vLLM NVIDIA GPU ServingRuntime for KServe
• Enable Add custom runtime arguments and paste:
  • --enable-auto-tool-choice
  • --tool-call-parser
  • hermes

Since this is the first time you are deploying the model, it will download the serving runtime (vLLM) and model image. This could take a while. You can continue once the model is Ready (Figure 5).

Now let's deploy the embedding model we'll use with AutoRAG. In this case, it will be an embeddinggemma-300m model from the model catalog interface using the following configuration parameters, as shown in Figure 6:

• Project: ogx
• Hardware profile: default-profile
• Deployment resource*:* vLLM CPU (x86) ServingRuntime for KServe

For this demo, we'll deploy only 1 embedding model for simplicity. But you can deploy multiple embedding models for a more diverse performance comparison with AutoRAG.

4. Deploy the OGX server

With our underlying models successfully deployed and accessible, the final step in establishing our infrastructure is to configure the OGX server. AutoRAG will use this OGX instance to evaluate the different RAG configurations.

First, create a Secret containing the configuration for the LLM (qwen3) and the embedding model (embeddinggemma) that OGX will use.

oc create secret generic llm-secret -n ogx \
  --from-literal=INFERENCE_MODEL='redhataiqwen3-8b-fp8-dynamic' \
  --from-literal=VLLM_URL='http://redhataiqwen3-8b-fp8-dynamic-predictor.ogx.svc.cluster.local:8080/v1' \
  --from-literal=VLLM_TLS_VERIFY='false' \
  --from-literal=EMBEDDING_MODEL='redhataiembeddinggemma-300m' \
  --from-literal=EMBEDDING_PROVIDER_MODEL_ID='redhataiembeddinggemma-300m' \
  --from-literal=VLLM_EMBEDDING_URL='http://redhataiembeddinggemma-300m-predictor.ogx.svc.cluster.local:8080/v1' \
  --from-literal=VLLM_EMBEDDING_TLS_VERIFY='false'

Next, create the OGXServer resource containing the OGX configuration. This YAML will load the variables from the secret and enable the built-in embedding model OGX provides (nomic-embedding-text).

oc apply -f 4-ogx/deployment.yaml -n ogx

5. Set up and evaluate with AutoRAG

AutoRAG requires creating a pipeline server to run the evaluation pipeline.

Create the PipelineServer that connects OpenShift AI to MinIO and allows the storage of the pipeline outputs into the pipeline-artifacts bucket:

oc apply -f 5-autorag/pipeline-server.yaml -n ogx

Finally, it's time to start creating the AutoRAG optimization run. We'll need to connect to the OGX instance then create an S3 data connection to load our documents from MinIO.

Create the data connection pointing to our autorag-data bucket and name it KnowledgeConnection so we can select it later:

oc apply -f 5-autorag/knowledge-connection.yaml -n ogx

Now, in your OpenShift AI console, navigate to Gen AI studio > AutoRAG. Select the ogx project and select Create run. Complete the following fields:

• Name: AutoRAG
• Open GenAI Stack connection > Add new connection
  • Connection name: OGXConnection
  • Base URL: http://ogx-custom-server-service.ogx.svc.cluster.local:8321
  • API key: fake

On the final form, set up AutoRAG with the following configuration, as shown in Figure 8:

• S3 connection: KnowledgeConnection
• Select file or folder*:* input_data
• Vector I/O provider: milvus-remote (remote Milvus)
• Evaluation dataset: benchmark_data.json

Select Create run to trigger the pipeline. OpenShift AI will automatically deploy a pipeline to test the foundation model and both embedding models against our RAG documentation. Wait for the pipeline to finish (Figure 9).

Select the top result generated by AutoRAG to see what details it suggests you use for your application, such as the vector store and number of chunks (Figure 10).

Select Retrieval to check the retrieval strategy of the RAG execution (Figure 11).

6. Deploy MCP servers from the MCP catalog

While LLMs provide the reasoning capabilities for our agent, they lack inherent knowledge of our specific operational environment. In order for our agent to interact with the OpenShift cluster (such as listing pods, viewing logs, or diagnosing workload issues), we need to use MCP servers.

At the beginning, we enabled the MCP catalog dashboard in Red Hat OpenShift AI. To start using it, a few custom resource definitions are needed.

First, install the MCP lifecycle operator. It provides a declarative API to deploy, manage, and safely roll out MCP servers on OpenShift:

oc apply -f 6-mcp/mcp-lifecycle-operator.yaml

We're going to connect our agent to a MariaDB MCP that points to a database containing sensitive information from the bank's customers. Let's deploy the MariaDB and fill it with client information.

oc apply -f 6-mcp/mariadb.yaml -n ogx

Now you can go to your Red Hat OpenShift AI dashboard and select AI Hub > MCP Servers > mariadb/mcp. Then, provide the following configuration:

• Deployment name: mariadbmcp
• Project: ogx

• YAML configuration:

  config:
   # ... (keep defaults)
    env:
      - name: DB_HOST
        value: mariadb.ogx.svc.cluster.local
      - name: DB_NAME
        value: mcp_db
      - name: ALLOWED_HOSTS
        value: "*"
      - name: DB_USER
        valueFrom:
          secretKeyRef:
            name: mariadb-credentials
            key: db-user
      - name: DB_PASSWORD
        valueFrom:
          secretKeyRef:
            name: mariadb-credentials
            key: db-password

Additionally, we need to patch its initial configuration because the logging handler needs a writable mount for its log file. Run this command:

oc patch mcpserver mariadbmcp -n ogx --type=merge \
    --patch-file=6-mcp/mariadb-mcp-logs-patch.yaml

Verify the MariaDB MCP server is now Available in your OpenShift AI console, as shown in Figure 12.

7. Try out the final agent!

We've tried out AutoRAG and enabled an MCP server through the catalog. Let's test the behavior of the agent with an interactive application based on Streamlit.

We will use the same OGX deployment we used for AutoRAG. Update the OGXServer configuration to use the newly deployed MCP server:

oc patch ogxserver ogx-custom-server \
    -n ogx --type=merge --patch-file=7-app/ogx-mariadb-connection.yaml

Deploy the application:

oc apply -f 7-app/deployment-mariadb.yaml -n ogx

Go to Workloads > Topology. Make sure you are in the ogx project.

Select the Deployment circle, and on the Resources tab, open the app route. The application will show up.

On the left, there's a panel where we can modify the configuration of the agent. On the right, you'll see a chat interface to interact with the agent.

Let's do the final check:

• Modify the parameters to match the best RAG results. You can find these by selecting the name of the best result in the AutoRAG ranking. In our example:
  • Vector store: vs_342e657d-e706-45f2-a17b-011a0cc53822
  • Number of chunks: 3
  • Search mode: vector
  • Ranker K: 0
  • Ranker Alpha: 1
• Make sure the MariaDB MCP server option is activated.
• Select Apply changes.

Ask the following questions (Figure 13):

Show me the information for the client with email elena.martinez@email.com

What's the limit of its Gold card?

Hurray! The agent correctly understands our company data and is ready to connect with additional MCP servers that expand its capabilities.

Ready to optimize your own RAG pipelines? Explore the OpenShift AI AutoRAG documentation or experiment with custom MCP servers in your cluster. Try Red Hat OpenShift AI to explore the MCP catalog and deploy your first MCP server on OpenShift.