From token consumer to token provider: Building your org's AI API

The journey from paying per token to running your own AI service on Red Hat AI

September 10, 2026
Markell Rawls
Related topics:
AI inference
Related products:
Red Hat AI

One of the biggest problems with AI right now is that it's expensive. If you've been anywhere near an enterprise IT budget in the last 2 years, then you already know that. Most companies that wanted to get AI into their workflows did the same thing: They signed up for a business account with an AI provider, handed out API keys, and started building. It made sense at the time because the models were good, the APIs were simple, and the alternative was standing up your own inference infrastructure, which nobody had bandwidth for. But then the bills started coming in.

Imagine 3 teams, each with their own API keys, each paying per token, with nobody tracking what anyone was actually spending. The Claude Code licenses alone across a mid-size engineering org can run into serious money, and that's before you factor in, for example, Anthropic API usage from the data science team and the OpenAI bills from the product team. The number only goes up quarter over quarter because more people keep finding ways to use AI in their work, which is exactly what you want to happen, except that the cost scales right along with adoption.

So everyone tried to take control

The logical next step was to bring inference in-house. Fine-tune open models, run them on your own hardware, and stop paying per token to external providers. A lot of teams did exactly that, but they all did it independently and at their own pace.

So now you've got Tom on the data science team who downloaded a model from Hugging Face and is running it on a GPU he requisitioned through a side channel. And you've got an engineer who set up a vLLM instance on a VM that may or may not have gone through the security review process. Marketing heard about what engineering was doing and asked their contractor to set something up too. So somewhere in finance, someone is still using a personal OpenAI key because they couldn't figure out how to get access to any of the internal options.

Describing your Tuesday

If any of that sounds familiar, you're not alone. This is what most mid-to-large organizations look like right now when it comes to AI infrastructure, and the problems compound on each other in ways that make the whole situation worse over time.

There's no single place to see which models are running, who's using them, or what they're costing. IT doesn't have visibility into half the AI workloads because they were never provisioned through official channels. Security can't audit what they don't know about, which means you've got models processing internal data with no governance around what goes in or what comes out. And when a model vendor deprecates a version (which happens every few months, sometimes without warning), you find out because applications break and teams file tickets at the same time.

The Stanford Digital Economy Lab's enterprise AI playbook of 51 case studies found that 70 to 80% of employees using AI at work rely on tools not approved by their employer. That's not a hypothetical risk, it's what shadow AI actually looks like in the real world: personal AWS accounts, unauthorized API credits, and models downloaded onto department machines, all without governance or cost control.

None of this happened because anyone made bad decisions. It happened because the tooling to do this well at the organizational level just wasn't there when everyone started building.

What if a single team could manage all of it?

Now imagine if there was a way for 1 team in your company to control a set of approved models. These models would be security tested, benchmarked against your hardware, and made available through a standard API that works with the tools developers already use. Instead of every team standing up their own inference stack, your platform team operates a shared service and developers get self-service access to AI the same way they get access to a database or a message queue.

That's what the Model-as-a-Service pattern is. As of Red Hat AI 3.4, it's a generally available set of capabilities that turns your platform team from a token consumer (buying tokens from external providers) into a token provider (serving models internally with full visibility and governance). Let me walk through what that actually looks like.

The admin experience: Deploying a model

Everything starts in the Red Hat OpenShift AI dashboard. From the Projects and Deployments tab, you can see what's already being served. For example, Gemma 4 and Llama 4 Scout, both running with distributed inference on NVIDIA L40s hardware profiles through llm-d.

Configuring the model availability setting exposes your model through the MaaS gateway instead of leaving it isolated.

When you need to deploy a new model, the model catalog, accessed through the AI hub within OpenShift AI is where you start. The platform provides quick access to a curated selection of validated models, including NVIDIA Nemotron variants, DeepSeek, and Granite. Each one comes with validated performance insights and benchmarks so you can see how it performs on your specific hardware before you commit GPU resources to it. You don't need to spend weeks evaluating, red-teaming and packaging models yourself because the catalog has already done that work.

The developer experience: Self-service access

Here's where things change for developers. Instead of filing a ticket and waiting days (or weeks) for someone on the platform team to provision an endpoint, a developer logs into the OpenShift AI dashboard, opens gen AI studio, navigates to AI asset endpoints, and sees the models their team has access to. They click into an endpoint and get an API route and an API key that's generated on the spot, scoped to their team's policy, and instantly revocable if needed.

The engineer takes that endpoint URL and API key and drops them into whatever tool they're already using, whether that's Claude Code, OpenCode, or something like Continue (an open source VS Code extension for AI coding assistance). The entire integration is 6 lines of code to repoint your API client's base_url and api_key in the ~/.continue/config.yaml file:

models:
  - name: Meta Llama 4 Scout
    provider: openai
    model: "llama-4-scout"
    apiBase: "http://maas.apps.ocp.example.com/llm/llama-4-scout/v1"
    apiKey: "YOUR_API_KEY"

Because the platform exposes an OpenAI compatible API, developers can use standard OpenAI SDKs and existing client libraries without rewriting their code. When a request exceeds the admin-defined rate limit, the application receives a standard HTTP 429 (too many requests) response. Standard client libraries and AI agents handle this back-off mechanism natively, requiring no custom error-handling or quota-tracking code.

What the platform team sees

Every request a developer makes flows through the AI gateway and lands in observability. The Observe and Monitor dashboard in OpenShift AI gives the platform team a unified picture of token use across the entire organization, broken down by team, by model, and by user.

For deeper performance tracking like P90 latency (the slowest 10% of requests), error rates, GPU utilization, and request queue length, the Models tab surfaces the metrics that SREs actually care about. And because it's all built on Prometheus and Grafana under the hood, your existing dashboards work too. You can pull token use by user and team directly from the MaaS metrics into the same Grafana instance where you're already watching cluster health.

When you want to correlate model behavior with what's happening on the cluster, like a spike in latency that lines up with a specific GPU node, you can navigate directly to the OpenShift console to inspect the underlying pods.

But is it actually worth the effort?

This is usually where people push back, and the objections are reasonable, so let's walk through them.

Is it hard to set up?

You just saw the developer integration: 6 lines of standard Python or JavaScript code. On the admin side, deploying a model from the catalog is a few clicks because the models come pre-validated with benchmarks already attached. The API gateway handles auth, rate limiting, and usage tracking out of the box. If you've ever set up a database-as-a-service or a shared message queue for your org, this follows the same operational pattern.

Does it take a long time?

The models in the catalog are already validated and benchmarked, so you're not starting from scratch. You can have a model serving requests within minutes of deploying it, and the self-service API key flow means developers can start building against it the same day.

We don't have months to fine-tune and red-team our own models

You don't have to. The catalog ships with models that are ready to serve right now. Fine-tuning is an option if you want to improve performance for specific use cases down the road, but it's not a prerequisite. You can start with a validated model out of the catalog and get immediate value while you figure out your fine-tuning strategy on your own timeline.

Won't this feel like we're controlling people?

This one comes up a lot, and it's worth spending a minute on because the reality is actually the opposite of what most people expect.

Here's a real scenario: your sales team keeps hitting rate limits on the smaller model they were assigned. Requests are queuing, responses are slow, and reps are waiting around instead of working. Without the usage dashboard, all you'd hear is "the AI is slow" and you'd have no idea whether it's a model problem, a capacity problem, or something else entirely.

But with showback, you can see exactly what's happening. The sales team is doing way more summarization work than anyone expected when their allocation was set, and they're bumping up against their token ceiling every day by mid-afternoon. So you bump their quota or give them access to a faster model that handles their workload better, and the bottleneck disappears.

You never would have spotted that without the data. The usage dashboard gave your platform team the visibility to catch a problem that the sales team just thought was "how AI works here," and the fix took five minutes. That's what it looks like when a platform team is actually set up to support the people using their services, and it only works because you have the usage data to see what's going on.

You never would have found that problem without showback. The data gave your platform team the visibility to identify an issue that HR didn't even know they had, and the fix was straightforward. That's what it looks like when a platform team is actually set up to support the people using their services, and it only works because you have the usage data to see what's going on.

The architecture behind it

Here's what's running when all of this is working together.

Infrastructure and orchestration

Red Hat OpenShift, which operationalizes Kubernetes, provides a single foundation across on-premises, cloud, and edge environments. The same control plane runs your AI workloads everywhere, which matters when sovereign AI requirements mean a model has to run inside a specific country.

A platform designed for AI workloads

Red Hat OpenShift AI provides standardized model-serving runtimes, GPU-aware scheduling, and model lifecycle management, creating an environment built for machine learning engineers and data scientists rather than cluster administrators.

The engine for model serving: vLLM and llm-d

vLLM is an open source inference engine that serves AI models through APIs, with optimizations for large language models that make serving AI at scale cost-effective. The llm-d runtime extends this with distributed inference for even higher throughput.

An AI gateway for standardization

The platform's integrated, Kubernetes-native gateway stack, built on open standards, provides built-in authentication, authorization, rate limiting, and usage tracking. This allows you to expose a single, OpenAI-compatible /v1/chat/completions endpoint that handles request routing to either locally served vLLM-hosted model or from external cloud-provider endpoints like AWS Bedrock, Azure, or Anthropic.

GPU cost attribution and governance

The architecture integrates with open source observability tools like Prometheus for metrics, Grafana for dashboards, and Jaeger for distributed tracing. View per-team token consumption, request rates, latency, and error rates alongside the GPU and infrastructure metrics your SREs are already watching.

Centralized inference saves money. Instead of N teams each running their own shadow deployments with dedicated GPUs sitting idle, one MaaS stack pools GPU resources across the organization for higher utilization and lower cost, and your platform team gets full visibility into how those resources are actually being used.

Try it on your own cluster

If any of this sounds like the problems you're dealing with right now, the fastest way to see how MaaS works is to set it up yourself. Red Hat OpenShift AI 3.4 made MaaS generally available, and the documentation walks through the full installation and configuration. You can also check out the companion interactive overview to see the end-to-end flow visually before you start building.

Go deeper

This blog covers the end-to-end MaaS story. If you want to dive deeper into specific capabilities, the following resources go further on each one: