From local prototype to enterprise production: Private speech transcription with Whisper and Red Hat AI

March 6, 2026
Carlos Condado Yuchen Fama
Related topics:
Artificial intelligence
Related products:
Red Hat AI Inference Server

Every time you use a cloud transcription service, your audio travels to someone else's infrastructure. For development workflows, that's friction. For sensitive use cases, it's a non-starter. While vLLM's architecture is primarily designed for CUDA/ROCm GPU acceleration on Linux, recent updates have expanded its reach to the local developer's machine, like the
experimental support for macOS users running Apple Silicon hardware.

This guide shows you how to run OpenAI's Whisper model through vLLM on Apple Silicon, giving you an OpenAI-compatible /v1/audio/transcriptions endpoint on localhost. Then, we discuss how to take this architecture into production using Red Hat AI Inference Server.

Why local automatic speech recognition (ASR) matters

Cloud transcription fails in a significant portion of real-world deployments. Some example use cases and the problems they solve include:

  • Healthcare dictation: HIPAA mandates data residency. Patient audio can't leave controlled environments.
  • Financial services: PCI-DSS and SOX compliance. Client conversations are regulated data.
  • Government / air-gapped: No internet by design. Cloud APIs aren't an option.
  • Legal transcription: Attorney-client privilege. Third-party exposure is an unacceptable risk.
  • Call centers: Customer PII in every call. Compliance teams block external APIs.
  • Enterprise IP: Board meetings, M&A discussions, product roadmaps - too sensitive for cloud.

The audio data explosion compounds this. Between meetings, calls, podcasts, and voice notes, knowledge workers generate hours of speech daily. That's raw intellectual output, and increasingly, organizations want to keep it internal.

The openai/whisper-large-v3 model is among the most downloaded on HuggingFace, with over 6.6 million times, and the newer large-v3-turbo already at 2.7 million downloads. At the time of writing, Qwen3 ASR had just been released with day-zero support in vLLM, and we're actively working on the MLPerf benchmark to share our results with the community.

vLLM supports all the popular audio file extensions, including WAV, OGG, MP3, and FLAC. Any format supported by librosa is supported.

Turn your words into assets, securely: 6 use cases for running Whisper locally

Whisper is an application that offers AI-enabled solutions to record everything said, and then organize it into searchable and actionable content, with end-to-end control. Its transcripts can become fuel for workflows that weren't practical before. Here are 6 examples.

1. AI-enabled communication coach

Feed your daily transcripts to an LLM that analyzes your speech patterns. For example, "You used ‘basically' 17 times today. Here are alternatives…"

You get continuous improvement, taken from your own data, for clarity, logic, and persuasiveness, without exposing your conversations to an external service.

2. Voice-first input

Stop typing the things you repeat all day (prompts, chat messages, emails, docs, and so on). Brain-dump verbally, and structure later. Your keyboard is optional for first drafts.

3. Personal voice journal

At the end of the day, your conversations are already captured for recap, like "What did I commit to in that 3 PM call?" This only works if transcription is local, private, and always available.

4. Automated compliance monitoring

Process call center recordings entirely on-premise to automatically flag regulatory breaches or script non-compliance without customer data ever leaving your network.

5. Real-time customer sentiment

Analyze live sales calls at the edge (for example, in a branch office server) to provide agents with real-time prompts and sentiment scores, reducing latency and bandwidth costs.

6. Secure Meeting Intelligence

Transcribe internal strategy meetings and R&D brainstorming sessions on secure internal servers, creating a searchable knowledge base of company IP that remains strictly confidential.

Quick start: The developer inner loop (10 minutes to a working API in your laptop)

For this article, here's the setup I used:

  • Apple Silicon (M1/M2/M3/M4)
  • macOS 26.2
  • 16 GB RAM for running inference

Step 1: Install uv

The uv application is a fast Python package manager, and vLLM's docs recommend using it. To install uv:

$ curl -LsSf https://astral.sh/uv/install.sh | sh
$ source ~/.zshrc

Step 2: Create environment

Now that uv is installed, create a virtual environment in Python where you can work:

$ mkdir ~/whisper-vllm && cd !$
$ uv venv --python 3.12 --seed vllm-env
$ source vllm-env/bin/activate

Step 3: Build vLLM

No prebuilt Python wheels for vLLM exist for Apple Silicon, so for macOS you must build from source:

$ git clone https://github.com/vllm-project/vllm.git
$ cd vllm

$ uv pip install -r requirements/cpu-build.txt --index-strategy unsafe-best-match
$ uv pip install -r requirements/cpu.txt --index-strategy unsafe-best-match
$ VLLM_TARGET_DEVICE=cpu uv pip install -e . --no-build-isolation

Also install these libraries for audio processing:

$ uv pip install librosa soundfile

Step 4: Start the server

In the whisper-vllm directory, start the local server:

$ vllm serve openai/whisper-large-v3 --dtype float16

Remember that in future sessions (after a reboot, for example), you must activate the Python virtual environment before starting the local server:

$ cd ~/whisper-vllm
$ source vllm-env/bin/activate

When you first run it, approximately 3 GB of data is downloaded. Your active terminal window is occupied by the server for as long as you have it running, so don't close it!

INFO: Resolved architecture: WhisperForConditionalGeneration
INFO: Using max model len 448
INFO: Starting vLLM API server on http://0.0.0.0:8000

Step 5: Transcribe

Create a short audio file in the whisper-vllm directory. Then open a new terminal window or tab:

$ curl -X POST http://localhost:8000/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F "file=@test_audio.flac" \
  -F "model=openai/whisper-large-v3"

In response, you get a transcript of your test audio:

{
  "text": "Mary had a little lamb."
}

Performance

Using CPU inference with no GPU acceleration, typical results are:

  • Audio duration: 11 seconds
  • Transcription time: About 15 seconds
  • Model memory: About 5 GB

For faster local transcription, mlx-whisper runs on bare metal and achieves approximately 2x better throughput. However, you lose the OpenAI-compatible API and the path to production parity. In general, I recommend vLLM Whisper for development and testing with in-production API parity, building apps for GPU deployment, or when OpenAI SDK compatibility is required. Use mlx-whisper for maximum local performance.

From laptop to production with a Red Hat AI Inference Server

You've built your prototype on your local machine. Now you're ready to deploy it at scale. This is where you move from upstream vLLM to the Red Hat AI Inference Server.

AI Inference Server is a hardened, enterprise-ready distribution of vLLM. It allows you to take the exact application code you wrote above and deploy it into a production environment supported by Red Hat. There are many reasons to run your application on AI Inference Server:

  • Validated models: Instead of pulling generic weights, you can use RedHatAI/whisper-large-v3-turbo-FP8-dynamic. This model has been rigorously tested and validated by Red Hat for performance and accuracy on enterprise hardware.
     
  • Production hardening: AI Inference Server includes optimizations for throughput, latency, and monitoring that go beyond the default developer setup, ensuring your transcription service remains stable under load.
  • Seamless transition: Because AI Inference Server exposes the same OpenAI-compatible API as your local vLLM instance, you simply change the base_url in your client code to point to your production server.
  • Enterprise supportability: This is a supported open source solution by Red Hat. You gain the confidence of having a dedicated support structure behind your AI infrastructure, ensuring reliable operations.
  • Deployment flexibility: AI Inference Server can be run on any Linux distribution, allowing you to deploy your AI models on a variety of different hardware and AI accelerators to match your specific infrastructure needs.

By developing locally with vLLM and deploying with Red Hat AI Inference Server, you bridge the gap between rapid prototyping and secure, scalable enterprise AI.

Resources