How to run WhisperX on Runpod

Whisper transcribes accurately but gives you loose segment timings and no idea who is speaking. WhisperX fixes both, and makes the whole thing considerably faster while it is at it.

This guide covers what WhisperX adds on top of Whisper, which GPU to run it on, and how to set up a transcription pipeline on Runpod that returns word-level timestamps and speaker labels.

What is WhisperX?

WhisperX is an open-source wrapper around OpenAI's Whisper that addresses three specific weaknesses:

• Word-level timestamps. Whisper produces timings at the segment level, often drifting by a second or more. WhisperX runs forced alignment with a phoneme model to pin each word to an accurate position in the audio.
• Speaker diarization. By integrating pyannote, WhisperX labels which speaker said what, turning a wall of text into an attributed transcript.
• Batched inference. WhisperX uses voice activity detection to cut audio into chunks it can process in parallel, which is where the large speed gains come from compared to running Whisper sequentially.

The practical result: a long recording that would take a while through vanilla Whisper comes back faster, with timings you can actually build a subtitle file from, and speaker turns already separated.

Why run WhisperX on Runpod?

Transcription is bursty. Most teams process a batch of recordings and then nothing for hours. Paying for an idle GPU between batches makes no sense, and Serverless endpoints scale from zero.

The whole pipeline is GPU-bound. Whisper, the alignment model, and diarization all want a GPU. Running the stack locally on CPU works, but slowly enough that it stops being useful at volume.

Model size is a real choice. large-v3 is meaningfully more accurate than medium on difficult audio. Being able to move between them without re-provisioning hardware matters when you are tuning quality against cost.

GPU requirements for WhisperX

WhisperX is not demanding by 2026 standards. The main variable is which Whisper model you load and whether diarization is running alongside it.

• medium, no diarization: RTX A5000 (24 GB) from $0.16/hr, or L4 from $0.44/hr
• large-v3, transcription only: around 10 GB. RTX 4090 (24 GB) from $0.34/hr
• large-v3 with diarization and alignment: around 16 to 20 GB. RTX 4090 from $0.34/hr comfortably
• High-throughput batch processing: L40S (48 GB) from $0.79/hr for larger batch sizes

An RTX 4090 is the practical default. There is little benefit in going higher unless you are batching aggressively, in which case the extra VRAM on an L40S buys you larger batches rather than faster individual transcriptions.

Step 1: Create a Runpod account

Go to runpod.io and sign up with email, GitHub, or Google. Add credits or a payment method so you can deploy a GPU.

Step 2: Deploy a pod

1. Open the Pods section and select an RTX 4090.
2. Choose a PyTorch template with CUDA support.
3. Set container disk to at least 30 GB for the Whisper and alignment model weights.
4. Attach a network volume if you will run this regularly, so models persist between sessions.
5. Deploy.

Step 3: Install and run

Connect to the pod through the web terminal or JupyterLab and install WhisperX from its repository. On first run it downloads the Whisper checkpoint and the alignment model for your target language.

A first transcription looks roughly like this:

whisperx audio.mp3 --model large-v3 --batch_size 16 --compute_type float16 --output_format srt

To add speaker labels, enable diarization and supply a Hugging Face token, since the pyannote models are gated and require accepting their terms first:

whisperx audio.mp3 --model large-v3 --diarize --hf_token YOUR_HF_TOKEN --min_speakers 2 --max_speakers 4

Supplying speaker count bounds when you know them materially improves diarization accuracy. If you are transcribing a two-person interview, say so.

Pro tips

Set compute type to float16. On any modern GPU this roughly halves memory use with no meaningful accuracy cost. int8 goes further if you are memory constrained.

Tune batch size to your GPU. Batch size 16 works well on a 24 GB card. Raising it increases throughput until you hit memory limits, at which point you will see an out-of-memory error rather than graceful degradation.

Give diarization speaker bounds. min_speakers and max_speakers are the highest-leverage accuracy setting in the whole pipeline when you know the answer in advance.

Specify the language. Auto-detection costs time and occasionally gets it wrong on short or noisy clips. If you know the language, pass it.

Attach a network volume. Pod storage is ephemeral, and re-downloading large-v3 plus alignment models on every deploy wastes several minutes each time.

Stop your pod when you are done. Runpod bills by the second while a pod runs.

Wrapping up

WhisperX is one of those tools where the improvement over the baseline is obvious within one test file. If you are producing subtitles, meeting transcripts, or anything where knowing who spoke matters, it is the sensible default. Pair it with pyannote for diarization and you have a full pipeline on a single 24 GB card.

Ready to try it? Deploy an RTX 4090 pod and run your first transcription in a few minutes.

FAQ

What is the difference between Whisper and WhisperX?

Whisper transcribes with segment-level timestamps and no speaker information. WhisperX adds forced alignment for word-level timestamps, integrates pyannote for speaker diarization, and uses voice activity detection to batch audio for faster processing.

What GPU do I need for WhisperX?

An RTX 4090 from $0.34/hr handles large-v3 with diarization and alignment comfortably. Smaller models run on a 24 GB A5000 from $0.16/hr. For high-volume batch work, an L40S at $0.79/hr allows larger batch sizes.

Do I need a Hugging Face token?

Only for diarization. The pyannote models WhisperX uses are gated, so you need to accept their terms on Hugging Face and supply a token. Transcription and alignment work without one.

How accurate are the word-level timestamps?

Substantially better than Whisper's native segment timings, which is the point of the forced alignment step. Accuracy varies with audio quality and language, since alignment relies on a phoneme model for the target language.

Can I run WhisperX as an API?

Yes. Containerize your WhisperX setup and deploy it to Runpod Serverless. Your application posts audio and receives a transcript, and the endpoint scales from zero between batches so idle time costs nothing.