Speaker diarization answers a question transcription cannot: not just what was said, but who said it. pyannote.audio is the open-source library most people reach for, and it is the engine behind the diarization in tools like WhisperX.
This guide covers how pyannote works, the access step that catches almost everyone on first run, and how to deploy it on Runpod.
What is pyannote?
pyannote.audio is an open-source toolkit for speaker diarization, built on PyTorch. Its job is to segment an audio recording by speaker: identifying when someone starts talking, when they stop, and which segments belong to the same person.
The pipeline runs several stages:
- Voice activity detection to find where speech occurs at all
- Speaker segmentation to detect turn boundaries, including overlapping speech
- Speaker embedding to produce a voice fingerprint for each segment
- Clustering to group segments that belong to the same speaker
The output is a timeline of speaker turns. It does not know anyone's name, only that speaker A is distinct from speaker B. Pairing that with a transcript from WhisperX is what turns it into something useful.
The gated model step
This trips up nearly everyone the first time. The pyannote pipelines on Hugging Face are gated, and you must do two things before any code will run:
- Visit the model pages on Hugging Face and accept the user conditions for both the diarization pipeline and the segmentation model it depends on.
- Generate a Hugging Face access token and pass it when you load the pipeline.
If you skip either step you get an authentication error that looks like a bug in your code. It is not. Accept the terms first, then deploy.
Why run pyannote on Runpod?
Diarization is genuinely GPU-bound. The embedding and segmentation models run far faster on a GPU, and on long recordings the difference is between minutes and a coffee break.
Workloads come in batches. Most diarization happens against a backlog of recordings, not a steady stream. Per-second billing suits that shape.
It pairs naturally with transcription. Running pyannote and WhisperX on the same pod means one environment, one GPU, and no moving audio between machines.
GPU requirements for pyannote
pyannote is light compared to most models in this space. The models are small; throughput is what you are paying for.
- Standard diarization: RTX A5000 (24 GB) from $0.16/hr is more than sufficient
- Diarization plus WhisperX transcription: RTX 4090 (24 GB) from $0.34/hr
- High-volume batch processing: L40S (48 GB) from $0.79/hr for parallel jobs
There is little reason to reach for an H100 here. Diarization does not benefit from the extra memory, and the money is better spent on the transcription side of the pipeline if you are running both.
Step 1: Accept the model terms
Before touching Runpod, go to Hugging Face, open the pyannote speaker diarization pipeline page, accept the conditions, do the same for the segmentation model, and generate an access token. Keep the token handy.
Step 2: Create an account and deploy
Go to runpod.io and sign up with email, GitHub, or Google. Add credits or a payment method.
- Open the Pods section and select an RTX A5000 or RTX 4090.
- Choose a PyTorch template.
- Set container disk to at least 20 GB.
- Add your Hugging Face token as an environment variable.
- Deploy.
Step 3: Run diarization
Connect through JupyterLab or the web terminal, install pyannote.audio, and load the pipeline with your token. A basic run looks like this:
from pyannote.audio import Pipeline
import torch
pipeline = Pipeline.from_pretrained(
"pyannote/speaker-diarization-3.1",
use_auth_token="YOUR_HF_TOKEN")
pipeline.to(torch.device("cuda"))
diarization = pipeline("meeting.wav", min_speakers=2, max_speakers=5)
for turn, _, speaker in diarization.itertracks(yield_label=True):
print(f"{turn.start:.1f}s - {turn.end:.1f}s: {speaker}")The pipeline.to(torch.device("cuda")) line matters. Without it pyannote runs on CPU and you have paid for a GPU that is doing nothing.
Pro tips
Always set speaker bounds when you know them. min_speakers and max_speakers are by far the highest-leverage accuracy setting. Clustering with a known speaker count is a much easier problem than estimating it.
Check your audio format first. Mono, 16 kHz WAV is the safest input. Odd sample rates and multi-channel files produce confusing results that look like model failures.
Expect overlapping speech to be hard. pyannote handles it better than most, but heavy crosstalk remains the hardest case in diarization. If accuracy matters, sample your worst audio before committing to a pipeline.
Run it alongside WhisperX. Diarization alone gives you speaker turns with no words. The combination is what people actually want.
Stop your pod when you are done. Runpod bills by the second while a pod runs.
Wrapping up
pyannote is the standard answer for speaker diarization, and once past the gated-model step it is straightforward to run. On a 24 GB card from $0.16/hr it is one of the least demanding GPU workloads you will deploy, and combined with WhisperX it produces attributed transcripts from raw audio in a single pass.
Ready to try it? Deploy a pod and start with a short two-speaker recording to validate the setup.
FAQ
What is speaker diarization?
The task of determining who spoke when in an audio recording. It segments audio by speaker without identifying anyone by name, producing labels like speaker A and speaker B along with the time ranges for each turn.
Why does pyannote fail to load with an authentication error?
The models are gated on Hugging Face. You must accept the user conditions for both the diarization pipeline and the segmentation model it depends on, then pass a valid access token. This is the most common setup problem.
What GPU do I need for pyannote?
Very little by current standards. An RTX A5000 from $0.16/hr handles standard diarization. If you are also running WhisperX transcription on the same pod, an RTX 4090 from $0.34/hr covers both.
How accurate is pyannote?
Good on clean audio with distinct speakers, and it degrades on heavy overlapping speech, poor recording quality, or very similar voices. Supplying min_speakers and max_speakers when known improves results substantially.
Can I use pyannote with WhisperX?
Yes, and that is the common setup. WhisperX integrates pyannote directly for its diarization step, so running both on one pod gives you transcripts with word-level timestamps and speaker labels in a single pipeline.
