The Binlog is the Source of Truth: How We Built Event Streaming Infrastructure at Runpod

Your application database knows what is. It doesn't know what was. When a row updates in place, the previous state is gone. There is no append log, no history table, no record that the transition happened at all — only the data in the current snapshot. That's fine until you need to react to changes in real time, audit what happened to a billing record, or answer the question: "when did that pod move from RUNNING to STOPPED, and how long did it take?"

At Runpod, we ran into this wall across multiple systems simultaneously. Billing polled the database to detect state changes. Notifications polled the API. Monitoring polled both. The data team ran its own extraction on top of raw change records. Every one of those loops introduced lag, and every one of them re-derived the same meaning from the same rows. 

Every consumer that polls the database has to reconstruct meaning from raw state. The billing system sees a row where status changed from RUNNING to STOPPED and has to infer "this pod stopped, and here is when." The data team sees the same row change and has to derive the same conclusion independently, with its own logic, in its own codebase. Everyone downstream reverse-engineers the same domain events from the same raw rows, and everyone does it slightly differently.

This post is about how we replaced that polling with a change data capture (CDC) pipeline built on the transactional outbox pattern, and why the biggest win was not simply lower latency. It was that the systems making the changes now say what happened, in domain terms, once, at the source. Instead of every consumer inferring "a pod stopped" from a row diff, the producer emits a pod.stopped event directly. Downstream teams consume meaning, not raw state. Today that pipeline powers our analytics and gives teams across Runpod faster, more accurate insight into what is happening on the platform. Operational consumers like billing and notifications still run on the older polling paths, and moving them onto events is where this is headed.

Architecture Overview

Two Kinesis streams power everything downstream. Both originate from the same source: Vitess VStream watching our outbox table via the MySQL binlog. The two halves of this post, the operational event infrastructure and the analytics pipeline, are built on the same CDC source, but they're independently adoptable. If you're here for the billing trust problem, Part 1 stands alone. If you're here for the Snowflake pipeline, Part 2 stands alone.

RunpodEventsStream carries more than two million domain events per day. This is the stream operational consumers like webhooks, billing, and notifications will read from as they move off polling,  via Restate, and it is also the stream our Snowflake analytics pipeline reads today. 

CDCEventsStream carries roughly 14 million raw CDC records per day. This is the complete change feed, every row mutation, feeding a separate pipeline that preserves full row-level history across arbitrary tables. It is not the analytics source; that role belongs to RunpodEventsStream.

The full flow:

Figure 1: Operational and analytic event streaming architecture

Both Kinesis streams matter operationally, but this post's analytics half follows RunpodEventsStream end to end: the same validated, schema-enforced events Part 1's consumers already trust, not a separate raw feed. The reason why is the subject of Part 2's closing section. 

Our technology choices were more pragmatic than principled. We were already on AWS, so Kinesis was the path of least resistance for streams, and its managed Snowflake connector turned out to be genuinely easy to stand up. That access and simplicity drove most of the stack. The one choice we did deliberate on was CDC tooling. Debezium is the common answer here, but running it well means running and tuning Kafka Connect, which is real operational overhead we didn't want to own. Vitess VStream reads the same MySQL binlog with far less to operate, so we built the CDC service on VStream directly. The rest of the decisions we'll only call out where they actually shaped design. 

Part 1: Event Infrastructure

Written by Zack McKenna

The Transactional Outbox: Emitting Domain Events at the Source

The dual-write problem is not solvable at the application layer. You cannot write to two systems atomically from application code without a distributed transaction, and distributed transactions introduce their own failure modes. The transactional outbox pattern takes a different approach: it moves the event write inside the same database transaction as the business write. The database enforces atomicity. If the business write commits, the event commits. If the business write rolls back, the event rolls back. There is no window where they can diverge.

In practice, every business write that needs to produce an event does two things in a single Prisma transaction:

await prisma.$transaction([
  prisma.pod.update({ where: { id }, data: { status: "STOPPED" } }),
  prisma.outboxEvent.create({
    data: {
      eventType: "pod.stopped",
      aggregateId: id,
      payload: { /* domain event body */ },
    },
  }),
])

Either both rows commit or neither does. The OutboxEvent table accumulates a durable, ordered record of domain events exactly as they happened, not as application logic attempts to record them.

Vitess VStream watches the binlog on the outbox table specifically, not on the application tables directly. That distinction matters. If you point VStream at every table in your database, you get raw row diffs for everything: schema migrations, internal bookkeeping updates, system writes. You cannot distinguish a semantically meaningful change from a background process touching a column. The outbox table is the filter. It only contains events that application code intentionally wrote, with the shape and semantics we defined.

Why not watch application tables directly? Domain events need a semantic layer. A raw binlog event tells you which columns changed and to what values. It does not tell you why they changed or what that means to downstream consumers. The outbox provides the interpretation at write time, where the author of the change has the most context.

The key guarantee is simple: database state and the event stream cannot diverge. Not "unlikely to diverge." Not "monitored for divergence." They cannot diverge because the database transaction is the guarantee.

Two Streams, One Analytics Source

A single CDC source feeding two streams with different semantics is not an obvious design choice. Here is why we made it.

RunpodEventsStream carries filtered, enriched, schema-validated domain events, and it is what both operational consumers and the analytics pipeline read. Operational consumers want exactly this: the billing service wants to know when a billable event occurred, in a stable format, with the required fields, not a stream of raw row diffs it has to interpret. Analytics wants it for a reason that turns out to matter even more, which Part 2 covers in full: reading the same validated stream means Snowflake inherits the schema guarantees the producers already enforce.

CDCEventsStream carries the raw change feed, every row mutation with full column snapshots. Its job is preserving complete row-level history across arbitrary tables for cases where the raw record is what you need. It is deliberately not the analytics source. Feeding analytics from raw row diffs would put us right back in the business of re-deriving domain meaning downstream, which is the exact problem the outbox exists to solve.

The CDC service sits between VStream and both Kinesis streams. It does three things:

1. Filtering. For each row change from VStream, the service decides whether it becomes a domain event on RunpodEventsStream, a raw record on CDCEventsStream, or both.
2. Enrichment. Domain events get wrapped in a CloudEvents v1.0 envelope before they hit RunpodEventsStream. Consistent structure means consumers can deserialize without knowing the source system.
3. Schema enforcement. Event schemas are defined in Zod. The same Zod schema generates TypeScript types for producer and consumer code, JSON Schema for validation, and entries in EventCatalog. Change the schema in one place and the types propagate. There is no separate schema registry to keep in sync.

The result: RunpodEventsStream carries ~2M filtered, enriched, schema-validated domain events per day. The CDC service adds ~48ms of p50 latency. End-to-end latency from DB mutation to when that change is processed and received by a downstream consumer is under two seconds.

Consumer Architecture: Restate for Durable Execution

Kinesis gives you at-least-once delivery. That means every consumer of RunpodEventsStream needs to handle duplicate events without producing duplicate side effects. A billing handler that charges twice for one event is a serious problem. A notification handler that sends the same webhook three times is a problem of a different severity, but still a problem.

Common solutions are idempotency keys and deduplication at the application layer. Every consumer team builds its own retry logic, its own state checkpointing, its own dead-letter queue handling. In practice, teams implement this at varying levels of rigor, and the edge cases surface in incidents.

We use Restate for durable execution instead. Restate persists execution state and retries failed handlers idempotently, which gives you effectively-once execution semantics without requiring each consumer team to build that infrastructure themselves.

Compared to Temporal, Restate has a smaller operational footprint. We don't need Temporal's full workflow orchestration surface. We need durable execution for event consumers, and Restate handles that consistently across every team.

The consumer architecture we're building towards works like this: each Kinesis shard is read by an AWS Lambda function that forwards events to a Restate handler endpoint over HTTP. Restate manages execution durability, retries, and exactly-once semantics from that point forward, so the Lambda stays thin and fast and its only job is delivery. Restate owns everything after the event arrives. This is what we are working towards rather than a description of what's live today. The consumers are actively in development. 

The consumer model from a developer perspective, using the billing consumer currently in development as the example:

const billingConsumer = restate.object({
  name: "billing-consumer",
  handlers: {
    async handlePodStopped(ctx, event: PodStoppedEvent) {
      await ctx.run("record-usage", () => recordBillableUsage(event))
    },
  },
})

Adding a new consumer means writing a Restate handler and pointing it at RunpodEventsStream. No coordination with the producer. No changes to the CDC service. No new infrastructure to provision.

There's a larger operational payoff here, and Command Query Responsibility Segregation (CQRS) is the name for it. The control-plane database stays the single authoritative writer, but the event stream lets us project that state outward into whatever shape a given system needs: a read-optimized store for one consumer, a search index for another, a separate database owned by a different team for a third. Each projection stays current from the same ordered event stream, and none of them adds read load or coupling to the control plane. Distributing control-plane state this flexibly, without every downstream system querying the source of truth directly, is the operational capability the transaction events unlock. Billing and notifications are the first consumers, but the pattern generalizes to any system that needs a live, consistent view of platform state. 

One number worth calling out: the outbox write adds ~5ms of latency to API requests. That is the full cost of this architecture on the hot path, and it is minimal.

Part 2: Analytics Pipeline

Written by Jeremy Jacobs

The Same Schema, One Layer Downstream

Zack's Part 1 solves a trust problem: database state and the event stream cannot diverge, because the database transaction enforces it. Part 2 is about a second trust problem that sits one layer downstream. Can Snowflake's schema diverge from what's actually being published? Six months ago, the answer was "yes, and a human has to notice." Today it's structurally no, for the same reason Part 1's answer is structurally no: one schema, enforced automatically, instead of a convention someone has to remember to follow.

RunpodEventsStream carries CloudEvents-enveloped domain events, validated against JSON Schemas generated from the same Zod definitions the producers write against (see Part 1). Snowflake's analytics pipeline reads that stream directly: the same validated events Restate hands to billing and webhooks. That matters for two reasons: the data arriving in Snowflake is already typed and schema-checked before it gets here, and, more importantly, those same JSON Schemas are the input to the system that keeps Snowflake's own schema honest.

‍

Figure 3. One schema, enforced structurally, twice.

From Domain Event to Typed Bronze Table

The whole ingestion side is three moving parts, and the first one is the only one that touches Kinesis: Snowpipe Streaming writes events from RunpodEventsStream into a single landing table, one row per event, envelope and payload kept as semi-structured columns. Every domain lands in the same place; there's exactly one ingestion path to operate, monitor, and reason about.

From there, each domain (pod lifecycle events, machine events, and so on) gets its own scheduled task that merges new rows from the landing table into a typed bronze table, BRONZE.EVENTS.<domain>, once a minute. Each task tracks its own position independently, so domains never interfere with each other, and a domain with no new events in a given minute costs nothing; the task checks and exits. Because every domain event carries a real event ID, the merge also dedupes Kinesis's at-least-once redelivery on the way in. What comes out is a typed table with normal columns, queryable within about a minute of the event happening, instead of a semi-structured blob every downstream query has to re-parse.

That's the entire pattern, and it's deliberately boring. Adding a domain means adding the same objects every time: a typed table and a task. The pipeline treats a domain doing a few hundred events a day exactly the same as one doing millions; the more than two million events per day flowing through it are heavily concentrated in a few high-frequency domains, and nothing about the pipeline knows or cares. Repetitive, schema-driven work with one repeatable shape is exactly the kind of work worth automating.

Serving in Near Real Time

The silver layer on top is plain views. No incremental jobs and no orchestration to keep fresh: because bronze already resolved dedup and out-of-order arrival at merge time, silver only needs to apply naming conventions and a handful of light transforms (decomposing a polymorphic actor field into user/service/impersonated variants, for instance).

Views are the load-bearing choice here: a view inherits its freshness from the table underneath it, so the moment a merge lands, the event is visible through silver with no additional pipeline step. End to end, an event is queryable in modeled, analyst-friendly form within about a minute of it happening, and the "pipeline" responsible for that last hop is a SQL definition, not a job that can fall behind.

Closing the Loop: Bronze DDL That Can't Drift

Here's the problem this pipeline had until recently. A new field gets added to a domain event's schema in the monorepo. Someone, a human watching for it, has to notice, then hand-write the matching ADD COLUMN in our Snowflake repo, then update the silver view to surface it. Miss that, and Snowflake silently has an incomplete picture of an event that's already flowing correctly through the operational side. This is precisely the kind of manual-synchronization failure mode Part 1 describes for dual-write, except one layer downstream. Until recently, nobody had fixed it here the way the outbox fixed it there.

Automigrate is that fix. The JSON Schemas that validate RunpodEventsStream are the sole source of truth for what BRONZE.EVENTS should look like, not a person's mental model of what changed, a schema diff someone remembered to write, or a ticket that might get deprioritized. When a schema change merges to the monorepo's main branch:

1. A dedicated, read-only GitHub App fires a webhook. The app is installed only on that one repo, deliberately separate from broader-scoped apps so a schema push doesn't fan out noise elsewhere.
2. A receiver verifies the signature, filters to changes that actually touch the event schema path, and fast-acks into an SQS FIFO queue, serialized and deduplicated by commit SHA.
3. A worker fetches the schemas at that commit, diffs them against live Snowflake, and runs a plan → gate → apply sequence.

The gate is the load-bearing piece. It's deterministic, not judgment-based: additive-only changes with no warnings and no orphan columns (new columns, or an entirely new domain's table and task) apply automatically, no human in the loop. Anything else (a type change, a removed field, any ambiguity at all, at either the whole-run or per-domain level) escalates instead of applying. The system is engineered to fail toward "a human looks at this" rather than toward "guess and apply." A daily reconciliation sweep re-runs the same check against head-of-main regardless of whether any webhook fired, because GitHub doesn't retry failed webhook deliveries. The sweep is the guaranteed-convergence backstop.

On a clean apply, the loop doesn't stop at DDL. It publishes to the same event infrastructure Part 1 describes, which our own data-engineering agent picks up to open a PR with the matching silver view definitions: new columns, or an initial silver view for a brand-new domain, generated from the pattern the existing domains already established. A schema change that used to require a human to notice, write DDL, and remember the silver side too now requires nothing, unless the gate decides it should.

Zack's outbox makes database state and the event stream unable to diverge, because the transaction enforces it. Automigrate makes the event schema and Snowflake's bronze schema unable to drift apart, because the schema generates the DDL directly. Same principle, applied to a different pair of systems that used to depend on someone remembering to keep them in sync. And it only works because the pipeline it manages is boring: one landing table, one task per domain, one repeatable shape a machine can generate.

‍

Figure 2: Automigrate control loop

Results

The architecture is running in production. Here are the numbers:

‍

What those numbers replaced:

Lessons from Building Event Streaming at Runpod

1. The binlog is already the truth. If you are running MySQL, Postgres, or Vitess, the CDC stream is there. You don't need to build new infrastructure to get a reliable event log; you need to expose what the database already tracks. The transactional outbox is the bridge between the database's internal truth and the event stream your consumers need.
2. Declare meaning at the source, not downstream. The most valuable thing the outbox does is not atomicity. It is that the producer, the code with the most context, says what happened in domain terms once, and every consumer reads that meaning instead of re-deriving it from raw rows. Before, the data team and each operational system inferred the same events independently and slightly differently. Now there is one authoritative statement of what happened. The correctness benefit is real, but the friction benefit is what teams feel day to day.
3. CDC isn't just for replication. The instinct when you think "change data capture" is to think "database mirror." We use the same change stream to power real-time operational consumers and a full analytics warehouse. The analytics pipeline doesn't mirror the source schema, it reshapes it into purpose-built models. Same source, two completely different downstream shapes.
4. Read your own operational stream for analytics when you can. Ingesting the same validated RunpodEventsStream that powers billing and webhooks means Snowflake inherits schema validation it didn't have to build itself, and creates the dependency that makes automigrate possible in the first place: the schema that keeps producers honest is the schema that keeps Bronze honest.
5. Boring pipelines are automatable pipelines. One landing table, one task per domain, plain views on top: every simplification made the shape more uniform, and the uniform shape is what let automigrate generate new domains mechanically instead of a human building each one. Cleverness per domain would have made the automation impossible.
6. Schema drift is a synchronization problem, and synchronization problems want a single source of truth, not a vigilant human. The dual-write problem in Part 1 and the DDL-lags-the-schema problem in Part 2 are the same shape: two systems that are supposed to agree, kept in sync by someone remembering to update both. The fix is also the same shape: designate one system as the source of truth and derive the other from it mechanically.

What's Next

The bigger direction is event-driven agent infrastructure: workloads that react to system state changes in real time and coordinate across services without polling. The automigrate loop already works this way. A schema change becomes an event, and an agent picks it up to open the silver view PR. That pattern only works with a reliable, low-latency event stream from the database, which is exactly what Part 1's architecture provides.

If you want the conceptual foundation this architecture is built on, the clearest treatment is still Kleppmann's Designing Data-Intensive Applications, specifically the chapters on stream processing and the log as a source of truth. The idea that the database log is the primary record and everything else is a derived view is exactly the mental model we applied here.

If you're working on similar infrastructure, change data capture pipelines, transactional outbox implementations, or durable event consumer patterns, we'd be glad to compare notes. Find us on the Runpod Discord or reach out directly.