Dead Letter Queue Design for CDC Event Processing

Change data capture pipelines need a dead letter queue design that accounts for schema drift, out-of-order write-ahead log entries, type mismatches, and exactly-once constraints, none of which standard DLQ advice covers. Generic guidance handles transient network errors, downstream outages, and retry exhaustion well enough, but it has nothing to say about a replication slot falling behind, a column changing type mid-stream, or what happens when an UPDATE event arrives before the INSERT it depends on. Building a DLQ for CDC means designing for those specifics at every layer: how events get routed, what gets recorded about each failure, and how redrive gets done without corrupting the target system.

CDC-specific DLQ design versus generic event systems

A CDC event is an ordered, stateful record of a committed database change, and it arrives carrying implicit guarantees about sequence, schema, and idempotency that a generic queue consumer never has to think about. Standard dead letter queue advice was built around a narrower set of problems: a service timing out, a downstream dependency going dark for a few minutes, a message that keeps failing until the retry budget runs out. Those problems exist in CDC pipelines too, but they sit alongside a second set that generic advice is silent on, including write-ahead log ordering, schema changes that land mid-stream, type coercion failures between source and target columns, and the idempotency demands that come from replicating row-level state.

The pace at which CDC operates raises the cost of getting this wrong. CDC doesn't offer that grace period. Managed platforms that stream row-level changes to a warehouse in under 60 seconds, Artie among them, make this immediate: a bad event can block the whole pipeline within the same minute it was produced. CDC exists to keep downstream systems in sync with a source database in real time, so when a consumer gets blocked, you don't just get a queue backing up quietly. It's a sync gap, and every second it stays open is a second when the warehouse, the search index, or the downstream service is simply wrong.

The four CDC-specific failure classes that a routing layer must distinguish

Before any routing decision gets made, a CDC pipeline needs a way to tell these four failure types apart, because a layer that can't tell them apart will lump replayable and non-replayable messages into the same bucket, turning bulk redrive into a liability and leaving manual, one-by-one triage as the only safe option. The first class covers transient infrastructure failures: network timeouts, downstream rate limits, a broker that's briefly unreachable. The second class is ordering and state violations, where an UPDATE event names a primary key that doesn't exist yet in the target, or a DELETE shows up before the INSERT it logically follows. Nothing about these messages is corrupt. The third class is type and value mismatches: a field that was an integer in the source schema arrives as a string, or a value overruns the precision of the target column. These failures are deterministic. The fourth class is systemic failure, the case where a bad deployment or a broken downstream dependency is failing every event of a given type at once, which calls for a different response entirely: stopping the line rather than treating each failure as an individual case.

That four-way split sets the routing logic for everything downstream. This principle mirrors one from IBM's event-driven architecture reference work, which separates failures worth retrying, like a temporary service outage, from failures that will keep failing no matter how many times they're attempted, like bad data or broken configuration. When you apply that same split to CDC's failure surface, you get the four classes above.

The retry-then-DLQ topology for CDC event streams

A three-tier Kafka topic structure gives a CDC pipeline a way through transient failures that doesn't send every recoverable event straight to a dead letter queue. The main processing topic, something like db-changes, carries normal traffic. A dead letter topic, db-changes-dlq, catches whatever has used up its retry budget or was flagged as a deterministic failure the first time it was seen.

IBM's reference implementation for its reefer container use case lays out this exact pattern. A separate retry service handles re-calling the downstream dependency, spacing out attempts according to how many times the event has already been tried, and anything that blows past the retry threshold lands on the dead letter topic, where a person can look at it.

Committing the read offset before processing finishes is the right call for a CDC pipeline, because it keeps the main topic moving and it avoids head-of-line blocking, when one stuck message holds up everything behind it. The systemic-failure case needs something that lives outside this per-message retry loop entirely: a circuit breaker that watches the overall failure rate across all event types and pauses consumption altogether once that rate crosses a threshold, rather than letting an entire in-flight batch get routed to the dead letter queue one message at a time after a bad deployment.

DLQ envelope fields for CDC events

If a dead letter queue entry is just the raw payload, it's technically there but useless in practice. Without that context, someone has to inspect each entry by hand, so you can't redrive at any scale without too much risk. The right test for an envelope is whether someone who wasn't on call when the failure happened could look at the entry and understand what went wrong, and whether redriving it is safe.

Some fields belong in any queue's dead letter envelope regardless of what kind of system produced the message: the original topic, partition, and offset for Kafka-backed pipelines; the error message, exception type, and stack trace; how many delivery attempts were made and when the first failure happened; a trace ID tying the entry back to its originating request; and a failure reason code, something like poison, retries_exhausted, or systemic, which is the single field that decides whether a dead letter queue can be triaged in bulk or has to be inspected entry by entry.

CDC adds a set of fields that generic dead letter queue advice never mentions, because generic systems don't have a write-ahead log or a schema contract to track. The consumer version, a build SHA or a deploy ID, matters because most CDC poison messages get produced under one deployment and discovered under a later one, and that field tells the operator whether a fix already shipped or whether redriving the message now will just fail again immediately. And the primary key value lets an operator check whether an ordering violation has since resolved on its own, by looking at whether the row now exists in the target.

Retry state needs to live in persistent message headers, not in a consumer's in-memory counter. Keeping attempt counts in memory is a known anti-pattern: a deployment restarts the consumer, the counter resets to zero, and a poison message that should have been dead-lettered keeps getting retried forever instead.

DLQ trigger conditions in Postgres, MongoDB, and DynamoDB

The three databases that show up most often at the source of a CDC pipeline, Postgres, MongoDB, and DynamoDB, each fail in their own particular way, so the routing and envelope logic has to account for source-specific context.

Postgres reads change events out of its write-ahead log through a replication slot using logical decoding, and that slot has to stay active or WAL starts piling up on disk. Schema changes are the single most common trigger for Postgres CDC failures landing in a dead letter queue. Whatever it decides, the dead letter entry for a schema-drift failure needs to carry the DDL statement that caused it, because a schema evolution team needs that detail to act on. This is exactly the kind of failure that automatic schema evolution is meant to prevent: production CDC services that treat compatible schema changes, like an added column, as a built-in primitive rather than an exception stop those changes from becoming poison messages in the first place.

MongoDB's Change Streams read off the oplog, and resume token management is the recovery primitive that matters most: lose the resume token, and the consumer can't pick back up from where it left off, forcing a re-read from an earlier checkpoint. If you automatically add a column for every new field across many different document shapes, you can end up with a target table full of sparse, mostly-empty columns over time, and this matters for dead letter queue design because the sparseness accumulates in the schema silently instead of showing up as a hard error.

DynamoDB Streams support four stream view types, KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, and NEW_AND_OLD_IMAGES, configured per table depending on what the consumer needs to see. DynamoDB's particular characteristics each create their own path to a dead letter queue: a 24-hour retention window on the stream means a consumer that falls more than a day behind loses those events for good, and a shard split happening mid-processing can produce delivery that arrives out of order, looking to the consumer exactly like an ordering violation even though the underlying cause is different. The dead letter envelope for a DynamoDB failure should record the shard ID and sequence number of the failed event, because those two values tell an operator whether the event is still inside the 24-hour window and recoverable.

Exactly-once constraints and the risks of redrive in CDC DLQ operation

Most production brokers build at-least-once delivery in as the baseline guarantee, so for CDC pipelines you want at-least-once delivery paired with idempotent consumers, not a broker-level exactly-once guarantee that struggles to survive a network partition anyway.

Redrive is where CDC pipelines carry more risk than generic queues, because CDC events describe state, not just independent facts. Reprocessing one out of sequence doesn't just produce a duplicate, it can corrupt the target entirely: an UPDATE applied before its INSERT isn't a harmless repeat, it's simply the wrong result written into the table. A safe redrive follows a short checklist. Use the primary key and operation type fields stored in the envelope to put redriven events back in their correct order, because if a batch of dead letter entries shares a primary key, it has to go back in the sequence the write-ahead log originally recorded.

AI agent workloads raise the bar on idempotency even further than traditional ETL does. A dead letter queue gives that failed task somewhere to stop and wait while someone works out what happened, so it doesn't fire again blind.

Durable-execution platforms such as Temporal apply their exactly-once guarantee to the orchestration logic of a workflow, not to the individual activities that workflow calls out to, and that's the right mental model for anyone operating a CDC pipeline: the orchestration layer can be made crash-safe, but the replication consumer underneath it still has to handle duplicates on its own.

Monitoring a CDC DLQ

Most teams graph dead letter queue depth and stop there, but depth alone can't tell you whether an incident is active or just old and never cleaned up. A large backlog of old messages needs someone to own it, but it isn't, by itself, a sign that something new just broke.

The rate at which new messages arrive is the real incident-detection signal: a sudden spike in arrivals points to an active failure in progress, even if the total depth hasn't crossed any alarming threshold yet. You should size the dead letter topic itself to a meaningful multiple of the average failure rate, so a spike doesn't drop messages before anyone gets a chance to look at them, and you should set retention to match how long it actually takes the team to investigate, including weekends and holidays.

A dead letter queue that keeps growing is often the first sign of a systematic bug or a brand new category of bad input, not just a pile of individual failures to clear one at a time. If nobody drains a dead letter queue, it can't work as a safety net. It's a slower, quieter way of losing the same data.

DLQ design with a managed CDC service

Everything covered here, the three-tier topology, the envelope fields, schema-drift routing, tracking WAL position, sequencing redrive correctly, adds up to a substantial amount of engineering, and that burden grows with every new source database and destination added to the pipeline. Building CDC infrastructure in-house means owning the entire failure surface, not just the happy path: schema evolution handling, exactly-once semantics, failure recovery, and the dead letter queue machinery that makes all of it visible to the people running it.

A managed real-time replication service changes which questions you actually have to answer. Artie, as one example of a managed CDC platform, handles schema evolution, exactly-once delivery, and failure recovery at the infrastructure layer, which turns the design question from "how do we build all of this ourselves" into "which application-layer failures are still ours to own." What a managed service built for this should take off a team's plate is the deeper infrastructure work, replication slot management, resume token recovery, detecting and propagating schema drift, and exactly-once delivery guarantees that have to hold even through failure and redrive. Teams evaluating a managed CDC service are better served asking exactly which of these failure modes it actually absorbs, rather than assuming the label "managed" covers all of it by default.