Databricks Lakehouse Architecture for Operational Data Replication

Replicating operational data into Databricks is not a matter of picking the right connector and letting it run. The real work sits in three interlocking architectural decisions that no connector makes on its own. A fast connector sitting on top of a storage layer without exactly-once guarantees produces duplicate rows, and a well-built ingestion layer sitting on top of a source without log-based CDC produces gaps between snapshots that nobody notices until a report doesn't add up. Understanding why each layer exists, and what breaks when it's missing, is the actual subject of building a reliable pipeline from operational systems into a lakehouse.

A reliable CDC pipeline into Databricks Lakehouse rests on three layers that each carry a distinct responsibility. At the source, log-based CDC (PostgreSQL WAL with replication slots, or DynamoDB Streams) must be used wherever possible, because snapshot-based alternatives permanently lose intermediate row states and leave no recovery path. Between the source and Delta Lake, Lakeflow Spark Declarative Pipelines and its AUTO CDC INTO API sequence every incoming event by a non-null sortable column (or a STRUCT tie-breaker such as struct("operation_timestamp", "source_sequence")) before applying upserts, so late-arriving records are dropped rather than allowed to overwrite more recent changes. Delta Lake's ACID guarantees and Spark Structured Streaming checkpointing then ensure that a pipeline restart resumes from the last confirmed checkpoint rather than re-applying already-committed changes, and that no reader can observe a half-merged table state. In the medallion model, bronze tables land raw CDC events intact via Auto Loader, silver tables enforce null-key and deduplication expectations ahead of the AUTO CDC flow, and gold tables aggregate from a silver layer that stays continuously current.

CDC's extraction of changes from operational databases without full table scans

Change data capture exists to avoid the alternative: scanning entire source tables over and over to find out what changed since the last pass. Four mechanisms can detect those changes, transaction logs, triggers, timestamps, and native change feeds, and they are not interchangeable. They differ in how much of the change history they preserve, how quickly they surface a change, and how much load they put back on the source database.

Log-based CDC, which reads the database's write-ahead log or its equivalent, is the most trustworthy of the four. It captures changes in the order they were committed, it preserves intermediate states that other methods throw away, and it does this without ever running a read query against the live database. PostgreSQL is a clear example of how this works in practice. Every insert, update, and delete is written to the Write-Ahead Log before it's applied to the actual table, and logical replication decodes that log stream into a format a downstream consumer can read. A replication slot functions as a cursor into that stream, holding onto WAL segments until the consumer confirms it has processed them. A consumer that disconnects and reconnects later picks up exactly where it left off rather than losing anything in between. DynamoDB takes a related but distinct approach: DynamoDB Streams exposes change feeds broken into shards with 24-hour retention, and the stream view type selected (KEYS_ONLY, NEW_IMAGE, OLD_IMAGE, or NEW_AND_OLD_IMAGES) determines how much of the before-and-after state is actually visible to a consumer. When DynamoDB splits a shard to scale a table, producers need to track that new shard topology themselves or risk missing events in the transition.

Not every source offers a log-based feed. Oracle and MySQL deployments without log access available fall back to snapshot-based CDC, which Databricks implements through the AUTO CDC FROM SNAPSHOT API. This method compares two consecutive snapshots of a table and infers the changes between them. The limitation is structural rather than incidental: only the net difference between the two snapshots is ever visible, so a row that changed and then reverted between the two snapshot exports leaves no trace at all. That change is gone, permanently, and no amount of tuning recovers it.

Moving from snapshot-based CDC to log-based CDC later means rebuilding the ingestion layer and, in many cases, fully refreshing every target table that was built on the snapshot approach.

The operational risks that live at the source database boundary

Choosing log-based CDC solves the fidelity issue, but it puts the pipeline's operational risk partly on the production database itself, and teams that ignore this end up debugging incidents on their source systems instead of their pipelines.

The PostgreSQL case illustrates this directly. A replication slot holds WAL files until the downstream consumer confirms it has processed them, and that's a feature right up until the consumer falls behind, whether from a slow downstream job, a reprocessing run, or a compute interruption. When that happens, WAL files pile up on the source, consuming disk and degrading performance on the very OLTP system the pipeline was supposed to leave alone. The way out of that state is binary. If the pipeline catches back up before the WAL retention limit is breached, ingestion resumes cleanly from the last checkpoint. If it doesn't, the only option is a full refresh of the target tables, which erases the incremental advantage that justified using CDC in the first place.

DynamoDB carries its own version of this risk. When a table scales and DynamoDB splits a shard behind the scenes, any producer that isn't actively tracking the new shard topology can miss events that fall in the gap between the old shard and the new ones. Nothing throws an error. The data is simply absent, and the absence is visible only when someone notices a downstream count doesn't match the source.

MongoDB's schemaless collections raise a related concern: fields can appear, disappear, or get renamed without warning, and an ingestion layer that doesn't explicitly handle schema evolution will either fail outright or silently drop data tied to the changed fields. None of these are bugs in a connector. They're the direct architectural consequence of routing live operational traffic through a change feed, and the way to manage them is with deliberate sequencing and schema-evolution logic at the ingestion layer, not by avoiding log-based CDC altogether.

Lakeflow's AUTO CDC APIs: processing changes before they reach Delta Lake

This is where Lakeflow Spark Declarative Pipelines sit: between the raw change feed coming off the source and the Delta Lake table that analysts and models eventually query. The work this layer does is what keeps a messy, out-of-order event stream from becoming an inconsistent table. What used to be called Delta Live Tables is now Lakeflow Spark Declarative Pipelines, the SQL APPLY CHANGES INTO API is now AUTO CDC INTO, and the Python dlt.apply_changes() function is now dp.create_auto_cdc_flow().

Sequencing is the core of what AUTO CDC does. The sequencing column has to be sortable, it can't be null, and it has to represent the true order of events at the source, with exactly one event per key at any given sequence value. A timestamp alone is sometimes not precise enough to guarantee that uniqueness, so Lakeflow supports a STRUCT, such as struct("operation_timestamp", "source_sequence"), as a tie-breaker. The platform orders every incoming event by this column before applying anything, so a record that arrives late gets dropped instead of overwriting a more recent change it shouldn't touch.

On top of sequencing sits the choice between SCD Type 1 and SCD Type 2. Type 1 overwrites a record in place and keeps no history. Type 2 appends a new row for every change and preserves the full history of that record over time. Teams declare which model they want, and Lakeflow generates the merge logic, windowing, and deduplication needed to produce it, which replaces hand-written MERGE statements that tend to grow brittle as source schemas shift underneath them.

For sources without a log-based feed, create_auto_cdc_from_snapshot_flow() compares consecutive snapshots and builds a synthetic change feed out of the differences. It's available only through the Python interface, and it carries the intermediate-state limitation established in the previous section. A newer capability, bitemporal tracking, is currently in Beta and extends SCD Type 2 to track two separate time dimensions at once, business time and system time, which matters for workloads where the moment something happened and the moment it was recorded are genuinely different facts worth keeping separate. AUTO CDC also supports partial updates: a change record that only touches a subset of columns doesn't require the whole row to be re-emitted and reprocessed.

AUTO CDC APIs require either serverless Lakeflow pipelines or the Pro or Advanced pipeline edition. They are not available in open-source Apache Spark Declarative Pipelines.

Delta Lake's ACID layer as a requirement for CDC workloads

Everything AUTO CDC does to sequence and merge changes would be wasted effort if the table it writes to couldn't guarantee a consistent read. Delta Lake's ACID guarantees are what turn a processed stream of changes into a table that behaves like a table, rather than a raw log that every reader has to reconstruct by hand.

Without those guarantees at the storage layer, a reader can land in the middle of a batch being applied and see a half-merged state, a row updated by the merge sitting alongside the still-unmerged version of a related row. Such a query reflects neither the state before the change nor the state after it, a condition arguably worse than stale data because it looks current.

Through Spark Structured Streaming checkpointing, a pipeline that fails and restarts resumes from its last confirmed checkpoint rather than re-applying changes that already committed successfully, which is the same mechanism Databricks relies on internally for its own continuous metric ingestion. Delta's transaction log, meanwhile, keeps a durable record of every change ever applied to a table, giving regulated industries like finance and healthcare the audit trail and lineage their compliance requirements demand.

How does the medallion architecture map to a CDC pipeline end-to-end?

The medallion architecture, bronze for raw ingestion, silver for validated and deduplicated data, gold for analytics-ready aggregation, gives a CDC pipeline a structure where each layer has one clear job.

Bronze is where raw CDC events land exactly as they arrived, operation type, sequence number, and every field value intact, with nothing filtered out yet. Auto Loader handles the incremental ingestion from cloud object storage, whether that's S3, ADLS, or GCS, and infers schema automatically as new fields appear. Databricks' own Lakeflow CDC tutorial demonstrates this with a customers_cdc_bronze table as the bronze target.

Silver is where the data earns its trust. Null key checks, deduplication, and format validation run as explicit data quality expectations before AUTO CDC applies its upserts, producing either a current-state table or a full history table depending on whether SCD Type 1 or Type 2 was declared. The same tutorial builds this as a customers_cdc_clean table, with expectations enforced ahead of the AUTO CDC flow rather than after it.

Gold holds the aggregations, dimensional models, and feature tables built on top of silver. Because silver stays continuously updated by the CDC pipeline feeding it, gold tables can refresh on a short cadence, or be built as materialized views that stay close to current state without a separate batch schedule driving them.

A newer piece changes where bronze's job even starts. Lakehouse Sync, in Public Preview as of May 12, 2026, gives teams continuous, low-latency replication of Lakebase Postgres tables directly into Unity Catalog managed Delta tables using CDC, functioning as the counterpart to change data feeds for teams pulling operational data into the lakehouse rather than exposing it where it sits. Databricks has confirmed support for up to 250 tables per pipeline for PostgreSQL sources. That capability also answers a question data architects have debated for years: whether a classic operational data store, sitting between OLTP systems and the warehouse for near-real-time reporting, is still worth standing up as its own system. With Lakebase CDF, the operational database itself takes on that role, and the separate ODS layer becomes unnecessary.

A hybrid real-time lakehouse serving layer's effect on query latency on replicated data

Getting the pipeline right, bronze through gold, ensures correctness. It doesn't solve the latency problem that occurs when a dashboard or an AI agent needs an answer in milliseconds rather than seconds, a separate architectural question about how replicated data gets served, not how it gets stored.

Lakehouse//RT, built on a new real-time engine called Reyden, is Databricks' answer to that question. It delivers millisecond query speeds directly against the lakehouse, without copying data into a second, purpose-built serving system. That avoids a cost that's easy to underestimate until a team has lived with it: a separate serving layer means a second ingestion pipeline, a second category of failure modes to monitor, and a second schema to keep synchronized with the first. Every time the CDC pipeline changes the underlying Delta table, a separate serving layer has to be re-synced to match, and every schema change on one side has to be propagated by hand to the other. Extracting data out of an open format like Delta into a proprietary store doesn't just add infrastructure. It adds a standing synchronization dependency that has to be maintained for as long as both systems exist.

Zerobus Ingest fits into this picture as the high-throughput, near-real-time write path for event data, built to work without a message bus in front of it. The trade-off is explicit rather than hidden: Zerobus is not designed to route a single event stream to multiple downstream systems at once, and it gives up multi-sink routing, event replay, and exactly-once delivery in exchange for the architectural simplicity that comes with writing directly and fast. That trade-off is the same one running through every layer discussed here: fidelity, latency, and operational simplicity rarely come free together, and the right choice depends on which of the three a given workload can least afford to lose.