How Your Warehouse Landing Zone Absorbs Column Additions and Type Changes From Live CDC Feeds

Automatic schema evolution for live CDC pipelines can absorb nullable column additions, but only when the ingestion path supports it or the pipeline applies warehouse DDL itself. Type changes need checks for compatibility, while renames and primary key changes need teams to work together on the source side. The warehouse landing zone’s task is to reconcile events captured before and after an ALTER without losing fields, blocking replication, or corrupting recovery.

Why the warehouse landing zone faces a structural mismatch when a CDC feed evolves mid-stream

A backend engineer adds delivery_tip, a NUMERIC field, to the orders table in Postgres on a Tuesday afternoon and tells nobody on the data team. By Wednesday, the revenue-per-order dashboard is missing tip data, dropping rows, or no longer updating. What happens depends on how the pipeline handles the schema. Events captured before the ALTER have the old shape, while later events include the new field. The landing zone must take in events with both shapes while the change is underway.

This is a structural disagreement. The destination may have no column in which to store the new field, even though the source metadata describes it. A permissive pipeline can silently discard that field while continuing to load rows, leaving the gap hidden until a metric reads wrong. A stricter pipeline rejects the write and stalls replication. If that block reaches the WAL reader, the consequences extend beyond the warehouse to the source database.

What the CDC event stream actually carries before and after an ALTER

A CDC event is a structured row record. Its standard anatomy includes an after-image, a before-image, source metadata containing the server ID, transaction ID and Log Sequence Number, and an operation code. Those codes are c for create, u for update, d for delete and r for snapshot read. The latest Relation message before the row tells the pipeline how to read it.

Postgres’s pgoutput protocol sends refreshed Relation metadata before the first row event captured under the new schema. There is no dedicated schema-change announcement containing the ALTER statement. The pipeline detects the change by comparing that metadata with its cached version. Events in flight around the boundary may or may not contain the new field, depending on the exact LSN at which the Relation message arrives. Consumers must tolerate that normal transition window.

How the warehouse landing zone resolves the mismatch for column additions

Adding a nullable column is the safest change because old events can resolve to null for the new field while new events supply values. Both fit the expanded destination schema. Streamkap classifies this as low-risk and the standard case schema registries are designed around. The destination column still has to be created. Some warehouse ingestion paths can create that column automatically.

A file-based path that writes Parquet to S3 and uses Snowflake COPY INTO can self-heal when ENABLE_SCHEMA_EVOLUTION is set, the COPY statement uses MATCH_BY_COLUMN_NAME, and the loading role has EVOLVE SCHEMA or OWNERSHIP privilege. Native evolution also covers file-based Snowpipe and Snowpipe Streaming, but it does not trigger on INSERT or MERGE. That exclusion is central to live CDC. A pipeline using MERGE to apply updates and deletes must issue warehouse DDL externally before the next MERGE containing the new field.

For a team deploying an addition, the safe sequence starts before the database changes. Streamkap recommends verifying backward compatibility, running the ALTER, confirming that the CDC engine picks up the new column, and checking that post-change events contain it. Automatic detection is only one part of that sequence.

Why type changes do not self-heal and require deliberate intervention

A type change modifies a column that both event populations reference by the same name. Unlike an addition, it does not create a separate place for new values. A strongly typed warehouse needs a destination type that can store the incoming values correctly. Snowflake’s native schema evolution does not change existing column types. A compatible source change therefore does not, by itself, establish that the warehouse will adapt.

Streamkap distinguishes three levels of type-change risk. Widening INT to BIGINT is low-risk because old values remain valid, and schema registries in BACKWARD mode permit it. Changing INT to DECIMAL is medium-risk because consumer parsing depends on the new wire type, even when values fit. Narrowing BIGINT to INT is high-risk because values may not fit, and BACKWARD compatibility blocks it.

The planning step is a schema registry compatibility check before the ALTER runs. Streamkap recommends BACKWARD compatibility for CDC topics because it permits additions while blocking drops, renames and narrowing changes. For changes that cannot be widened automatically, the standard resolution is a full table refresh that re-snapshots the source with the new destination type. That brings back downtime and source load. The decision belongs in migration planning.

Column renames and primary key changes as the cases the landing zone cannot resolve

A landing zone cannot infer migration intent from changed names alone. A rename therefore needs source-side coordination. The standard migration starts by adding the new column beside the old one. The application then dual-writes for a window long enough for all consumers to switch. Only after no consumer references the old column does the migration drop it.

CDC uses the primary key to identify rows, so changing it can leave both old and new keys present as inserts or leave a delete unmatched at the destination. The result can be duplicate rows, missing rows, or both. Streamkap classifies primary key changes as high-risk, together with renames and type narrowing. Neither case can be resolved correctly by destination configuration alone because the required intent must be encoded in the source migration.

How WAL slot behavior and Postgres replication mechanics interact with schema change handling

A rejected warehouse write becomes a source stability problem when the resulting stall stops the replication slot from advancing its LSN. Postgres keeps the WAL that the slot still needs. As the stall continues, pg_wal grows, and disk exhaustion can cause the primary to refuse new writes. A manual schema repair directly lengthens the period of WAL accumulation.

The default max_slot_wal_keep_size = -1 allows unlimited retention by the slot. This favors resumability but exposes the source to disk exhaustion during a long block. A retention cap changes the failure mode. If the connector falls too far behind, Postgres can purge required WAL, invalidate the slot and force a full re-snapshot. The tradeoff is between preserving the replay history and protecting source storage.

PostgreSQL 17, released 2024-09-26, added logical replication failover slot synchronization. Slots created with failover = true propagate to physical standbys through the slotsync worker, allowing a promoted standby to resume CDC without re-snapshotting. This reduces the impact of a primary failure during a schema-change window. It does not resolve the destination mismatch. Detecting changed metadata, applying warehouse DDL and resuming writes quickly still limits the WAL pressure created by that mismatch.

What a bronze landing layer adds when the warehouse cannot absorb the change in place

When a typed warehouse table cannot accept a change and a refresh is not ready, a bronze layer can preserve raw events during the resolution window. The pattern stores raw CDC events as Parquet or Avro files partitioned by timestamp, without enforcing the destination’s strong typing at landing. That separation buys time for schema repair. It preserves data that would otherwise be discarded or rejected.

A semi-structured field in the warehouse can hold raw events in much the same way. BigQuery’s JSON type provides this inline escape hatch. The silver layer must supply deduplication and merge reconstruction before consumers can query that state reliably. Bronze provides a temporary buffer against schema-change failure while the type conflict awaits resolution.

Where exactly-once delivery fits into schema change recovery

Recovery introduces retries as a stalled pipeline resumes or a table is re-snapshotted. Correctness depends on the sink’s treatment of those repeated events. Apache Flink provides exactly-once processing semantics at the stream processor layer, together with stateful operations and windowing for complex transformations. The landing zone still needs a write pattern that tolerates replay.

MERGE provides that property for repeated updates identified by primary key. Applying the same update twice produces the same result as applying it once, which makes MERGE a standard CDC sink statement despite its exclusion from Snowflake’s native schema evolution. During a stall, events accumulate in the replication slot or message bus and replay in order after recovery. The warehouse DDL must precede that replay. Idempotency protects against repetition, but it cannot make an incompatible table accept the buffered event shape.

What to expect from AWS DMS when schema changes arrive mid-stream

AWS documents that in AWS DMS, ALTER TABLE ADD COLUMN may be supported when the task allows it, while an unconfigured task may silently ignore the new column. A team assessing DMS needs to check that its task configuration handles the planned change even when replication is running.

AWS also documents that CDC apply is single-threaded by default, with ParallelApplyThreads set to 0. Full-load tasks, by contrast, load up to 8 tables in parallel by default. Those are different operating paths. Under high-change-rate workloads, CDC lag can accumulate, and a schema change that requires a task restart extends that delay. AWS states that DMS offers no SLAs for CDC latency.

An incompatible type change may require a re-snapshot in DMS and other systems because the typed destination cannot accept the new representation in place. DMS requires manual schema-change handling, with documented failure modes including silent omission of an addition and a recovery process that increases replication lag.

What a managed replication service handles that a DIY pipeline requires the team to build

A DIY pipeline must connect every stage of schema handling into one recovery sequence. It needs to compare Relation messages in the way Postgres requires, judge the risk, run destination DDL, manage stalls and check that the change worked. Each stage has a separate failure mode. Detecting a column addition without altering the destination still loses the field, while altering the destination after replay starts leaves the original write failure unresolved.

The operational work continues after the schema decision. WAL pressure needs management during the stall, replay needs primary-key idempotency, and incompatible type changes need re-snapshot orchestration. A managed replication service’s scope is therefore best judged against those named responsibilities. Compare which steps it performs automatically, which require configuration, and which remain coordinated source migrations.

File-based nullable additions can fit Snowflake’s native evolution, while MERGE-based CDC requires external DDL orchestration. Renames, primary key changes and incompatible types still need teams to work together, even with a feature flag or a delivery guarantee. Building that handling from scratch takes engineering work outside the team’s product, and the brief’s two-year estimate for production-grade CDC infrastructure is an upper bound on that work.