CDC Tools That Automatically Handle Column Additions and Drops Without Pipeline Restarts

Automatic schema evolution for live CDC pipelines needs detection, a policy for each kind of change, and updates to the destination that all work together. CDC tools that automatically handle column additions and drops without pipeline restarts must do all three as well as keep replication running. Safe additions can pass automatically, but drops need a policy that protects downstream consumers, and dangerous changes need to be reviewed.

Why schema changes break CDC pipelines silently rather than loudly

At Friday at 5pm, an engineer renames ship_date to shipped_at. The pipeline stays green all weekend. On Monday morning, the revenue dashboard is broken because the destination retains a null-filled ship_date column beside the new shipped_at column, while dashboards still query the old name. Replication appears healthy because events continue moving, but the data no longer means what its consumers expect.

That is the defining danger of schema drift. An explicit failure exposes the broken contract, while a silent failure lets incorrect results accumulate. A change from INTEGER to VARCHAR can leave values truncated, miscast, or silently null without stopping ingestion. Similarly, a new source column can disappear from destination records or cause an ingestion error because the destination has no corresponding column. Monitoring that checks only pipeline uptime cannot establish that the data remains complete.

The problem begins with the contract established at setup. The pipeline interprets source events and writes them into a destination structure, but the core log-based CDC protocol does not itself notify downstream consumers that their contract changed. Schema handling therefore needs its own mechanisms. “No error” is not evidence that evolution succeeded and can be the least informative outcome.

How schema changes differ in risk, and why that classification has to be built into the tool

A change is safe when old and new data can coexist without information loss. Adding a nullable column meets that test because older events can use null for the field they lack. Widening INT to BIGINT also preserves existing values, while adding a table to the CDC scope does not disturb existing streams. These changes need not stop replication. A tool that rejects them all makes routine expansion unnecessarily brittle.

Adding a NOT NULL column without a default leaves existing events without a value they must have and may mean the table needs to be rewritten. Moving from INT to DECIMAL can depend on consumer parsing, despite the wider representation. Drops, narrowing types, and primary key changes are more dangerous because consumers can lose expected fields, values can stop fitting, or row identity can change. Duplicate or missing rows can follow a primary key change.

Renames create a particularly difficult ambiguity for automated handling. A log representation of a drop followed by an add does not distinguish a rename from deleting one field and creating an unrelated one. The tool cannot safely infer that meaning. Propagating both operations without qualification can turn continuity at the source into apparent data loss downstream.

Schema registry compatibility modes turn these distinctions into enforcement policy. BACKWARD lets the new schema read old data, allowing optional additions and widening while blocking drops, renames, and narrowing in this framework. It is the practical default for CDC topics because additions are common and consumers read historical data. FORWARD lets the old schema read new data and blocks required additions without defaults, while FULL needs compatibility both ways, which leaves optional fields with defaults. NONE removes the safety net. BACKWARD_TRANSITIVE and FORWARD_TRANSITIVE compare against every prior version, which is important when consumers replay a topic from its beginning.

What the log-reading layer must do to detect schema changes at all

Postgres logical replication through pgoutput does not emit DDL statements or a dedicated schema-changed event. Instead, a Relation message describes the table’s current columns and types, and later row events are interpreted against that description. The CDC tool must compare it with its internal schema state. Reading the log is not enough by itself. Detection requires database-specific interpretation of what the log exposes.

This mechanism also determines when detection can happen. The tool learns about an added column when the first row event after ALTER TABLE arrives. Its response is reactive and event-driven. A system that promises automatic handling must therefore understand both the signal and its timing, instead of treating source DDL as an immediately available notification.

pgstream v1.0.0, released February 2026 and updated August 2026, addresses that gap with a custom event trigger that captures DDL directly. It emits messages into the WAL through pg_logical_emit_message, allowing downstream processors to interpret the change itself. This avoids a schema log table, a reconstructed schema view, and source database schema state to maintain. It also replicates all DDL statements.

MongoDB has a different source contract through its oplog, which records logical operations on every data-bearing replica set member. Drift is visible differently in document inserts and field updates than in changes to relational columns, but typed warehouse destinations still need a schema they can use. Kafka pipelines using Avro add another gate. After Postgres logical decoding notices the new shape on a row change, the Avro converter attempts to register a new schema version, and the registry applies its compatibility check. Detection is consequently a chain of source interpretation and schema validation.

What the schema management layer must do to propagate changes without stopping

Detection without propagation policy produces only a better error message. Once a change is known, the tool needs to decide whether to apply it, transform it, preserve the previous destination structure, or send it for review. One global response cannot express those distinctions. Automatically accepting ADD COLUMN may be appropriate while automatically accepting DROP COLUMN would break the same consumers.

Flink CDC 3.x makes these choices explicit through configurable schema synchronization modes. exception forbids schema changes and fails, while evolve applies upstream changes downstream. try_evolve attempts those changes but tolerates failures. By contrast, lenient transforms changes to avoid data loss, including preserving original columns on type changes and suppressing destructive drop or truncate table events by default. ignore silently discards schema change events without applying them to the destination. These modes and controls for each event type show how keeping a process alive differs from keeping its output correct.

Drops show why policy must be specific. Retaining a destination column while stopping new values protects consumers more than physically removing it, although it no longer mirrors the source exactly. Physically dropping it matches the source but can break consumers immediately. Routing the change to a dead letter queue instead reserves the decision for human review. None of these outcomes is interchangeable with the others.

The application-level “expand, transition, contract” pattern gives that decision a migration context. A replacement structure appears while the original remains, writers and readers move, and the original is removed after validation. A CDC tool that understands the pattern can allow expansion but hold destructive operations until contraction is confirmed. Before deployment, testing the proposed Avro schema against the registry can expose blocking incompatibilities before ALTER TABLE introduces them into the stream.

What the destination layer must do to absorb schema changes that reach it

INSERT and MERGE bypass Snowflake’s native schema evolution, a consequential gap for CDC pipelines that use MERGE to apply updates and deletes directly. A tool using that path must manage schema changes itself. Native evolution can add columns and relax NOT NULL constraints on supported ingestion paths, including COPY INTO, file-based Snowpipe, and Snowpipe Streaming, but it does not rename columns, change existing column types, or drop columns.

Apache Iceberg uses a different foundation for schema evolution. Each column has a unique field ID, so renames and drops can be metadata changes without rewriting data files. A column gets a new ID when it is added. This keeps it from reading an old column’s data by mistake when a name is reused. Types can change only through safe widening, which includes int to long, float to double, and an increase in decimal precision.

Databricks illustrates why connector behavior belongs in the same assessment. Its SaaS and CDC connectors evolve schemas automatically with an automatic restart on column changes, and dropped columns become soft deletes with NULL in new rows. That is not zero-restart evolution. Type widening and other type changes generally require a full refresh, although some CDC connectors, including the Oracle integrated CDC connector, support widening automatically. The Delta connector with mergeSchema adds columns automatically, but renames and drops need explicit Spark configuration, column mapping, and schemaTrackingLocation or the stream stops.

Auto Loader introduces another restart boundary when automatic evolution encounters a new column. The stream initially fails and must restart to use the evolved schema. For unsupported type changes, rescue behavior depends on rescuedDataColumn and schemaEvolutionMode, with rescue mode capturing those changes when configured. Otherwise, the type does not evolve and mismatched values become NULL, with values routed to the rescued data column in most modes. A CDC tool must compensate for these destination-specific limits rather than assume that accepted source changes will be absorbed intact.

Where AWS DMS specifically fails across these three layers and why it matters as a reference case

AWS Database Migration Service provides a concrete test of the three-layer framework. Its limitations are representative of tools in which schema evolution was not treated as a first-class runtime requirement, rather than evidence that one product is uniquely deficient. DMS can detect a source table change without automatically propagating every required target change.

People may need to step in for complex operations such as dropping columns, changing types, or adding nested structures. Replication tasks may halt or fail silently. In some cases, a task continues running successfully while new fields are dropped without alerts or validation failures. This is primarily a propagation failure, even though its visible consequence appears at the destination. The source change was not converted into a complete, usable target schema.

Diagnosis becomes harder because DMS does not maintain schema history or version tracking. Operators cannot rely on a native version record to reconstruct drift after the fact. The AWS Knowledge Center guide instructs users resolving missing schema changes to set HandleSourceTableAltered to true, resume the task, and reload the table. That is an operator-led recovery sequence. The guide does not require stopping the task, so the limitation should not be described as a universal stop requirement.

AWS has also added GenAI-assisted conversion to DMS Schema Conversion using Amazon Bedrock foundation models. That capability addresses a different stage of the work. It supports one-time database schema conversion rather than runtime handling of schema drift, so it does not close the propagation gap in a live CDC pipeline.

How do you handle upstream schema changes without stopping your CDC pipeline?

The tool must detect changes in a way that fits each source, without an operator having to trigger a schema refresh. For Postgres, that means interpreting Relation messages or capturing DDL directly through an appropriate mechanism. For MongoDB, it means interpreting oplog operations in the context of a typed destination. Each source adapter must expose enough information for the policy layer to identify the change.

That policy must automatically pass safe nullable additions and supported widening while protecting consumers from destructive operations. Soft-deleting dropped columns preserves the destination interface. Primary key modifications and breaking type swaps need quarantine in a dead letter queue with alerting. Ambiguous renames likewise need explicit handling. The tool must base its automatic decisions on what the source signal tells it.

Destination compensation must then complete the operation. A Snowflake MERGE pipeline needs schema-aware DDL before the statements that bypass native evolution. An Iceberg destination needs field IDs respected rather than columns matched only by name, while Delta streaming needs the required configuration for renames and drops. Delivery semantics belong in this same contract. If schema handling introduces a restart, exactly-once guarantees must survive that boundary or schema changes can produce duplicates or gaps.

A managed replication service can put these responsibilities behind one operational interface without requiring the operator to assemble the layers. Its qualification is behavioral, not merely a feature label. It must demonstrate database-specific detection, differentiated policy, destination compensation, and delivery continuity together. “Automatic schema evolution” earns its meaning only when source changes produce the intended destination result without turning routine migrations into operator recovery work.