Real-Time Feature Store Architecture Backed by CDC

A feature store has to do two jobs at once, and those jobs pull against each other. The offline store needs to reproduce exact historical feature values so a model can be trained correctly, usually sitting in a columnar warehouse like BigQuery, Snowflake, or a data lake, built to support point-in-time lookups for training data. The online store needs to serve the freshest possible values the moment a model makes a prediction, usually a low-latency key-value layer like Redis or DynamoDB. The architecture most teams built in the batch era handles the first job well and fails the second by design. A Spark job runs on a schedule, computes features, writes them to Redis, and the model reads from Redis at inference time. For anything that needs to reflect what just happened, a fraud score based on the last ten minutes of transaction activity, a support ticket's current status, a price that should update when inventory changes, the schedule becomes the failure. Feature freshness is one of the most common reasons machine learning models underperform once they leave the lab and enter production. A second failure compounds the first: training reads from the warehouse and serving reads from Redis, and those two systems rarely compute the same feature the same way. Different join semantics, different null handling, different window boundaries mean the feature a model trained on and the feature it sees in production can diverge even when the batch job ran an hour ago. That divergence, known as training-serving skew, means a model can perform well in evaluation and still make bad calls in production, because the input it was tuned on was never quite the input it actually gets.

Why 2026 AI agents make the freshness gap intolerable

AI agents that approve transactions, route support tickets, adjust prices, or generate personalized offers act on data directly, so a stale feature doesn't just produce a stale number. By 2026, AI agents run in production at organizations of every size, and the decision loop they operate on has shrunk to seconds, a world away from the overnight batch cycle that feature pipelines were built around. A new category of infrastructure, agentic data streaming, exists specifically to close this gap: it combines CDC, stream processing, governance, and agent-native APIs to keep continuously delivering real-time data to AI agents rather than to the dashboards and reports humans used to read. InHire offers a concrete case. The company used a CDC pipeline from DynamoDB, built on Streamkap, to stream every change (new applications, profile updates, status transitions) directly into the data layer its AI agent queries. Batch latency isn't something you can tune your way out of in this context. It makes one the instant it's asked. This is part of why Databricks acquired Tecton in 2025, a move that pushed teams not already committed to the Databricks ecosystem to look harder at infrastructure-native alternatives for real-time features rather than treating batch feature stores as the default.

Log-based CDC as the data supply chain for a real-time feature store

Log-based change data capture is the mechanism that closes the freshness gap, because it propagates row-level changes from the source database as a continuous, ordered stream of events, with minimal overhead on the system producing them. CDC can work three ways: log-based, which reads transaction logs like the MySQL binlog or the PostgreSQL WAL; trigger-based, which captures changes into shadow tables through database triggers; and query-based, which polls tables on a timer using timestamps or version columns. The architectural shift this enables is straightforward to describe and significant in effect: instead of a Spark job reading a snapshot of the database on a schedule, the feature pipeline subscribes to the log and processes each change event as it arrives, so a feature value updates within milliseconds of the underlying row changing. This same shift also resolves training-serving skew. When a feature is defined as a streaming materialized view over the log, training and serving pull from the same computation logic instead of two pipelines built separately by two teams at two different times, the usual source of skew.

How PostgreSQL, MongoDB, and DynamoDB expose their logs for CDC

Each database exposes its internal log differently, and the differences matter once a pipeline has to survive a restart, a failover, or months of continuous operation. PostgreSQL CDC reads the Write-Ahead Log, the same log Postgres uses to recover from a crash, and decodes it into a stream of row-level insert, update, and delete events. The replication slot has one job: it tells Postgres not to recycle WAL segments past the point the consumer has last read. That job is also the source of the most common failure mode. PostgreSQL 18 made parallel streaming the default for subscriptions and added the option to replicate generated columns, but that setting defaults to off, so a team has to turn it on deliberately.

MongoDB's answer is Change Streams, introduced in version 3.6 and substantially improved in 4.0. The hard prerequisite is that Change Streams only work on replica sets or sharded clusters. A standalone MongoDB instance keeps no oplog, so there's nothing to stream from.

DynamoDB exposes item-level changes through DynamoDB Streams. The InHire case described earlier ran on this mechanism, so its agent could see every application and status change without waiting on a batch cycle.

The two CDC architectures for feeding a feature store

Choosing a CDC architecture for a feature store means choosing between operational surface area and integration flexibility, and by 2026 the two dominant approaches have diverged enough that picking the wrong one for a team's size is a real liability. The first is the connector-to-Kafka model. A CDC connector reads the source log and publishes events to Kafka topics, and a separate stream processor, Apache Flink being the common choice, consumes those topics, computes features, and writes them to both the online store and the offline lakehouse. Kafka itself, with Zookeeper or KRaft behind it, Connect workers, a schema registry, a stream processor, and separate serving infrastructure means five or more systems running before a single feature gets written.

The second approach is the integrated streaming database model. A system like RisingWave reads the transaction log directly, processes changes using incremental SQL materialized views, and serves results over the PostgreSQL wire protocol, so capture, transformation, and serving all live in one system. It cuts out the Kafka layer and the separate stream processor and shrinks the operational footprint substantially, with built-in CDC connectors for PostgreSQL, MySQL, MongoDB, and SQL Server, so there is no separate connector infrastructure to run. Between 2023 and 2026, operational budgets shrank across the industry, and teams have been actively cutting tool sprawl, a real force pushing architectures toward fewer moving parts.

A managed CDC service changes this calculation for teams that don't want to run either stack themselves. Artie, as one option, streams row-level changes from the source database to the warehouse in under 60 seconds, letting a feature pipeline subscribe to that stream to keep its online store continuously current instead of waiting on the next scheduled job, closing the staleness gap that causes training-serving skew. Underneath either architecture, Apache Iceberg has become a common unifying layer for storage: it provides ACID transactions, schema evolution, and time travel, so CDC events written by either the Kafka-based pipeline or the integrated streaming database become immediately queryable by Spark, Trino, Presto, or any other Iceberg-compatible engine.

Exactly-once delivery in a CDC feature pipeline as a sink problem

Exactly-once delivery is a property a team has to build at the sink, through idempotent writes and deterministic versioning, because retries and duplicates are normal, expected behavior at every stage of a production CDC pipeline, not edge cases to be designed around later. Out-of-order events are one of the clearest ways this chain breaks. An update arriving before its corresponding insert will leave feature state inconsistent, usually the result of parallel partitions or ordinary network delay, and the fix is to order by LSN or log offset, deduplicate on primary key plus version, and never trust the order in which events happen to arrive. At the sink, a reconciliation pattern based on conditional upserts handles the three change types correctly: an UPDATE modifies the existing row, a DELETE removes it, an INSERT creates it, all applied idempotently by matching on the entity's primary key. For a feature store specifically, the online store, whether Redis or DynamoDB, has to accept upserts keyed on entity ID and feature version, and a replay of an older event must never overwrite a newer feature value, which is what LSN-based versioning prevents.

None of this removes the need to plan for failure. A replication slot can get cleaned up unexpectedly because of a WAL retention policy, a network glitch can force a connector restart, a primary database promotion can invalidate a batch that was mid-flight, and each of these needs a tested recovery runbook on hand, not just an alert that fires after the fact. The realistic target for a production pipeline is at-least-once delivery from the log paired with an idempotent sink. That combination is achievable and has been proven in production repeatedly. True exactly-once delivery across heterogeneous systems end to end isn't a guarantee any pipeline can honestly make.

Schema evolution as the most common reason a feature pipeline breaks in production

A CDC feature pipeline that handles schema changes by hand will break the first time a developer adds a column to a source table, because DDL changes don't replicate through native logical replication on their own. For AI agent pipelines, the stakes are higher still. Handling schema changes automatically, so a pipeline doesn't break the moment someone adds a field to a table, is an operational resilience requirement for any system feeding an agent, not a convenience worth paying extra for. Teams evaluating CDC tools for a feature store should test this directly before committing to one: add a column to a source table while the pipeline is running, and watch whether the downstream feature definition updates cleanly, throws an error, or silently drops the new field.

Putting it together: the CDC-to-feature-store data flow from log to model serving

A production-grade real-time feature store built on CDC breaks down into five layers, and the whole system's latency and correctness are only as good as the weakest one in the chain, which in most implementations turns out to be the data supply chain between the source database and the online store, not the model or the serving layer that gets most of the attention.

Layer one is source log capture. Layer three is feature computation, where streaming SQL materialized views compute aggregations, joins, and transformations incrementally as each event arrives, cutting out the batch computation step entirely and making features available within milliseconds of the underlying data changing. Defining features this way, as materialized views, means training and serving run on the same computation logic: training reads a historical export from the view, serving reads the live view, and training-serving skew disappears by construction.

Layer four is dual-store serving. The online store, Redis or DynamoDB, gets populated through idempotent upserts keyed on entity ID and feature version, built for sub-millisecond lookups at inference time, while the offline store, whether Snowflake, BigQuery, or an Iceberg table, accumulates the full history needed for training data generation and point-in-time lookups. Feature freshness at serving time is a function of layers one through three, not layer five. A team chasing better serving-layer performance while ignoring the capture and transport layers is solving the wrong problem.

A common variant of this architecture routes CDC events through Flink or Spark Streaming into Apache Iceberg, which brings ACID transactions, schema evolution, and time travel to the table and makes those events immediately queryable by any Iceberg-compatible engine, unifying the offline store with the streaming layer. Managed CDC services handle layers one and two as a turnkey offering, including exactly-once delivery guarantees, automatic schema evolution, and failure recovery, which lets a team's engineering effort go toward feature computation and serving logic instead of toward building log capture and transport infrastructure from scratch. That's where most teams are genuinely under-invested, and it's also where a managed layer creates the most leverage, because the supply chain feeding the feature store matters more to the final result than almost anything built downstream of it.

Monitoring feature freshness and knowing when the supply chain has failed

A real-time feature store can degrade quietly. Connector lag can build up, a replication slot can stall, a schema change can drop a field, and none of it necessarily trips an error at the model serving layer. Feature freshness monitoring has to be designed into the pipeline from the start. In the connector-to-Kafka model, consumer group lag is a useful leading indicator: if lag keeps growing under stable write volume, the consumer is falling behind, and stale features appear downstream before the model ever flags a problem.

On Postgres, WAL accumulation works as both a freshness signal and a system health check. All of this belongs in the feature store's operational runbook, alongside concrete alerting thresholds. A freshness SLO, something like requiring features to reflect changes within 30 seconds under normal load, gives a team an actual target to monitor against. For a pipeline feeding an AI agent, a stale feature is a wrong decision made on the agent's behalf, so the monitoring posture has to treat feature freshness as a correctness requirement for the system, not a metric to optimize when there's time for it.