Event-Driven Data Architecture: Building with Events as the Source of Truth

Most enterprise data architectures are state-oriented: they record the current state of things. The orders table shows the current status of every order. The customers table shows the current state of every customer record. When state changes, the record is updated.

Event-driven architecture is different. Instead of storing current state, it stores a log of everything that happened — every event that caused state to change, in the order it occurred. The current state of any entity is derived by replaying all the events that affected it. This inversion — events as the primary record, state as a derived view — has profound implications for how data flows, how systems integrate, and how historical data is preserved.

The Core Concept: Events as the Source of Truth

An event is an immutable record of something that happened at a specific point in time. Events are not updated; they are never deleted. The event log is append-only.

Examples:

- order_placed (order_id, customer_id, items, total_amount, timestamp)

- order_status_changed (order_id, old_status, new_status, timestamp)

- payment_received (order_id, amount, payment_method, timestamp)

- order_shipped (order_id, carrier, tracking_number, timestamp)

To find the current status of an order, you query the most recent order_status_changed event for that order. To find the order history, you query all events with that order_id in timestamp order. To find revenue for a period, you sum the payment_received events in that period.

This is different from the traditional approach where the orders table has a status column that is updated when the status changes. With that approach, you can find the current status but you cannot recover the full state history without additional change data capture infrastructure.

Apache Kafka as the Event Backbone

Apache Kafka is the dominant infrastructure for event-driven architectures. Kafka provides:

**Durable, ordered event logs:** Kafka topics are partitioned, append-only logs. Events within a partition are strictly ordered by offset. Events are retained for a configurable period (typically days to weeks) or indefinitely.

**Consumer groups:** Multiple independent consumer groups can read from the same topic simultaneously without interfering with each other. The analytics pipeline, the notification service, and the risk scoring system can all consume the same order events independently, each at their own pace.

**Decoupling:** Event producers (the order service) do not need to know who consumes their events. New consumers can be added without modifying the producer. This decoupling is the architectural property that makes event-driven systems composable.

**Log compaction:** For key-based topics, Kafka's log compaction retains only the most recent event for each key — effectively providing the current state for each entity while still using the event log model.

In a typical event-driven analytics architecture, operational systems publish events to Kafka topics. A stream processing layer (Flink or Kafka Streams) processes those events — enriching them, joining event streams, computing aggregations — and writes results to the data warehouse or a downstream service. The data warehouse may also consume events directly via Kafka connectors.

Event Sourcing

Event sourcing is the pattern of using the event log as the primary persistence mechanism for application state — not just for analytics. In an event-sourced system, the application database stores events; the current state of any entity is computed by replaying events.

The database for an event-sourced order management system does not have an orders table with a status column. It has an events table (or an event store like EventStoreDB or Axon). The "current state" of an order is a projection — computed by replaying the order events.

Projections can be materialised as read-optimised views. The orders_current_state projection is a table that stores the latest status of each order, updated every time an order event is processed. It is derived from the event log, not the source of truth.

### Benefits of Event Sourcing

**Complete audit trail:** Every state change is recorded as an event with a timestamp and the information about what changed. The audit trail is a natural consequence of the architecture, not an add-on.

**Temporal queries:** Because every historical state is preserved in the event log, you can query "what was the state of this order at time T?" without needing time-travel database features — you replay events up to time T.

**Event replay for system recovery:** If a projection is corrupted or needs to be redesigned, you replay the event log from the beginning to rebuild it. The event log is the source of truth; projections are derived views that can be regenerated.

**Decoupled consumers:** New analytical use cases can be served by building new projections from the same event log, without touching the production application.

### Challenges of Event Sourcing

**Schema evolution:** Events are immutable, but event schemas evolve over time. How do you handle a new field added to an event type that did not exist in historical events? Event versioning strategies (upcasting, event adapters) add complexity.

**Eventual consistency:** Read models (projections) are updated asynchronously as events are processed. In a high-throughput system, projections may be slightly behind the event log. Applications that require immediate read-after-write consistency need to either read from the event log directly or wait for the projection to catch up.

**Query patterns:** Querying an event log for complex aggregations is less intuitive than querying a normalised relational database. Teams accustomed to SQL against current-state tables need to adjust their mental model.

CQRS: Command Query Responsibility Segregation

CQRS separates the write model from the read model. Commands (writes, state changes) are processed against the event log. Queries (reads) are served from pre-built read models (projections) optimised for specific query patterns.

In the order management example:

- The write model accepts commands: PlaceOrder, UpdateOrderStatus, RecordPayment. Each command is validated and, if valid, produces an event appended to the event log.

- The read models are projections built from the event log: orders_by_customer (read model for customer order history), orders_by_status (read model for fulfilment team dashboard), revenue_by_period (read model for finance reporting).

CQRS enables each read model to be optimised independently for its specific query pattern. The revenue_by_period projection is a pre-aggregated table by day and category — extremely fast for finance queries. The orders_by_customer projection is indexed by customer_id for fast customer portal lookups. Neither projection needs to serve both query patterns efficiently.

When Event-Driven Architecture Is Worth It

Event-driven architecture introduces significant complexity. It is the right choice when:

**Audit trail is a regulatory requirement:** Financial services, healthcare, and government environments where every state change must be auditable and retraceable benefit directly from the event log model.

**Multiple systems need to react to the same events:** When an order placement needs to trigger inventory reservation, customer notification, fraud scoring, and analytics pipeline update — all independently — Kafka-based event distribution is architecturally cleaner than polling or direct service calls.

**You are building a data-producing system from scratch:** Adding Kafka event publishing to a new system is much less costly than retrofitting it to an existing system. If you are designing a new order management or customer engagement system, event-driven design is worth considering from the start.

**Real-time analytics on operational data:** Operational events in Kafka are the most reliable source for real-time analytics. Streaming the same events that drive operational systems directly into an analytics pipeline eliminates the polling and CDC complexity of extracting data from operational databases.

When Traditional Architecture Is Better

**For existing systems where state-oriented data is already available:** Adding event sourcing to an existing CRUD system is a significant refactoring. If the analytical need can be served by change data capture (extracting changes from the operational database) and batch ELT, the simpler approach is usually preferable.

**For teams without Kafka and stream processing expertise:** Kafka requires operational expertise. Running Kafka in production — managing brokers, partitioning, consumer groups, schema registries — is a non-trivial commitment. The cloud-managed options (Confluent Cloud, AWS MSK) reduce the operational burden but not to zero.

**For analytical workloads that do not need event granularity:** If the analytical use cases are served by hourly batch loads of current state, the additional complexity of event-driven architecture does not deliver proportionate value.

Our data architecture consulting practice designs event-driven data architectures and streaming pipelines — contact us to discuss whether event-driven architecture fits your analytical requirements.

A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.