What Is Lambda Architecture? Batch and Streaming Combined

Lambda architecture is a data processing design pattern that addresses a specific problem: how do you serve both accurate historical data and low-latency recent data from the same system? It solves this by running two separate processing layers in parallel and merging their outputs at query time.

The pattern was articulated by Nathan Marz (creator of Apache Storm) around 2011 and describes a system with three layers: a batch layer for accuracy over completeness, a speed layer for low latency over accuracy, and a serving layer that merges both for queries.

The Problem Lambda Solves

The fundamental tension in data systems: batch processing is accurate but slow; stream processing is fast but complex and error-prone.

Batch pipelines process all historical data, apply complete business logic, and produce fully accurate results — but they run on schedules (hourly, daily) and cannot serve queries about what happened in the last few minutes.

Stream processing handles data as it arrives with low latency — but stream processing systems are complex, stateful operations are hard to reason about, and subtle bugs in streaming logic can produce incorrect results that are difficult to detect and correct.

Lambda architecture resolves this tension by accepting both: run a batch system for accuracy on historical data, run a streaming system for speed on recent data, and at query time merge the results of both.

The Three Layers

**Batch layer** — processes the complete historical dataset on a schedule. Uses a reliable batch processing framework (originally Hadoop MapReduce; more recently Spark). The batch layer recomputes views from raw data on each run, correcting any errors made by the speed layer. Because it processes all data with complete context, results are highly accurate. The trade-off is latency: the most recent data in the batch layer is as old as the last batch run.

**Speed layer** — processes data as it arrives in real time. Uses a stream processing framework (Apache Storm, Spark Streaming, Flink). The speed layer compensates for the batch layer's latency by serving recent data that has not yet been processed by the batch layer. Because stream processing is stateful and complex, the speed layer may produce approximate results — but it delivers them immediately.

**Serving layer** — merges batch views (from the batch layer) and real-time views (from the speed layer) to answer queries. For time ranges covered by the batch layer, the batch view is used (accurate). For the recent time window not yet covered by the batch layer, the speed layer's real-time view fills in. The serving layer must be capable of combining both views in a way that provides complete, approximately correct results.

A Concrete Example

An analytics dashboard shows "orders by region for the last 30 days."

The batch layer runs nightly, producing accurate order counts through yesterday's close. The speed layer processes today's orders as they arrive, maintaining an approximate count since midnight.

When a user queries the dashboard at 3pm, the serving layer returns the batch layer's accurate data for the previous 29 days, plus the speed layer's count for today's orders so far.

The result is approximately correct: the 29-day history is accurate; today's figure is as accurate as the streaming pipeline can make it. By tomorrow, the batch layer will process today's orders and produce a corrected count.

Lambda's Operational Complexity Problem

Lambda architecture was influential and solved a real problem — but it introduced significant operational complexity that became its defining criticism.

**Two separate codebases** — the batch processing logic and the stream processing logic must produce equivalent results but are implemented in different frameworks, often different languages, with different operational characteristics. When business logic changes, both codebases must be updated consistently. Divergence between the two is common and hard to detect.

**Operational overhead of two systems** — running, monitoring, and maintaining both a batch processing cluster and a streaming processing cluster doubles the infrastructure complexity. Failures in either system must be diagnosed and resolved separately.

**Merging complexity at query time** — the serving layer that merges batch and real-time views must correctly identify which time ranges come from which source and handle edge cases correctly. This logic is often subtle and error-prone.

**Speed layer debt** — stream processing bugs produce incorrect approximate results that persist until the next batch run corrects them. If the batch runs daily, users see incorrect data for up to 24 hours. The batch layer's periodic correction is both the feature (it corrects errors) and the acknowledgment that the speed layer produces errors.

Kappa Architecture: The Simpler Alternative

The kappa architecture, proposed by Jay Kreps (co-creator of Kafka) in 2014, proposes eliminating the batch layer entirely. All data is processed through a single streaming layer; historical reprocessing is handled by replaying the event log from the beginning.

The kappa insight: if stream processing is reliable enough and the event log is retained long enough, you can reprocess all historical data through the streaming pipeline whenever batch logic needs to run (for recomputing historical views) or when business logic changes. The "batch" processing is just a streaming replay at faster-than-real-time speed.

**Kappa advantages**: single codebase for all data processing, no separate batch and streaming infrastructure, simpler operational model, no merging logic in the serving layer.

**Kappa requirements**: a durable, replayable event log (Kafka is the standard); stream processing that can handle both real-time and replay workloads; enough retention in the event log to replay the full history when needed.

When Lambda vs Kappa vs Modern Alternatives

Lambda architecture was designed for the Hadoop era, when batch processing was the reliable foundation and streaming was an emerging addition. In 2024, the landscape has changed:

Modern stream processors (Flink, Spark Structured Streaming) are reliable enough for production use cases that previously required batch for accuracy. Modern cloud warehouses can handle near-real-time ingestion (Kafka connector to Snowflake or BigQuery with sub-minute latency). The failure modes that made Lambda's batch correction valuable are less common with mature streaming infrastructure.

For most new analytical systems:

- If sub-minute latency is required and business logic is suitable for streaming: kappa architecture or direct streaming to a warehouse sink

- If 15-minute to hourly freshness is sufficient: batch ingestion (Fivetran) plus dbt — no streaming required

- If a legacy system built on Lambda architecture is operational: it may work fine and the migration cost may not be justified

Lambda architecture is primarily relevant as context for understanding older streaming analytics designs and as a framework for thinking about the accuracy-latency trade-off in any data system.

Our data architecture practice designs modern data platform architectures for both batch and real-time analytical requirements — contact us to discuss the right architecture for your freshness and accuracy requirements.

A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.