The one number that matters: bytes read

Most lakehouse cost is paid at read time — bytes pulled from storage, decompressed, and pushed through a join. The clearest way to compare architectures is therefore the simplest: how many bytes did you have to read to answer the query?

Same query, same 10 GB dataset: Spark + Iceberg read ~1.7–5 GB (whole hour/day partitions, 128 MB row groups) Streaming Lake read 128 MB (1 MB pages, pruned before decode)

Reading ~70× fewer bytes is not a micro-optimization: fewer bytes read generally means less CPU, less memory, lower latency, and lower cost, all at once. Everything below is how Streaming Lake gets there — and where the claim stops.

What we measured

The workload is the hard case: a selective inner join on fresh data, the query pattern that lakehouses handle worst.

SELECT ... FROM Person p INNER JOIN Employee e ON p.personId = e.personId WHERE p.eventTime IN [t0, t0 + window) -- prune by event time (5-min data ledgers) AND p.age <= 52 -- clustered predicate -> prune pages AND e.salary <= 63000 -- clustered predicate -> prune pages

Metadata (the catalog, per-page min/max + bloom footers, and immutable column segments) is built for the full layout, so pruning runs against the true dataset. Only the pages that survive pruning are decoded and joined, and the off-heap build side spills to a real file on disk — so pruning ratios, join correctness, and the off-heap working set are all measured, not estimated.

Results

Measured on this branch (single broker JVM). The narrower the query, the more it prunes — a 1-hour window reads 1/72nd of the data. The join's entire working set for the selective query is 2.2 MB.

Why so little I/O — pruning before you decode

Two design choices do the work:

• Fine-grained pages. Streaming Lake's prune-and-read unit is a 1 MiB page. A lakehouse's is typically a 128 MiB row group (Iceberg's documented default for both the read split and the Parquet row group). A matching row anywhere inside a 128 MiB block forces the whole block to be read; a 1 MiB page is 128× finer, so far less irrelevant data comes along for the ride.
• Metadata-first pruning. Each page carries min/max ranges plus a bloom (or exact set) for text columns, rolled up into immutable per-column segment indexes. Pruning runs on that metadata — before any Arrow batch is decoded — across three levels: event-time → segment → page. Only the survivors are ever read.
• Client-side columnar encoding. Producers send already-columnar Apache Arrow batches with a trailing stats footer. The broker stays a dumb, low-latency pipe: it persists the bytes and slices the footer into a durable page index. It never parses Arrow on the write path.

On the probe side the join is late-materialized: only the join-key column is read per row, and the full row is reconstructed only when a key matches — so non-matching rows never pay to be decoded.

One engine, not five

The other half of the cost story is architectural. The standard stack copies data through five systems before a query can run, and pays for each one continuously:

Producer → Kafka → Spark Streaming → S3 + Iceberg → Spark SQL ▲ ▲ ▲ ▲ brokers always-on storage + shuffle · sort · ETL cluster compaction hash join per query

Streaming Lake keeps a single copy in BookKeeper and serves both pub-sub and analytics from it — no ETL job, no second store, no manifest maintenance, no object-store GET/LIST charges:

Producer → Pulsar → BookKeeper → Segment metadata → Broker query (one durable copy = stream + table + history)

Cost comparison

Modeled on published AWS list prices for a sustained pipeline (~10 MB/s ingest, 7-day hot set, ~1,000 selective joins/day). Streaming Lake numbers are the measured read behavior applied to that infrastructure; the competitor column is modeled from list prices and each system's documented defaults.

No object store, by design. Streaming Lake holds the entire dataset on BookKeeper — there is no S3 line at all. At RF-2 (Pulsar's default) on cheap sc1 Cold HDD the total is ~$2,700/mo (~2.8×); on warmer st1 HDD it's ~$4,255 (~1.8×), and RF-3 on st1 is ~$5,420 (~1.4×). Per byte, replicated HDD is dearer than S3's erasure coding — the win is deleting the object store, the ETL, and the query cluster around it, not the storage rate.

What you actually run — line by line

Streaming Lake spends money only where speed is needed: NVMe on the bookie journal and the query-broker spill, cheap Cold HDD (sc1) for the bulk ledger data, and cheap stateless pub-sub brokers. The five-component stack pays for a Kafka cluster, two always-on Spark roles, and object storage on top of each other.

AWS on-demand list prices, US-East-1 (verify current). The competitor side is modeled from list prices and documented defaults; the Streaming Lake side applies the measured read behavior to this hardware. Discounts (reserved / spot, often 30–70%) apply to both. One honest caveat: sc1 is cold HDD — cheapest per byte, but low random IOPS — so keep the fresh/hot working set on faster media (st1 or NVMe + page cache) via bookie affinity groups; cold data served from sc1 answers in seconds, not the ~1 s measured on NVMe.

How it stacks up

A qualitative read of where the selective-join workload lands across four representative systems:

Each system is excellent at what it was built for. ClickHouse in particular remains a best-in-class analytical engine; the distinction here is architectural — Streaming Lake unifies ingestion, storage, indexing, and query on one source-of-truth log.

What's next

To turn a strong result into an unarguable one, the roadmap for this benchmark is:

• Scale it — 100 GB, 1 TB, 10 TB. The architecture should hold; showing it matters.
• Real baselines — the same SQL on Spark + Iceberg, DuckDB, and ClickHouse, side by side.
• Publish bytes read, CPU-seconds, and shuffle GB directly — not just ratios and latency. "Spark read 10 GB, Streaming Lake read 128 MB; Spark shuffled X GB, Streaming Lake shuffled 0" is the comparison readers can act on.

Reproduce it

The benchmark is a normal broker test, opt-in via an environment variable, driving the same join engine that ships in the query tier:

SL_BENCH_RUN=true SL_BENCH_TOTALGB=10 SL_BENCH_WINDOWHOURS=8 \ ./gradlew :pulsar-broker:test --tests "*StreamLakeJoinBenchmark" \ --no-daemon --no-build-cache --rerun-tasks

Harness: StreamLakeJoinBenchmark. Full methodology, byte-level formats, and the complete cost model live in the project's DESIGN.md and BENCHMARK.md.