Watch it run: a ~1 TB query demo on a single host

The ratios above come from a compact harness. Here is the same engine on a much larger, in-the-wild dataset, recorded end to end: two event-time tables — Person (62.48B rows, 500 GiB) and Employee (108.72B rows, 500 GiB), ~1 TB together — queried live from a single broker. Nothing is pre-aggregated; every figure on screen (pages pruned, bytes read, latency) is printed by the engine for each query.

Why this scale is the whole point. Those tables are a single day of a topic ingesting at ~5 MB/s — about 500 GB/day. Let it run for a year and each table becomes 500 GB × 365 ≈ 178 TB. Because Streaming Lake prunes to only the pages a query touches, the cost of this one-day query barely moves whether the table holds 500 GB or 178 TB — you pay for the answer, not the archive.

What the video shows — every number measured live

Single broker JVM; GROUP BY and joins are backed by an off-heap LSM store spilling to local NVMe — no Spark, no ETL. A terabyte-scale inner join returns in 151 ms after reading 6.6 MB; the hardest 5-predicate join returns in ~1 s reading 7 MB.

Why Streaming Lake is the big winner

Every system here is excellent at something. What the video makes concrete is that Streaming Lake wins the whole pipeline for fresh, selective analytics — not just one stage of it:

• vs Kafka + Spark + Iceberg — no cluster to spin up, no manifests to scan, no 128 MiB row groups dragging along data you never asked for. The join in the video reads 6.6 MB and answers in 151 ms; a Spark job hasn't finished starting in that time.
• Native pub-sub — zero data copy — Streaming Lake is the message log: producers publish, consumers subscribe, and that same durable ledger is exactly what SQL reads. No CDC connector, no Kafka → Spark → Iceberg materialization, no second store to keep in sync — the stream you publish is the table you query. A Kafka + Spark + Iceberg pipeline has to write at least one full copy of the data before the first query can run; Streaming Lake queries the original bytes in place.
• Native geo-replication — the log replicates across regions natively: the same durable ledger is mirrored to other clusters for disaster recovery and read locality. Your queryable table is multi-region out of the box — no external replication job, no cross-region ETL, no second copy to reconcile. Publish once, and both the stream and the SQL table follow your users around the globe.
• vs ClickHouse — a best-in-class analytical engine, but you must first load / ETL the stream into its own store. Streaming Lake queries the log in place — the data is already there, live, with zero copies.
• vs Lance & other columnar files — no separate dataset to write, register, and compact; the durable BookKeeper ledger is the columnar store, indexed as it lands.
• vs Snowflake — no external ingestion pipeline and no warehouse to keep warm; freshness is live and there is no per-query warehouse credit.

The common thread: everyone else pays to move data into a second system before they can query it, then pays again to read far too much of it. Streaming Lake keeps one copy on BookKeeper, prunes at the 1 MiB page level, and runs the join right next to the storage — so a query over a terabyte touches kilobytes to a few megabytes and finishes in milliseconds. That is the big win: not merely a faster SQL engine, but an architecture where cost tracks the answer, not the dataset.

Pruning, concretely: the second predicate is where it separates

In one line: for a filter off the sort key, Iceberg prunes at the 128 MB row group while Streaming Lake prunes at the 1 MiB page — on every column, exactly. Both engines skip data with metadata before reading it, so the win is in how fine that metadata is and what it can promise. Iceberg's finest equality index — a Parquet bloom — is per row group and probabilistic, and its per-page min/max only prunes the one column a file is physically sorted by. Streaming Lake keeps min/max plus a bloom plus an exact SET(N) on every 1 MiB page of every column, so a selective filter on a non-sorted column still narrows to a 1 MiB page — 128× smaller than the 128 MB row group Iceberg falls back to when its per-page min/max can't prune — and to an exact answer, not a “maybe.”

Why can't Iceberg match that? Two limits. First, min/max only prunes where a column is physically clustered, and a file has one physical order: a single sort key clusters one column tightly, and Z-order spreads that tightness across a few columns but loosens each. Any column that isn't clustered has wide, overlapping per-page ranges, so its min/max says “maybe” everywhere — exactly the age = 35 case. Second, the equality index that would cover a non-clustered column — the bloom — is per column chunk (row group) and probabilistic; there is no per-page bloom and no exact SET(N) in Parquet. So a match narrows, at best, to a 128 MB row group, and for a dense value like age = 35 the row-group bloom matches everywhere and prunes nothing. (A bloom on a sparse, high-cardinality column does let Iceberg drop whole row groups — but still at row-group, never page, granularity.)

Put numbers on it. From the demo, layer a non-sort filter on a key slice: personId 0–1,000,000 and age = 35 over the 500.19 GiB Person table. Both engines prune the personId slice; only Streaming Lake also prunes age down to pages:

Same rows back — but on the second (non-sort) predicate Streaming Lake skipped 8,976 of the 10,000 candidate pages and read 29.1 MiB, while Iceberg, clustered on personId, can page-skip that key — and even a bloom on age only works per row group — so it scans the whole personId slice to find the matches. A bloom changes which row groups you open, never how much you read once a matching row is inside one. The index that decides skip vs read lives at the page, on every column, and confirms membership exactly — not at the row group, on the clustered columns, and only probably.

Where ClickHouse specifically breaks down

ClickHouse is a superb scan-and-aggregate engine, but its MergeTree design — one physical sort order per table, indexed only by a sparse key index over 8192-row blocks — leaves four gaps that Streaming Lake's per-page, per-column indexing closes:

• Filters on a non-sort-key column — ClickHouse sorts each table one way and has no per-row or secondary index, so a predicate on any other column can't prune: its values are scattered across every block, so minmax/bloom/set skip indexes say “maybe” everywhere and it full-scans. Streaming Lake keeps min/max + bloom + exact SET on every column at 1 MiB-page granularity, so multi-column, non-key predicates prune to the exact page — with no global sort to commit to.
• One physical order, or many copies — the only ClickHouse fix for a second access pattern is a projection or re-sorted materialized view: another physical copy, re-merged on every insert. Streaming Lake keeps one durable copy with every column indexed as data lands — no re-sorted duplicates, no merge write-amplification.
• Point lookups — with no per-row index, returning a single row still reads and decompresses a whole 8192-row block per column (many blocks if the key isn't the sort key). Streaming Lake prunes straight to the ~1 MiB page that holds the row — the demo's point lookup returned 21 rows reading just 586 KiB in 5 ms.
• High-QPS point serving — each ClickHouse query grabs many CPU threads, so thousands of tiny concurrent queries thrash threads and memory and hit its ~100-query concurrency cap; teams bolt a Redis/KV cache in front (yet another system and copy). Streaming Lake's tiny, page-pruned working sets and query-in-place on the log suit selective, concurrent access without a separate serving tier.

In fairness, ClickHouse is excellent at big full-table aggregation scans with a well-chosen sort key — that is its wheelhouse. The point is architectural: for fresh, selective, multi-column analytics on a stream, it forces a second system, a full copy, and a single sort order that Streaming Lake doesn't.

The one number that matters: bytes read

Most lakehouse cost is paid at read time — bytes pulled from storage, decompressed, and pushed through a join. The clearest way to compare architectures is therefore the simplest: how many bytes did you have to read to answer the query?

Same selective join, ~1 TB dataset (500 GB ⋈ 500 GB): Spark + Iceberg read many GB (whole matching partitions, 128 MB row groups) Streaming Lake read 6.6 MB (1 MB pages, pruned before decode)

Reading megabytes where a lakehouse reads gigabytes is not a micro-optimization: fewer bytes read generally means less CPU, less memory, lower latency, and lower cost, all at once. Everything below is how Streaming Lake gets there — and where the claim stops.

What we measured

The workload is the hard case: a selective inner join on fresh data, the query pattern that lakehouses handle worst.

SELECT ... FROM Person p INNER JOIN Employee e ON p.personId = e.personId WHERE p.eventTime IN [t0, t0 + window) -- prune by event time (5-min data ledgers) AND p.age <= 52 -- clustered predicate -> prune pages AND e.salary <= 63000 -- clustered predicate -> prune pages

Metadata (the catalog, per-page min/max + bloom footers, and immutable column segments) is built for the full layout, so pruning runs against the true dataset. Only the pages that survive pruning are decoded and joined, and the off-heap build side spills to a real file on disk — so pruning ratios, join correctness, and the off-heap working set are all measured, not estimated.

Results

Measured live on a single broker JVM over the full ~1 TB dataset. The more selective the query, the more it prunes — the inner join skips 98.7% of candidate pages and reads just 6.6 MB; the five-predicate join skips 99.4% and reads 7 MB. Joins run on an off-heap LSM store spilling to local NVMe — no Spark, no ETL.

Why so little I/O — pruning before you decode

Two design choices do the work:

• Fine-grained pages. Streaming Lake's prune-and-read unit is a 1 MiB page. A lakehouse's is typically a 128 MiB row group (Iceberg's documented default for both the read split and the Parquet row group). A matching row on any non-clustered filter inside a 128 MiB block forces the whole block to be read (min/max page-skipping only helps on the columns the file is clustered by, and Parquet blooms prune whole row groups, not pages); a 1 MiB page indexed on every column is 128× finer, so far less irrelevant data comes along for the ride.
• Metadata-first pruning. Each page carries min/max ranges plus a bloom (or exact set) for text columns, rolled up into immutable per-column segment indexes. Pruning runs on that metadata — before any Arrow batch is decoded — across three levels: event-time → segment → page. Only the survivors are ever read.
• Client-side columnar encoding. Producers send already-columnar Apache Arrow batches with a trailing stats footer. The broker stays a dumb, low-latency pipe: it persists the bytes and slices the footer into a durable page index. It never parses Arrow on the write path.

On the probe side the join is late-materialized: only the join-key column is read per row, and the full row is reconstructed only when a key matches — so non-matching rows never pay to be decoded.

One engine, not five

The other half of the cost story is architectural. The standard stack copies data through five systems before a query can run, and pays for each one continuously:

Producer → Kafka → Spark Streaming → S3 + Iceberg → Spark SQL ▲ ▲ ▲ ▲ brokers always-on storage + shuffle · sort · ETL cluster compaction hash join per query

Streaming Lake keeps a single copy in BookKeeper and serves both pub-sub and analytics from it — no ETL job, no second store, no manifest maintenance, no object-store GET/LIST charges:

Producer → Pulsar → BookKeeper → Segment metadata → Broker query (one durable copy = stream + table + history)

Cost comparison

Modeled on published AWS list prices for a sustained pipeline (~10 MB/s ingest, 7-day hot set, ~1,000 selective joins/day). Streaming Lake numbers are the measured read behavior applied to that infrastructure; the competitor column is modeled from list prices and each system's documented defaults.

No object store, by design. Streaming Lake holds the entire dataset on BookKeeper — there is no S3 line at all. At RF-2 (Pulsar's default) on cheap sc1 Cold HDD the total is ~$2,700/mo (~2.8×); on warmer st1 HDD it's ~$4,255 (~1.8×), and RF-3 on st1 is ~$5,420 (~1.4×). Per byte, replicated HDD is dearer than S3's erasure coding — the win is deleting the object store, the ETL, and the query cluster around it, not the storage rate.

What you actually run — line by line

Streaming Lake spends money only where speed is needed: NVMe on the bookie journal and the query-broker spill, cheap Cold HDD (sc1) for the bulk ledger data, and cheap stateless pub-sub brokers. The five-component stack pays for a Kafka cluster, two always-on Spark roles, and object storage on top of each other.

AWS on-demand list prices, US-East-1 (verify current). The competitor side is modeled from list prices and documented defaults; the Streaming Lake side applies the measured read behavior to this hardware. Discounts (reserved / spot, often 30–70%) apply to both. One honest caveat: sc1 is cold HDD — cheapest per byte, but low random IOPS — so keep the fresh/hot working set on faster media (st1 or NVMe + page cache) via bookie affinity groups; cold data served from sc1 answers in seconds, not the ~1 s measured on NVMe.

How it stacks up

A qualitative read of where the selective-join workload lands across four representative systems:

Each system is excellent at what it was built for. ClickHouse in particular remains a best-in-class analytical engine; the distinction here is architectural — Streaming Lake unifies ingestion, storage, indexing, and query on one source-of-truth log.