Messaging and streaming, in one platform

Publish/subscribe messaging is asynchronous communication between services, common in serverless and microservices architectures: messages sent to a topic are delivered to all subscribers, decoupling applications and improving performance, reliability, and scalability. Stream processing handles that data continuously and in real time — and as businesses rely on timely insight, the challenge is not just volume but processing data fast enough to react instantly.

Pulsar and Kafka are both strong alternatives to traditional brokers, offering higher throughput, built-in partitioning, replication, and fault tolerance. But they differ in message modeling and capabilities — which is exactly what matters when picking the right system for a use case.

Horizontally scalable by design

Apache Pulsar achieves horizontal scalability through Apache BookKeeper's segment-oriented architecture, which distributes topic data across many bookies (BookKeeper storage nodes). This makes Pulsar fundamentally different from Kafka.

Why Pulsar scales

• Segmented storage model — unlike Kafka, where a topic's partitions are tied to a specific broker, Pulsar splits topic data into segments distributed across multiple bookies, enabling independent scaling of storage and processing.
• Decoupled brokers & storage — Pulsar brokers are stateless routers handling message distribution, while BookKeeper manages storage. Kafka brokers do both, which creates a bottleneck.
• Dynamic load distribution — new bookies can be added at any time, and Pulsar automatically redistributes topic segments across them.
• Parallel reads & writes — multiple bookies handle different segments of a topic; Kafka restricts a partition to a single broker.
• Fault tolerance & replication — segment-based replication ensures redundancy; if a bookie fails, Pulsar redistributes segments while maintaining availability.

Why Kafka can't scale the same way

• Topic–broker binding — a topic's partitions are bound to a single broker, so data can't be distributed dynamically across brokers.
• Broker bottleneck — adding brokers doesn't increase a partition's throughput; the only way to scale is adding partitions, which brings rebalancing overhead.
• Limited storage scalability — Kafka brokers store topic data, so capacity is bounded by a broker's disk, whereas Pulsar offloads storage to BookKeeper.

Geo-replication

Geo-replication replicates persistently stored messages across clusters — often across data centers — for disaster recovery and distribution. Kafka offers MirrorMaker, a standalone tool the operator must install and manage. Pulsar provides built-in geo-replication, reliably and asynchronously replicating across clusters with no extra tooling.

Streaming and queuing

Message consumption falls into two patterns, and Pulsar supports both natively.

Streaming

Used where messages must be consumed in the order they were written. Both Kafka and Pulsar guarantee ordering in their streaming models — common in multi-stage data pipelines where raw input is aggregated, enriched, and written onward.

Queuing

Queuing processes messages without a strict order, in parallel across workers, with individual acknowledgement. Kafka only supports exclusive consumers — each partition is processed by one consumer in a group, so parallelism scales only by repartitioning. Pulsar supports shared consumers: scale throughput by adding consumers without repartitioning, and acknowledge individual messages. This makes Pulsar far more flexible for parallel consumption.

Storage, retention, and committing

Kafka uses time-based retention — messages are deleted after a configured period whether or not they were consumed. Pulsar retains messages until acknowledged by all subscribers, with optional explicit expiry or extended retention per topic.

For acknowledgement, Kafka uses high-watermark commits: a consumer can only say "I've processed up to here", forcing extra bookkeeping logic. Pulsar natively supports individual message acknowledgement and keeps unacknowledged messages until they're explicitly committed — with configurable ack timeouts for automatic redelivery.