📨 Kafka Partitions to Broker RAM
Each Kafka partition consumes ~1-2 MB broker RAM before a single message is stored — leader state, replica fetchers, log segment indices. 10,000 partitions = 2-4 GB RAM overhead alone, before page cache. That 8 GB heap broker is half-full with metadata. Model the partition tax against your heap budget before the GC logs light up.
📐 Kafka Cluster Profile
Configure partition count, replication factor, avg message size, and throughput. The engine computes leader overhead (~0.25 MB/partition), replica overhead (~0.20 MB/partition), plus page cache. Total heap + OS cache = your broker sizing target.
Kafka Partition Overhead: ~0.25 MB RAM per Partition
Apache Kafka's performance is fundamentally constrained by per-partition memory overhead within the broker's JVM heap. Each partition — whether acting as a leader or replica — consumes heap memory for index structures, log segment metadata, producer state tracking, and fetch session caches. As a rule of thumb, plan for approximately 0.25 MB of JVM heap per leader partition and approximately 0.20 MB per replica partition. A broker hosting 1,200 leader partitions and 2,400 replica partitions (RF=3) requires approximately 780 MB of heap overhead before accounting for any message data or network buffers.
How This Calculator Works
| Component | Formula | Derivation |
|---|---|---|
| Leader Overhead | Partitions × 0.25 MB | ~0.25 MB per leader for index files (time + offset), producer ID snapshots, fetch session cache, and segment metadata |
| Replica Overhead | Partitions × (RF − 1) × 0.20 MB | ~0.20 MB per replica for replication fetch buffers, log-end-offset tracking, and ISR bookkeeping |
| Total JVM Heap | Leader + Replica Overhead | Sum of leader and replica memory. Does not include OS page cache — that is off-heap. |
| Page Cache | Partitions × AvgMsgKB × 2 | Conservative estimate of OS page cache for hot data (×2 multiplier for read/write path buffering) |
Leader vs. Replica Memory Profiles
Leader partitions carry more heap overhead than replicas because they maintain additional data structures: producer ID snapshots for exactly-once semantics (EOS), fetch session caches per consumer group, and active log segment indices. Replicas maintain a lighter footprint — primarily the offset index and a fetch buffer for each active replication stream. The asymmetry means that partition leadership rebalancing (controlled by kafka-reassign-partitions.sh or Cruise Control) changes each broker's memory profile, not just its I/O and CPU load. A broker that suddenly becomes leader for 500 additional partitions can see its heap usage spike by 100–125 MB within seconds.
32 GB RAM = ~4,000 Partitions (Practical Limit)
Given ~0.25 MB per leader partition and assuming a conservative 6 GB JVM heap allocation (the remaining RAM dedicated to OS page cache), a single broker can practically host approximately 4,000 leader partitions before heap pressure becomes a concern. But this is a ceiling, not a target. Partitions beyond 4,000 per broker increase GC pause frequency (Kafka uses G1GC by default), extend controlled shutdown time, and amplify the blast radius of a broker failure. Confluent and LinkedIn production guidance recommends an upper bound of 4,000–6,000 partitions per broker and 200,000 partitions per cluster. Exceeding these limits requires careful tuning of num.recovery.threads.per.data.dir, num.replica.fetchers, and the G1GC MaxGCPauseMillis parameter.