Size Your Message Queue Before It Becomes an Outage
The most common mistake when adopting a message queue is sizing it for average traffic alone. In production, traffic spikes, consumer outages, and deploy-time pauses happen repeatedly, and every one of them causes messages to pile up in the queue. The storage you actually need is the product of peak messages per second, average message size, and how long you must retain messages — and both a longer retention window and a higher peak rate grow that number fast.
Consumer count matters just as much. Dividing your peak incoming rate by what a single consumer can process gives you the minimum number of consumers required. If your partition (or shard) count is lower than that number, adding more consumers won't help, because there's nothing left to parallelize onto. That's why partition count should be set comfortably above the minimum consumer count — typically at least 20% higher.
This calculator combines those factors into required storage, minimum consumers, and a recommended partition count in one pass. For a fully accurate sizing, also factor in message compression, replication factor, and broker overhead before finalizing your design.
Frequently Asked Questions
Consumer outages or processing delays cause messages to pile up, so peak traffic and failure scenarios should be factored into your capacity.
Partition count should be at least your minimum required consumer count, with extra headroom for future traffic growth.
If consumer throughput can't keep up with the incoming rate, messages keep piling up, increasing latency and eventually exceeding storage capacity.