Kafka Share Groups and Parallelizing Consumption — Part 1: Tuning max.poll.records

All tests were executed against Kafka 4.2.0 using Dimster. In the last post we measured the overhead that the mechanics of share groups adds, and saw that it is pretty small. Likewise we saw that raw throughput was also comparable to consumer groups and even saw it exceed consumer group throughput on one test.In this post we’re going to simulate processing time in the consumers to make these benchmarks more realistic and show the utility of share groups (namely the ability to parallelize processing beyond the partition count).We’ll see how the following two configurations play an important role in parallelizing consumption with share groups:max.poll.records (consumer config)group.share.partition.max.record.locks (broker-side config)Theoretical max throughputIf we know the average processing time and the number of consumers, we can calculate the theoretical max throughput of a topic: Consume throughput = per-consumer-rate * number of consumers where: Per-consumer-rate = 1000 /…