Kafka Share Groups and Parallelizing Consumption - Part 2: Producer Batches and share.acquire.mode

All tests were executed against Kafka 4.3.0 using Dimster. In the last post we used simulated consumer processing time to reveal how important it is to set an appropriate value for max.poll.records to ensure the consumer parallelism that we expect. With a uniform distribution of messages over partitions, the rule of thumb was a value somewhat lower than: group.share.partition.max.record.locks / number of consumers per partition But there’s more to parallel consumption than max.poll.records. The size of producer batches also plays a role when using the default share.acquire.mode (batch_optimized).Some background on share consumer assignment and share.acquire.modeShare group members are assigned to partitions like consumer group members are, except that share group assignment allows multiple consumers to be assigned to the same partition.Share group assignmentsIf the number of share consumers is less than the partition count, then each consumer will be assigned multiple partitions. If…