Kafka Share Groups - Pathological fetch waits with record_limit

In this post we’re going to see how share.acquire.mode=record_limit combined with:fewer consumers than partitionsand various cases of “partition skew”…can result in subpar performance with share groups. I stumbled on these issues when running large sets of dimensional tests with Dimster’s explore-limits mode, which finds the highest sustainable throughput while staying within a target end-to-end latency target. There was a specific subset of the tests that explore-limits mode would consistently fail to complete, and they all happened to be with record_limit and a consumer count lower than the partition count. In this test, we’ll understand why Dimster had such a hard time with this combination.Some background on share group internalsKafka share groups have two methods of acquiring records:share.acquire.mode=batch_optimizedshare.acquire.mode=record_limitI already explained the difference in Kafka Share Groups and Parallelizing Consumption - Part 2: Producer Batches and…