Apache Kafka performance #1 - linger.ms

This is the first in an ongoing ad-hoc series of posts on Apache Kafka performance. I have no general direction, I’ll just share interesting insights based on the performance testing I do on Apache Kafka.Recently I was curious to see if there was any general performance improvement since Kafka 3.X. So I ran a suite of benchmarks with Dimster against 3.7.2 and 4.3.0. I saw two common patterns:Pattern 1: Low load benchmarks showed that end-to-end latency was higher with Kafka 4.3 compared to 3.7.2. The following is a 45 minute no-record-key workload of 5000 record/s, 20 topics (120 partitions), fan-out 2 (240 consumers), full TLS, on 3 brokers each allocated 8 SMT CPUs in k8s (on my Threadripper 9980X). Fig 1. Low load: end-to-end latency over time (p99 over 10 second intervals) Pattern 2: On more stressful loads, 3.7.2 would show much more spiky end-to-end latency compared to 4.3. The following is for the same workload at 100K records/s (200K out). Fig 2. High load: end-to-end latency…