Database research, frontier AI, and idiot savants, oh my!

I’m currently working on a pretty deep optimization cycle in how RavenDB and Voron (its storage engine) handle I/O. The idea is to be able to squeeze more performance out of the system by being able to utilize a lot of moving pieces at once (parallelizing compute, durable I/O, async I/O, etc.).At the end of the day, however, it all must reside on disk, of course. That means that I’m essentially seeing how close I can dance to the edge of the capabilities of the system. For example, let’s consider AWS GP3 drives. They offer 3K IOPS and 125MB/sec on the basic plan.We have 50 clients, each generating a request every 15 - 20 ms and writing ~32KB each. This is the kind of load that they’ll generate on the system. Here is what this looks like:Now, let’s add just 3 additional clients to the mix, add just enough to hit the limit (not even grossly exceed it!). Here is what this looks like:Note that in very short order, we moved very quickly from < 2ms to > 250 ms. That is the kind of dangerous…