Lessons Learned from Fixing Flaky Tests with Claude

We recently sped up our integration tests by a factor of ten by running many of them in parallel. There were several flaky tests in the new test suite. Claude was really useful in getting rid of almost all flakyness, but I made some mistakes along the way. Here is what I learnt. Our system is very well covered by integration tests. We have a complete local system running, and all integration tests interact with the running system via FIX connections (traffic) and HTTP connections (administration). Previously, our 1360 integration tests ran sequentially, and took around 40 minutes to complete. A recent rewrite got the running time down to around 4 minutes instead. The gain comes from being able to run many tests in parallel. To be able to run tests fast in parallel, the test suite relies on instruments that are defined once, then reused for many tests. This however makes the tests vulnerable to interference between tests. After the rewrite, there were typically a few tests each run…