Speeding up Elixir test suites

Last week I took our Elixir/Phoenix suite (13,000 tests, 940 files) from 313 seconds to 64 seconds. Below is what was done with most impactful listed first. These are Elixir specific but I think the learnings apply to non-Elixir stacks. Your mileage will vary. Make your tests async. This sounds obvious but when you encounter flaky tests the first thing you do is make them synchronous which usually "fixes" things, and over time these pile up. Every other point here is essentially in service of this point. We had 134 of 940 files running async: true, which means an 18-core machine was running one test module at a time. ExUnit runs async modules concurrently up to max_cases (default: 2x your cores) and the Ecto sandbox gives each test its own transaction, so DB tests parallelize fine. After the flip we had ~830 async files and the wall time floor became "the slowest single module" instead of "the sum of everything". The catch is that async doesn't fail loudly when it's unsafe, it fails…