A new/old video coding scheme

Now that AV2 has officially been released it’s time to remind that these codec advances are not for free. Here it is claimed that AV2 offers about 25% improvements in compression efficiency compared to AV1 with decoding being five times slower (and that’s for an extremely optimised decoder; no idea how much slower encoding is). Back in the day I proposed to go full AI with the codec design but it was not a fully serious idea, here I’d like to propose something more realistic that would use “AI” hardware that we should have plenty of by now—and with no AVn codecs in mind. The basic idea is that video compression is being improved mostly by trying more and more different approaches on varying blocks of data that depend on previously coded blocks of data. For modern mainline video codecs the structure is simple: divide frame into, say, 128×128 tiles, split those tiles into e.g. 64×64 blocks, decide whether you want intra or inter prediction there, split those blocks further and try some…