The Last Architecture Designed by Hand

I bet there is another new architecture to find that is gonna be as big of a gain as transformers were over LSTMs. Sam Altman, the CEO of the company most invested in the transformer is telling a room of students it isn’t the final form. So what comes after the transformer? He’s probably right that something will, and the evidence is no longer anecdotal. Several recent papers have proved that the transformer’s worst properties are structural, not engineering problems to be fixed with better...