Predicting and designing proteins (4): What protein language models learn from sequences

Calling proteins a language is useful only if the analogy is handled carefully. Amino acids are not words with dictionary meanings, and evolution does not compose sentences with an intention. A protein sequence is a physical polymer that must be synthesized, folded, and selected in an organism. Yet sequences do have context-dependent regularities. Some residue combinations are common, some are incompatible, and a residue that is acceptable in one region may be destructive in another. A model trained to predict missing residues can learn those regularities without being given a separate label for every structure or function. A protein language model (PLM) is trained on amino-acid sequences using objectives adapted from natural-language processing. In masked modelling, some residues are hidden and the model predicts them from their context. In autoregressive modelling, the model predicts the next residue from the preceding sequence. Training at large scale produces internal numerical…