Why do OpenAI's GPT-2 weights beat mine? Part two: IFT dropout

Wait, dropout! https://arxiv.org/pdf/2101.03961 -- p11 -- Regularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks. Could that be it?! Model conf has dropout in it, so... Drop rate was zero in the GPT-2 weights JSON files I had! Nothing in the paper but Raschka and also https://huggingface.co/docs/transformers/v4.48.2/en/model_doc/gpt2#transformers.GPT2Config When I did the extended/two-epoch runs, I wound up doing many more…