A Stupid Idea for AI Alignment We Came up with by Looking at the List of Specification Gaming Behaviours

Creatures bred for speed grow really tall and generate high velocities by falling over. An evolved player makes invalid moves far away in the board, causing opponent players to run out of memory and crash. A game-playing agent accrues points by falsely inserting its name as the author of high-value items. These bizarre exploits and dozens more can be found in the list of specification gaming behaviours [sic; British], a document put together by DeepMind Safety Research. “A reinforcement learning agent can find a shortcut to getting lots of reward,” they explain, “without completing the task as intended by the human designer. These behaviours are common.” Specification gaming is when an agent, like an AI, tries to succeed on a task by following the letter of the law rather than the spirit. In other words, it looks for loopholes, it tries to get off on a technicality. Even very simple AI can come up with very creative ways of solving their assigned problems. This is a problem. “…a…