AI Vs. Human Value Drift

I don’t see why people concerned with value drift should be especially focused on AI. … Your real problem is that you want long term stability in a universe that more naturally changes. (Me in 2018)The last week has seen an explosion of talk on the scenario of AIs killing all humans; many say “P(doom)” is high, and argue for slowing and regulating AI. My last post said why AI regs look risky. Here I consider the main argument for doom, and compare it to the issue I’ve focused on for the last three years: human value drift.The core doomer idea is that AIs will plausibly dominate the world eventually, at which point humans will be at AI’s collective mercy. And so it matters what AIs then want. The worry isn’t so much that some AIs would want to kill all humans; others who don’t want that could overrule them. The worry is that mid AI values, i.e., the middle of the distribution of AI values weighted by AI power, might want to kill all humans.Mid AI values today are clearly far from that…