LLM-judged policy debates

In many policy areas, the usual convention wisdom on policy is actually in substantial conflict with many of the specific details that experts know about that policy area. However, experts who point this out are usually accused of being politically biased. Which is why we stay stuck in this situation.However, while LLMs usually offer the usual conventional wisdom when asked simply about policy, they also know about and understand most of those details which are in conflict with it. Furthermore, at least for now LLMs are much less likely to be accused of political bias for reacting to such conflicts. Yes, they don’t notice such conflicts unless someone points them out, and even then they don’t usually update their main models as a result of noticing conflicts.Even so, this seems to create an opportunity for LLM-judged policy debates. Let an analysis be a <2000? word text citing expert sources/results, and let a position be a <100? word summary, perhaps including a 0-100 numerical…