On Humphreys opacity, Reverse Engineering, and Social Externalities of LLMs.

I start with characterizing a term, ‘Humphreys opacity’ (or, if you prefer, ‘epistemic opacity’):1 this involves the inability to surveil the steps of a process from a known input to a known desirable (or truthful, useful, beautiful, etc.) output in a timely manner to the decision-maker or responsible agent. (For more on the origin and nature of this characterization, recall this post.) In what follows, I set aside to what extent such Humphreys opacity is the effect of features of physical reality or is merely the result of a pragmatic cost-benefit analysis. Humphreys opacity is in the news because the ideal to generate a so-called ‘glassbox’ AI — in which AI systems and machine learning models where the internal processes are fully visible, transparent, and interpretable to humans — seems so hard to achieve. In fact, Humphreys opacity is a design feature of contemporary LLMs that are rapidly being deployed in all kinds of organizations. At the moment neither end-users nor engineers…