I am dreading our LLM-written incident report future

Lorin Hochstein writes about generative AI in the context of incident reports, but the points are more broadly applicable. I have seen a big wave of “don’t let AI do your thinking for you” posts recently[^1], so I think lots of folks are pulling back a little bit on the “just let AI do everything” rhetoric (a good thing in my opinion!). As to why Lorin isn’t a fan: In my view, LLM-generated incident write-ups are more dangerous than using LLM for coding or for AI SRE style tasks. For coding tasks, there’s always a testing step to check that the code exhibits the desired behavior, even if nobody looks at the code itself for meaningful details. For AI SRE tasks, either the LLM output helps you resolve the incident, or it doesn’t. In both cases, Nature is the ultimate arbiter of the LLM output. But incident write-ups aren’t like that. The consequences of a poor report aren’t immediately apparent the way incorrect code or an incorrect operational diagnosis are in the moment. Instead, we…