How to send a real 404 to agents when a page doesn't exist

Every documentation site has an AI agent and a human reader nowadays, where AI agents are dominating the number of requests sent to the server to fetch a docs page. Agents fetch pages with HTTP clients and either use tools like web fetch to read the context of HTML or if provided, the markdown content of that page. When a URL is missing or is referenced in a wrong way, these two readers need different responses. For humans, you would use a 404 page with a link to homepage or a better and a traditional way to handle is to provide redirects on the client-side. Redirects can also be provided on server-side to correct the page. In both cases, a redirect dictionary is usually maintained by authors. However, an agent or a human can equally run into 404 links. In case of an agent, it cannot know a page exists and may end up reading the wrong page. This post lists down some suggestions that I have recently come across and implemented in the Expo docs, which I work on in my day job. Serve…