Can the Robot Exclusion Protocol apply to Gopher URLs? 0 ▲ The Boston Diaries 1 hour ago · Tech · hide · 0 comments A month ago, slugmax wrote a post about LLMs invading gopher via an MCP server and now that MCP server supports the /robots.txt standard. He gives a sample file that should work for Gopher: User-agent: gopher-mcp Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: Claude-User Disallow: / User-agent: Google-Extended Disallow: / There are problems with this advice. First off, Gopher doesn't support additional information per a request, unlike HTTP. There is no way a Gopher client can identify itself to a Gopher server. Well, there is a hack one could use: include a search query when retrieving the /robots.txt file, but the current Gopher standard says little about how a server responds to a search query on a non-search-query selector. I mean, it's probably okay as most servers will (hopefully?) ignore the… No comments yet. Log in to reply on the Fediverse. Comments will appear here.