Search gate traffic – how a cookie helped performance

On 17 October, at 10:17 UTC, the search gate went live on www.freshports.org. A search from a visitor without a login or a search_ok cookie now gets a small challenge page instead of a search. I wrote about the first day in the FreshPorts October speed-up post. This post looks at the twelve days since. I expected the traffic to drop once the bots got nothing for their trouble. It didn’t. What dropped was the cost of that traffic. What the search gate is The gate is described in detail in Deterring bots, but letting people in. In short: A search runs only if you are logged in or have a search_ok cookie. Without either, you get a page with one button. Press it, and you get the cookie and go back to your search. The cookie lasts a day. Logged-in users never see the page. The cookie, and the token behind the button, are signed with a secret only the server knows. The button stops working after an hour. The Referer header no longer counts. The botnet faked it, which is how it got past the…