fluffy rambles: Futility

xkeeper posted a blog post about their issues running TCRF.On a related note, here’s my webserver’s HTTP response code stats for today, across all of my websites:fluffy@nautilus[/var/log/nginx] fluffy$ cut -f3 -d\" access.log | cut -f2 -d\ | sort | uniq -c | sort -n 3 408 4 502 4 503 10 410 20 504 56 307 60 403 165 406 225 401 462 499 662 405 670 500 819 202 1739 304 2989 400 6299 206 16352 302 32402 301 52673 404 83035 303 192512 200 308859 429 This is pretty disheartening. For those who don’t know, 429 is Too Many Requests, and is what my site responds with when it detects bad-behavior crawler activity, primarily from attempting to crawl over the tag browser, which is not presented to properly-declared bots (because there is no reason for bots to crawl them).So, of the nearly 700,000 requests served up today, around 44% were these crawler bots. Most of these bots come from residential proxy networks, meaning people have installed malware like Honeygain to earn tiny amounts of money…