Trapped Some Web Scrapers

Recently, I found On Bots, an interesting project from 2006-05-03. Someone constructed 2 billion numbered web pages, arranged as a binary search tree via hyperlinks. They tracked accesses of the web pages for a year. It looks like they were interested in how much of their binary trees got explored, and in what order more than anything else.