SwissDelphiCenter Delphi Tips: The Scraper Scripts

Here are the two Python scripts I used to create the SwissDelphiCenter Delphi Tips archive: download. scrape_tips.py downloads the tip pages from the Wayback Machine. It waits 10 seconds between requests and it is resumable, so you can stop it and continue later. Before you run it, you must put your email address into the CONTACT constant near the top, because it goes into the User-Agent of every request so that the Internet Archive knows who is calling. The script refuses to start without it. convert_tips.py turns those downloaded pages into the collection that the GExperts Code Librarian can import, in the format I described in the previous post. They are not exactly general purpose tools, they know a lot about how SwissDelphiCenter built its pages. But if you ever want to rescue some other site from the Internet Archive, the interesting part might be that the scraper does not follow the links on the site. It asks the Wayback CDX index which pages exist instead, because the category…