Django: introducing django-crawl

I recently migrated one of my client projects from the legacy django-csp package to Django 6.0’s built-in Content Security Policy (CSP) support (release note). This security header is a powerful tool for preventing unwanted content from being loaded on your site, so configuration correctness is paramount. The migration was fairly straightforward, but a few pages had complicated overrides, so I wanted to be sure that no CSP headers had been changed by my swapping of CSP implementations. I had the idea to verify no page had changed its content-security-policy header by crawling the site with Django’s test client, outputting URL and header contents during the process. By diffing the output from crawls before and after the migration, I could check for changes and track down which pages had been affected. The core loop of that script looked something like this: from collections import deque from django.test import Client client = Client() client.force_login(superuser) queue: deque[str] =…