CLI

django-crawl provides a crawl management command that iteratively crawls through your site using Django’s test client. It renders pages within the same process and avoids serializing requests and responses to HTTP, making it somewhat faster and a lot more flexible than regular HTTP crawlers.

crawl follows internal links, follows redirects, and reports every broken page it finds before failing. It always follows relative links, and it follows absolute http(s) links when their host is the test client’s host or in ALLOWED_HOSTS (a '*' entry is ignored). Links to another allowed host are requested with that host in the Host header, so host-based routing works. It only makes GET requests.

Asset links, like those in <img src> and <script src>, are checked too. If django.contrib.staticfiles is installed, the command serves static file URLs that your URL configuration or middleware doesn’t otherwise handle, using the staticfiles finders like runserver does. This means missing static files are reported as broken links, without needing any static-serving setup.

To get started, run the crawl management command:

$ ./manage.py crawl

By default, the command starts at / and crawls up to 1000 URLs, up to five links deep. Pass one or more start URLs to crawl specific areas:

$ ./manage.py crawl /admin/ /accounts/

Use --depth to control how many links are followed from each start URL:

$ ./manage.py crawl --depth 2

Use --max-urls to change the overall URL limit:

$ ./manage.py crawl --max-urls 500

By default, django-crawl crawls up to 10 query string variants per path. This avoids it getting stuck in large spaces of sorting and filtering links, such as Django admin changelists. Use --max-query-variants to change this limit, or unlimited to disable it:

$ ./manage.py crawl --max-query-variants 20
$ ./manage.py crawl --max-query-variants unlimited

Use --exclude to skip crawling discovered URLs that match a regular expression, searched against the normalized URL, like /reports/?year=2026. Repeat the option to exclude several patterns:

$ ./manage.py crawl --exclude '^/reports/' --exclude 'format=csv'

--exclude applies to discovered links and redirect targets. Start URLs passed on the command line are always crawled.

Use -v 2 (--verbosity 2) or higher to report each URL as it is crawled:

$ ./manage.py crawl -v 2

The command also extracts URLs from XML sitemaps and sitemap indexes, like those served by django.contrib.sitemaps, and from RSS and Atom feeds, like those served by django.contrib.syndication. Crawl your sitemap to check that every page it lists renders:

$ ./manage.py crawl /sitemap.xml

The command follows redirects. Each response counts towards the --max-urls limit, including each redirect hop. It reports HTTP 400, 404, 500, and other 4xx/5xx responses, including Django exception tracebacks when available. It keeps crawling after errors and exits non-zero after reporting them all.

Authentication

By default, if django.contrib.auth is installed, django-crawl logs in as the first active superuser, ordered by the user model’s username field. Disable this with --no-login:

$ ./manage.py crawl --no-login

Use --login to log in as a specific user by username or email address:

$ ./manage.py crawl --login alice
$ ./manage.py crawl --login alice@example.com

No password is required, because django-crawl uses Client.force_login() to log in the user without checking credentials.

Setup code

Use --setup-code for small snippets that configure the client before crawling. Setup code runs after login, so it can inspect or adjust the logged-in session. For example, to set the x-site header:

$ ./manage.py crawl --setup-code 'client.defaults["HTTP_X_SITE"] = "docs"'

For multi-host sites where middleware selects request.urlconf from the host, set the host header:

$ ./manage.py crawl --setup-code 'client.defaults["HTTP_HOST"] = "docs.example.com"'

Response checks

Pass -c (--command) to run Python code for every response. The response is available as response in locals, like manage.py shell -c. For example, to audit all URLs’ content-security-policy headers:

$ ./manage.py crawl -c 'print(response.wsgi_request.path, response.headers.get("Content-Security-Policy", ""))'

The code runs in the same namespace as --setup-code, so client, settings, and any state set up before crawling are also available. The namespace persists between responses, so it can accumulate state. If the code raises an exception, the command reports the URL and traceback, then continues crawling.