CLI
===

django-crawl provides a ``crawl`` management command that iteratively crawls through your site using Django’s test client.
It renders pages within the same process and avoids serializing requests and responses to HTTP, making it somewhat faster and a lot more flexible than regular HTTP crawlers.

``crawl`` follows internal links, follows redirects, and reports every broken page it finds before failing.
It always follows relative links, and it follows absolute ``http(s)`` links when their host is the test client’s host or in ``ALLOWED_HOSTS``  (a ``'*'`` entry is ignored).
Links to another allowed host are requested with that host in the ``Host`` header, so host-based routing works.
It only makes ``GET`` requests.

Asset links, like those in ``<img src>`` and ``<script src>``, are checked too.
If ``django.contrib.staticfiles`` is installed, the command serves static file URLs that your URL configuration or middleware doesn’t otherwise handle, using the staticfiles finders like ``runserver`` does.
This means missing static files are reported as broken links, without needing any static-serving setup.

To get started, run the ``crawl`` management command:

.. code-block:: console

    $ ./manage.py crawl

By default, the command starts at ``/`` and crawls up to 1000 URLs, up to five links deep.
Pass one or more start URLs to crawl specific areas:

.. code-block:: console

    $ ./manage.py crawl /admin/ /accounts/

Use ``--depth`` to control how many links are followed from each start URL:

.. code-block:: console

    $ ./manage.py crawl --depth 2

Use ``--max-urls`` to change the overall URL limit:

.. code-block:: console

    $ ./manage.py crawl --max-urls 500

By default, django-crawl crawls up to 10 query string variants per path.
This avoids it getting stuck in large spaces of sorting and filtering links, such as Django admin changelists.
Use ``--max-query-variants`` to change this limit, or ``unlimited`` to disable it:

.. code-block:: console

    $ ./manage.py crawl --max-query-variants 20
    $ ./manage.py crawl --max-query-variants unlimited

Use ``--exclude`` to skip crawling discovered URLs that match a regular expression, searched against the normalized URL, like ``/reports/?year=2026``.
Repeat the option to exclude several patterns:

.. code-block:: console

    $ ./manage.py crawl --exclude '^/reports/' --exclude 'format=csv'

``--exclude`` applies to discovered links and redirect targets.
Start URLs passed on the command line are always crawled.

Use ``-v 2`` (``--verbosity 2``) or higher to report each URL as it is crawled:

.. code-block:: console

    $ ./manage.py crawl -v 2

The command also extracts URLs from XML sitemaps and sitemap indexes, like those served by ``django.contrib.sitemaps``, and from RSS and Atom feeds, like those served by ``django.contrib.syndication``.
Crawl your sitemap to check that every page it lists renders:

.. code-block:: console

    $ ./manage.py crawl /sitemap.xml

The command follows redirects.
Each response counts towards the ``--max-urls`` limit, including each redirect hop.
It reports HTTP 400, 404, 500, and other 4xx/5xx responses, including Django exception tracebacks when available.
It keeps crawling after errors and exits non-zero after reporting them all.

Authentication
--------------

By default, if ``django.contrib.auth`` is installed, django-crawl logs in as the first active superuser, ordered by the user model’s username field.
Disable this with ``--no-login``:

.. code-block:: console

    $ ./manage.py crawl --no-login

Use ``--login`` to log in as a specific user by username or email address:

.. code-block:: console

    $ ./manage.py crawl --login alice
    $ ./manage.py crawl --login alice@example.com

No password is required, because django-crawl uses |Client.force_login()|__ to log in the user without checking credentials.

.. |Client.force_login()| replace:: ``Client.force_login()``
__ https://docs.djangoproject.com/en/stable/topics/testing/tools/#django.test.Client.force_login

Setup code
----------

Use ``--setup-code`` for small snippets that configure the client before crawling.
Setup code runs after login, so it can inspect or adjust the logged-in session.
For example, to set the ``x-site`` header:

.. code-block:: console

    $ ./manage.py crawl --setup-code 'client.defaults["HTTP_X_SITE"] = "docs"'

For multi-host sites where middleware selects ``request.urlconf`` from the host, set the host header:

.. code-block:: console

    $ ./manage.py crawl --setup-code 'client.defaults["HTTP_HOST"] = "docs.example.com"'

Response checks
---------------

Pass ``-c`` (``--command``) to run Python code for every response.
The response is available as ``response`` in locals, like ``manage.py shell -c``.
For example, to audit all URLs’ ``content-security-policy`` headers:

.. code-block:: console

    $ ./manage.py crawl -c 'print(response.wsgi_request.path, response.headers.get("Content-Security-Policy", ""))'

The code runs in the same namespace as ``--setup-code``, so ``client``, ``settings``, and any state set up before crawling are also available.
The namespace persists between responses, so it can accumulate state.
If the code raises an exception, the command reports the URL and traceback, then continues crawling.