Skip to content

fix(django): don't force ROOT_URLCONF import during django.setup() - #19467

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
christophe-papazian/django-lazy-endpoint-discovery
Aug 5, 2026
Merged

fix(django): don't force ROOT_URLCONF import during django.setup()#19467
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
mainfrom
christophe-papazian/django-lazy-endpoint-discovery

Conversation

@christophe-papazian

@christophe-papazian christophe-papazian commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

APPSEC-69515

Description

Alternative to #19465 for the same regression (#19454). Endpoint discovery walked the root resolver at the end of
traced_populate, so django.setup() imported ROOT_URLCONF and every view module behind it — 154 MB of RSS per
dramatiq worker that serves no requests. The eager call arrived in #17695, so every release from 4.9.0rc1 is affected.

This moves the walk to traced_load_middleware: building a BaseHandler is the earliest reliable signal that a
process will serve HTTP, so workers pay nothing while gunicorn/uwsgi/daphne still report endpoints at startup.
BaseHandler.load_middleware is now wrapped unconditionally so discovery no longer depends on
DD_DJANGO_INSTRUMENT_MIDDLEWARE.

#19465 instead guards the walk on ROOT_URLCONF already being in sys.modules, but Django imports the URLconf
neither during django.setup() nor during WSGIHandler.__init__ — only at the first resolve(). Measured with
get_wsgi_application(), which is what gunicorn imports:

urlconf imported endpoints at startup
main yes 32
#19465 no 0
this PR yes 32

Also adds '@django' and '@flask' to the telemetry suite in suitespec, which runs real apps of both but was
gated only on shared contrib infrastructure — which is why #19465's CI is green while it deterministically fails
test_endpoint_discovery_event.

Testing

test_setup_does_not_import_root_urlconf and test_endpoint_discovery_skipped_without_http_handler fail on main
and pass here. test_endpoint_discovery_event now builds the WSGI application and still passes (it fails on #19465).
test_wsgi_application_collects_endpoints pins #17695's include/test/ prefix joining at startup, and a paired test
pins that discovery survives DD_DJANGO_INSTRUMENT_MIDDLEWARE=false while middleware wrapping does not.

On a bare django.setup() (py3.13 / django 5.1): sys.modules 1243 → 1220, live GC objects 109,218 → 106,123, max
RSS 82,884 → 81,496 KB. Small here only because the test app's view closure is small — the delta is that closure.

contrib::django passes except 5 test_django_appsec_snapshots.py failures that reproduce identically on main.

Risks

Serving processes import the URLconf when the handler is built rather than on the first request — earlier than Django
would, but later and safer than the previous Apps.populate timing. Processes that build no handler no longer report
endpoints, which is the point. Management commands are unaffected either way: Django's own check_url_config imports
the URLconf whenever system checks run.

Additional Notes

Pre-existing and out of scope: _collect_routes_once marks a resolver collected in its finally block even when the
walk raised; the wsgi, appsec_integrations_django and ddtracerun suites have the same suitespec gap. ASGI
collection is reasoned but untested.

🤖 Generated with Claude Code

Credit to @bellini666 for diagnosing the regression and for test_setup_does_not_import_root_urlconf, carried over
from #19465.

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codeowners resolved as

ddtrace/contrib/internal/django/patch.py                                @DataDog/apm-core-python @DataDog/apm-idm-python
releasenotes/notes/fix-django-eager-urlconf-import-aa3eff2dd008ec0f.yaml  @DataDog/apm-python
tests/contrib/django/test_django_patch.py                               @DataDog/apm-core-python @DataDog/apm-idm-python
tests/suitespec.yml                                                     @DataDog/python-guild @DataDog/apm-core-python
tests/telemetry/test_writer.py                                          @DataDog/apm-python

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 3 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace.trace
ddtrace -> ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace
ddtrace -> ddtrace.trace -> ddtrace._trace.tracer -> ddtrace.internal.debug -> ddtrace.internal.runtime.runtime_metrics -> ddtrace

@pr-commenter

pr-commenter Bot commented Aug 3, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-08-04 13:35:09

Comparing candidate commit ffbd66d in PR branch christophe-papazian/django-lazy-endpoint-discovery with baseline commit 11c2498 in branch main.

Found 0 performance improvements and 6 performance regressions! Performance is the same for 598 metrics, 10 unstable metrics.

scenario:httppropagationinject-ids_only

  • 🟥 execution_time [+1.489µs; +1.622µs] or [+7.047%; +7.677%]

scenario:iastaspects-upper_aspect

  • 🟥 execution_time [+58.393µs; +65.217µs] or [+19.440%; +21.712%]

scenario:iastaspectsospath-ospathbasename_aspect

  • 🟥 execution_time [+99.922µs; +105.592µs] or [+24.752%; +26.156%]

scenario:span-start

  • 🟥 execution_time [+1.484ms; +1.663ms] or [+9.339%; +10.466%]

scenario:telemetryaddmetric-1-count-metric-1-times

  • 🟥 execution_time [+206.452ns; +240.601ns] or [+9.837%; +11.464%]

scenario:tracer-small

  • 🟥 execution_time [+31.814µs; +33.882µs] or [+9.489%; +10.106%]

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ffbd66d | Docs | Datadog PR Page | Give us feedback!

christophe-papazian and others added 3 commits August 4, 2026 15:06
Endpoint discovery walked the root resolver at the end of traced_populate, and
reading resolver.url_patterns imports ROOT_URLCONF along with every view module
behind it. Processes that never serve a request loaded that entire graph for
nothing, costing one reporter 154MB of RSS per dramatiq worker.

Move the walk to traced_load_middleware. Building a BaseHandler is the earliest
reliable signal that a process will serve HTTP, so workers, management commands
and cron jobs never import the URLconf, while gunicorn, uwsgi, daphne and
runserver still report endpoints at startup rather than only after traffic
lands. Wrap BaseHandler.load_middleware unconditionally so endpoint discovery
no longer depends on DD_DJANGO_INSTRUMENT_MIDDLEWARE.

Also add '@django' and '@Flask' to the telemetry suite in suitespec, which runs
real apps of both but was only gated on shared contrib infrastructure.

Fixes #19454

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Scope the claims to what was measured. Django's own check_url_config imports
the root URLconf whenever system checks run, so most manage.py invocations
import it with or without ddtrace; the beneficiaries are processes that call
django.setup() without running checks. The release note and the two test
docstrings said "management commands and cron jobs", which was wrong for the
common case, and the telemetry negative test used the help path only because it
is one of the few command paths that skips checks -- it now bootstraps with a
bare django.setup(), which is what a Celery or dramatiq worker actually does.

Assert middleware wrapping state in both handler tests. Moving the wrapping
under a runtime check meant the disabled-flag test would have passed even if
wrapping had accidentally stayed on, and nothing pinned that wrapping still
happens when the flag is at its default.

Correct _collect_routes_once's docstring: the walk happens once per distinct
resolver, not once per BaseHandler.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…note

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@christophe-papazian
christophe-papazian force-pushed the christophe-papazian/django-lazy-endpoint-discovery branch from f0b229c to ffbd66d Compare August 4, 2026 13:06
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 238e994 into main Aug 5, 2026
1272 of 1273 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the christophe-papazian/django-lazy-endpoint-discovery branch August 5, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants