⚡ speed up migrate_endpoints_to_locations (~14× fewer queries)

[Maffooch]

This is a perf bugfix to an existing one-shot data migration; no new features or surface area, no new dependencies, single file changed. The migration is currently producing latently-incorrect data in addition to being unusably slow on real instances (16+ hours for ~650 endpoints / 22k findings — see Description), so I went ahead.

Description

dojo/management/commands/migrate_endpoints_to_locations.py is a one-shot migration that maps each Endpoint → URL/Location, each Endpoint_Status → LocationFindingReference, and each (endpoint→product, finding→product) pair → LocationProductReference. On a real instance with 650 endpoints and 22k findings, it had produced 80 locations in 2 hours (~90 s/endpoint) and was projected at ~16 hours total. Per-endpoint queries on a 50-endpoint / 1,000-finding local benchmark came in at 613.

This PR keeps changes inside the management command itself (no edits to shared Location model methods) and reduces per-endpoint queries to 44 and per-endpoint wall-clock to 17 ms on the same benchmark — a 13.9× query reduction / 14.1× wall-clock improvement. Idempotent re-runs validated; strict accuracy checks pass.

The work was applied as five logically-distinct changes; each row of the table below was measured independently against a freshly-reset migration target so the gain attribution is honest:

Step	Wall-clock	ms/ep	Queries/ep	Notes
Baseline (instrumented, no logic change)	12.00 s	240	613	14 product-ref status warnings (pre-existing bug)
+ select_related/prefetch_related	10.63 s	213	528	Eliminates lazy joins on tags, endpoint_meta, status_endpoint, and finding → test → engagement → product/mitigated_by
+ tags.add(*names) splat	10.08 s	202	507	tags phase −61%
+ DojoMeta.bulk_create(ignore_conflicts=True)	10.24 s	205	498	meta phase −67% — DojoMeta.unique_together = (location, name) makes ignore_conflicts equivalent to the previous get_or_create
+ bulk_create LFR/LPR + in-memory product status	0.85 s	17	44	Bypasses BaseModel.save → full_clean() validate_unique queries AND the inherit_tags_on_linked_instance post_save signal that fires all_related_products (a finding/test/engagement/product join) on every save

Where the savings actually came from, surfaced by the --query-count flag on a single endpoint with 20 statuses:

• ~3 validate_unique queries × 2 saves per status = ~120 redundant queries / endpoint just from full_clean
• 1 all_related_products + 1 system_settings per save × ~40 saves per endpoint = ~80 redundant queries from the inherit-tags post_save signal
• 3 lazy joins (finding.test.engagement.product) × 20 statuses = 60 queries
• The .exists() → .get() → update_or_create triple in associate_with_finding/associate_with_product — 2 redundant queries per status

bulk_create(ignore_conflicts=True) for the reference rows skips all of those at once.

One Django gotcha worth flagging in review

LocationFindingReference.created is auto_now_add=True. The original migration set created to the source Endpoint_Status.date via a post-save UPDATE, which works because auto_now_add only fires on the initial INSERT (add=True). With bulk_create we don't get a post-save UPDATE — and Django's SQLInsertCompiler.pre_save_val does call Field.pre_save(add=True) even from bulk_create, so auto_now_add would overwrite our explicit value with now(). The PR uses a small _suspend_auto_now_add context manager that toggles auto_now_add=False on the field during the bulk write, then restores it. Single-process management command — no thread-safety concerns.

Behavioral diffs you should know about

1. LocationProductReference.status is now correct (semantic improvement). The previous code called Location.associate_with_product(...), which short-circuits if the (location, product) pair already exists — never updating the status when a later Active finding comes in for the same product. With the seed's 70/30 Active/non-Active distribution, ~28% of product refs landed on Mitigated when the logical truth was Active. The new code derives product status in-memory across all of an endpoint's finding statuses (Active iff any of them is Active, else Mitigated) and bulk_creates the row. The strict mode of the local accuracy verifier explicitly catches this — baseline fails strict, this PR passes strict.

2. Inherited product tags via post_save signals do NOT fire. bulk_create skips signals, so inherit_tags_on_linked_instance (dojo/tags_signals.py:65-68) does not run during the migration. For deployments where any product has enable_product_tag_inheritance=True (or the system-wide setting is on), inherited product tags will not be propagated onto migrated Locations by this command. Two reasonable mitigations, both deferred to a follow-up:

   • Add a one-time post-bulk pass that walks the migrated Locations and calls inherit_tags() (cheap — one query per location, paid once instead of per-LFR).
   • Use the existing _bulk_inherit_tags helper at dojo/importers/location_manager.py:486, which is designed for exactly this scenario.

   I'd be happy to land either as a follow-up PR if you confirm your prod has tag inheritance enabled. The benchmark seed does not exercise this path.

Things intentionally NOT changed

• Location.associate_with_finding / associate_with_product are untouched. They still have the .exists() → .get() → update_or_create pattern and the first-write-wins behavior on product status. The migration just stops calling them on its hot path. Other callers (e.g., importers) are unaffected.
• No DB migration. Pure code change, no schema impact.
• No --start-from <id> / --skip-validation flags. Considered during planning, dropped to keep blast radius small. Easy follow-ups if the prod run reveals a need.
• No verbose-per-endpoint logging mode. The default Migrated X/Y (z%) — N ep/sec — ETA … line is enough for ops; --benchmark covers diagnostic depth.

Test results

Verified locally on the docker dev stack with DD_V3_FEATURE_LOCATIONS=True against a representative seed (5 products × 10 endpoints × 20 findings, with the four non-Active status flag distributions exercised: 70% Active / 15% Mitigated / 5% RA / 5% FP / 5% OOS):

• Strict accuracy verifier passes after the migration: counts (URLs, Locations, LFR, LPR, location-DojoMeta), per-LFR field equality (status, created, audit_time, auditor), endpoint→location tag subset, and DojoMeta (location, name) parity all agree with source data.
• Idempotent: running the migration a second time produces no new rows and no errors; verifier still passes.
• python manage.py check clean. ruff check clean.
• The verifier and seed scripts used during development are local-only scratch tools; they are not in this PR (intentionally — single-file diff).

I have not exercised this on a 22k-finding instance directly. The 50/1,000 numbers above are linear-scaling from a per-endpoint perspective; if anyone has a prod-shape instance handy, validation there would be welcome.

Documentation

No user-facing docs to update — this is a one-shot internal data migration command. The new --benchmark / --query-count / --batch-size / --progress-every flags are documented via argparse help= strings.

[valentijnscholten]

Sounds good. But wouldn't it make sense to also add the step to inherit the tags as it's cheap / easy?

[mtesauro]

Approved