ENG-3517: Upload and promotion for user uploaded files - schema variant, storage util, hooks, config

[mikeGarifullin]

Ticket ENG-3517

Stack: depends on #8110 (data-layer foundation). Merge that first.

List of related changes

#	Ticket	Repo	PR	Description	Approved
1	ENG-3517	fides	#8110	Foundation — attachment_user_provided model + repository (lifecycle, locking, orphan-sweep query)	[merged]
this PR	ENG-3517	fides	#8113	FileUploadCustomPrivacyRequestField schema variant, storage util (magic bytes, allowed types), promotion hook, config flags	[]
3	ENG-3517	fidesplus	#3524	Attachment service + PrivacyRequestService promotion hook implementation	[]
4	ENG-3517	fidesplus	#3534	POST /privacy-request/attachment HTTP endpoint (multipart upload, per-field constraints, rate limit)	[x]
5	ENG-3518	fidesplus	#3526	Hourly Celery orphan-sweep: deletes temp storage objects + uploaded rows older than cutoff	[x]
6	ENG-3519	fidesplus	#3543	Local-dev clamav-icap container (Ubuntu + c-icap + libclamav, profile-gated in compose)	[x]
7	ENG-3520	fidesplus	#3544	Wire ICAP scanner into upload path (RESPMOD call, X-Infection-Found handling)	[x]

Description Of Changes

Schema + storage + service-extension half of the upload foundation. Pairs with the data-layer PR (#8110); the upload service and endpoint live in fidesplus and consume both.

Adds the Privacy Center config variant for file fields, a magic-byte sniff catalog so the upload route can verify a payload's claimed type at byte-zero (client-supplied Content-Type is not trusted), two config knobs for the global upload ceilings, and two PrivacyRequestService hook points so fidesplus can plug attachment-resolve and attachment-promote into the existing submission flow without overriding create_privacy_request wholesale.

Code Changes

• schemas/privacy_center_config.py: FileUploadCustomPrivacyRequestField discriminated-union variant (field_type="file", max_size_bytes > 0, allowed_file_types non-empty + must be a subset of AllowedFileType).
• schemas/attachment.py: AttachmentUploadResponse (the {id: "att_..."} payload returned by the fidesplus upload route).
• service/storage/util.py:
  • FilesMagicBytes — known signatures (pdf, jpg/jpeg, png, gif, ...) and from_bytes(...) for byte-zero detection.
  • AllowedFileType.mime_types(), MIME_TO_EXTENSION, extension_for_mime(...) helpers.
  • FileUploadConstraints — runtime-validated config bag (per-field cap + allowed types) reused by the upload route and the schema validator.
• service/privacy_request/privacy_request_service.py:
  • _resolve_attachment_state(submitted_data, *, db) and _promote_attachment_state(privacy_request, state) — overridable no-op hooks on the OSS service. Default impls do nothing so OSS behavior is unchanged.
  • create_privacy_request calls _resolve_attachment_state before persisting and _promote_attachment_state after caching/masking-secrets. On promotion failure the request row is deleted (clear_cached_values() runs as part of delete()) and the original error is rewrapped as PrivacyRequestError.
• common/urn_registry.py: route constant for the future fidesplus upload endpoint.
• config/security_settings.py: request_attachment_max_bytes global ceiling (caps the per-field max_size_bytes).
• config/execution_settings.py: attachment_orphan_ttl_seconds for the orphan-cleanup sweep.
• Tests for the schema variant, magic-byte catalog, and FileUploadConstraints.

Steps to Confirm

1. Stack rebased onto ENG-3517: Foundation - attachment_user_provided model + repository #8110. From this branch run nox -s "pytest(ops-unit-non-api)" -- tests/ops/schemas/test_attachment.py tests/ops/schemas/test_privacy_center_config.py tests/ops/service/storage/test_util.py — all green.
2. Construct a FileUploadCustomPrivacyRequestField with max_size_bytes=0 or allowed_file_types=[] — pydantic should reject. Construct one with an extension outside AllowedFileType — pydantic should reject.
3. FilesMagicBytes.from_bytes(open("sample.pdf","rb").read(8)) returns the pdf member. The same call against random bytes returns None.
4. From a Python shell: instantiate PrivacyRequestService and call create_privacy_request(...) with a non-file payload — flow runs unchanged (hooks are no-ops).
5. (Cross-PR) The fidesplus upload route consumes FileUploadConstraints, FilesMagicBytes, and the new hook points — verify in the companion fidesplus PR.

Pre-Merge Checklist

• Issue requirements met
• All CI pipelines succeeded
• CHANGELOG.md updated
  • Add a db-migration This indicates that a change includes a database migration label to the entry if your change includes a DB migration
  • Add a high-risk This issue suggests changes that have a high-probability of breaking existing code label to the entry if your change includes a high-risk change (i.e. potential for performance impact or unexpected regression) that should be flagged
  • Updates unreleased work already in Changelog, no new entry necessary
• UX feedback:
  • All UX related changes have been reviewed by a designer
  • No UX review needed
• Followup issues:
  • Followup issues created
  • No followup issues
• Database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!
  • No migrations
• Documentation:
  • Documentation complete, PR opened in fidesdocs
  • Documentation issue created in fidesdocs
  • If there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
  • No documentation updates required

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
fides-plus-nightly	Ready	Preview, Comment	May 14, 2026 3:33pm

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
fides-privacy-center	Ignored		May 14, 2026 3:33pm

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.59%. Comparing base (78fd8e0) to head (88538b2).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8113      +/-   ##
==========================================
+ Coverage   85.57%   85.59%   +0.01%     
==========================================
  Files         657      658       +1     
  Lines       42727    42847     +120     
  Branches     5012     5018       +6     
==========================================
+ Hits        36565    36675     +110     
- Misses       5055     5064       +9     
- Partials     1107     1108       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JadeCara]

Left a few comments - thinking the bool change is the most important to verify.

[JadeCara]

Thanks for making the updates! I ran a quick security review on it too and it all came back clean!