Skip to content

Fix tag-based manifest pulls 404ing while digest-based pulls succeed - #2438

Open
wussh wants to merge 6 commits into
pulp:mainfrom
wussh:fix-2417-tag-manifest-404
Open

Fix tag-based manifest pulls 404ing while digest-based pulls succeed#2438
wussh wants to merge 6 commits into
pulp:mainfrom
wussh:fix-2417-tag-manifest-404

Conversation

@wussh

@wussh wussh commented Jul 26, 2026

Copy link
Copy Markdown

Summary

  • ManifestResponse.from_tag rejected requests based on the client Accept header, raising ManifestNotFound when the tag's manifest media type wasn't explicitly listed. from_manifest (the digest lookup path) never had this check, so the identical content served 200 by digest and 404 by tag — most visible for manifest-list/OCI-index tags whose media type clients often omit from Accept.
  • Regression from Omit redirects to content-app #1974 (8479d527), which moved manifest serving from a content-app redirect (that ignored Accept) into the registry API view.
  • from_tag now mirrors from_manifest: it always serves the tagged manifest, only rewriting schema v1 to the signed content type.

Fixes #2417

Test plan

  • Reworked test_pull_content.py::test_api_performes_schema_conversion (previously asserted a 4xx with the comment "I don't understand what this is testing") into test_api_serves_tag_regardless_of_accept_header, which pulls the same manifest by tag and by digest with a narrow Accept header and asserts identical 200 responses.
  • Reproduced end-to-end on the live domain-scoped, on-demand, follow-latest environment that originally reported Domain-scoped container registry: tag-based manifest fetch 404s, digest-based works (pulpcore 3.113.0 / pulp_container 2.28.0) #2417 (Domains enabled, S3-backed storage, pulpcore 3.113.0 / pulp_container 2.28.0). Pulled the exact live registry_api.py out of the running pulp-api pod, applied only the from_tag diff from this PR on top of it (no unrelated version drift), wrote it back into the running container, and reloaded gunicorn workers (no pod restart). Results:
    • Before patch: tag pull 404, digest pull 200 (matches the issue exactly).
    • After patch: tag pull 200, digest pull 200, tags/list 200, and the tag/digest response bodies are byte-identical.
    • Verified pod stayed healthy throughout, then reverted the pod back to the original unpatched image and re-confirmed the original 404 baseline — no lasting changes were left on the cluster.

@wussh

wussh commented Jul 26, 2026

Copy link
Copy Markdown
Author

Verified this fix end-to-end against the live cluster that originally hit #2417 (Domains + on-demand + follow-latest distribution, S3 storage, pulpcore 3.113.0 / pulp_container 2.28.0):

  • Before: GET /v2/default/quay/prometheus/prometheus/manifests/latest404; the same manifest by digest → 200.
  • After (applying only this PR's from_tag diff to the live registry_api.py, no unrelated changes, then reloading gunicorn workers): tag pull → 200, digest pull → 200, tags/list200, and the tag/digest response bodies are byte-identical.
  • Cluster was left in its original (pre-fix) state afterward — no permanent changes.

@gerrod3

gerrod3 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@wussh Did you see my comment on the original issue? I'm not sure we should be removing the accept header check. In fact we should probably add it to the digest route for consistency. I went and fix some bugs with the accept header check #2424

Comment thread lint_requirements.txt Outdated
wussh added 6 commits July 31, 2026 17:26
ManifestResponse.from_tag rejected requests based on the client Accept
header, raising ManifestNotFound when the tag's manifest media type
wasn't explicitly listed. from_manifest (the digest lookup path) never
had this check, so identical content served 200 by digest and 404 by
tag — most visible with manifest-list/OCI-index tags whose media type
clients often omit from Accept.

This was introduced in 8479d52 (pulp#1974) when manifest serving moved
from a content-app redirect (which ignored Accept) into the registry
API view. from_tag now matches from_manifest: it always serves the
tagged manifest, only rewriting schema v1 to the signed content type.

Fixes pulp#2417
Fixes a ruff UP032 finding introduced by the new test added in this branch.
pulpcore treats any task exception that isn't a PulpException subclass
as deprecated: it logs a pulpcore.deprecation warning today and will
sanitize the error message away entirely in pulpcore 3.130, losing the
actual podman/buildah stderr. build_image_from_scratch and the
deprecated build_image_from_containerfile both raised bare Exception
on a failed podman build/push, tripping this path — surfaced as CI
noise in the "deprecations" check, but a real forward-compat bug.

Add ContainerBuildError(PulpException) and raise it with the decoded
stderr instead; pytest.raises(PulpTaskError) assertions in
test_build_images.py are unaffected since the exception still
propagates through task.error the same way.
ContainerNamespace has a PROTECT foreign key to Domain. cdomain_factory's
teardown cleared content-redirect guards but never ContainerNamespace
objects, so any domain-scoped ContainerDistribution left one behind
(auto-created by ContainerNamespaceSerializer.get_or_create). When
pulpcore's own domain_factory teardown then deleted the domain, Django
raised ProtectedError — sanitized into a pulpcore.deprecation warning
that failed the "deprecations" CI check.

test_pull_in_domain creates two distributions and cleaned up neither
namespace, making it the concrete trigger. Query-and-delete namespaces
by domain in cdomain_factory's teardown instead of relying on each test
to remember; safe to run even for tests that already clean their own
namespace via add_to_cleanup since it re-lists current state.
PulpException.__init__(error_code=...) is itself deprecated in current
pulpcore — passing error_code positionally now logs its own
pulpcore.deprecation warning. error_code must be a predefined class
attribute instead.
- error_code: use the CON00XX naming convention (CON0001) jobselko
  and gerrod3 agreed on, instead of PLP_CONTAINER_BUILD_0001.
- CHANGES fragment: use single backticks, since they render as
  markdown.
- cdomain_factory: adopt gerrod3's version of the namespace cleanup
  from pulp#2424 (suppress errors per-namespace/guard instead of letting
  one failure abort the rest, and wait on the delete task via
  monitor_task).
- Drop the ruff<0.16 pin: no longer needed after rebasing on main,
  where the 0.16 lint findings were already fixed elsewhere.
@wussh
wussh force-pushed the fix-2417-tag-manifest-404 branch from 806b30a to 5fae027 Compare July 31, 2026 10:28
@wussh

wussh commented Jul 31, 2026

Copy link
Copy Markdown
Author

Addressed the mechanical review feedback:

  • Dropped the ruff<0.16 pin — rebased on main, where the 0.16 lint findings were already fixed elsewhere, so it's no longer needed.
  • ContainerBuildError.error_code renamed to CON0001 per the naming convention you and @jobselko settled on.
  • Changelog fragment now uses single backticks (renders as markdown).
  • cdomain_factory now uses your version of the namespace cleanup from Fix 404 on tag fetch due to overly strict Accept header matching #2424 (suppress + monitor_task) instead of mine.

Still not addressed: your point about the accept-header check itself (removing it from from_tag vs. adding it to the digest route for consistency, and #2424 fixing the underlying matching bugs). Wanted to get the mechanical stuff out of the way first — let me know how you'd like to proceed on that part, happy to rework or close this in favor of #2424 if that's the better fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Domain-scoped container registry: tag-based manifest fetch 404s, digest-based works (pulpcore 3.113.0 / pulp_container 2.28.0)

3 participants