initial commit for multidb - #7862
Conversation
|
/retest |
getattr(value, 'pulp_domain_id', None) silently returns None for models that reach their domain transitively rather than through their own field (RepositoryVersion.repository, RepositoryContent.repository, ContentArtifact.content, PublishedArtifact.publication). RepositoryVersion is the single most common CreatedResource target in pulpcore (every sync/publish creates one), so this meant most real-world post-move CreatedResource rows for satellite-hosted domains silently resolved content_object on the wrong (control-plane) alias instead of raising or logging -- the exact KI-18 landmine, reached via an uncovered path. Found via live validation against a real two-Postgres environment: a genuine post-move CreatedResource pointing at a satellite RepositoryVersion resolved to a stale pre-cleanup copy on 'default' instead of the live row on the satellite alias. Fixes by walking the target's concrete FK/O2O fields (bounded depth, cycle-guarded) to find any related object exposing pulp_domain_id, rather than a hardcoded model list -- covers all four known cases generically and any future/plugin model with the same shape. Co-authored-by: Cursor <cursoragent@cursor.com>
…ntext test_content_object_domain_id_set_for_repository_version created the CreatedResource outside with_task_context(task), so CreatedResource.task (a required NOT NULL FK) never got auto-populated -- IntegrityError on insert. The aborted transaction from that failure can poison the connection state for whatever DB-touching test runs next in the same session, which is why this surfaced downstream (test_reconciliation.py) in a full-suite run rather than pointing at the actual buggy test. Verified clean: full pulpcore.tests.unit suite (363 passed, 2 skipped, 0 failed) against two real local Postgres instances mirroring the CI 'multi_db' matrix leg, plus a separate single-DB (no data_1) run confirming the skip path still works. Co-authored-by: Cursor <cursoragent@cursor.com>
|
@gerrod3 What do you think? Can I get a review? |
gerrod3
left a comment
gerrod3
left a comment
There was a problem hiding this comment.
Round 1 of reviews. This is honestly quite unreviewable in its current state. The AI comments are a nightmare and make references to docs and comments I don't have access to. The commits are not logically structured either. I would probably have ordered them something like:
- The initial adding of the database-alias and db-router
- Fixing management commands and other models (GenericReleation)
- Adding the database domain migration command
- CI work and tests
I would like to set expectations now that this will require major changes and many iterations before we are close to a state that might be mergeable.
| if domain is not None: | ||
| return getattr(domain, "database_alias", "default") |
There was a problem hiding this comment.
Does this getattr not trigger something crazy too?
| CROSS_PLANE_RECONCILIATION_GRACE_MINUTES = 60 | ||
|
|
||
| # KI-11: how long, in days, a confirmed-orphaned cross-plane row is kept (logged/alerted on every | ||
| # sweep) before the reconciliation sweep deletes it outright. 0 disables purging entirely -- | ||
| # orphans are only ever logged, never deleted, which is the safe default. | ||
| CROSS_PLANE_RECONCILIATION_PURGE_AFTER_DAYS = 0 |
There was a problem hiding this comment.
Why do we need these two different settings?
| if len(settings.DATABASES) > 1: | ||
| DATABASE_ROUTERS = ["pulpcore.app.db_router.PulpDomainRouter"] | ||
| settings.set("DATABASE_ROUTERS", DATABASE_ROUTERS) |
There was a problem hiding this comment.
Remove this. Routers must be explicitly set by the user.
| if len(settings.DATABASES) <= 1: | ||
| return super().filter(*args, **kwargs) |
There was a problem hiding this comment.
This check should be if the router is the special PulpMultiDBRouter, not if there are more than one 1 database.
| _DOMAIN_WALK_MAX_DEPTH = 2 | ||
|
|
||
|
|
||
| def _resolve_domain_id(value, _depth=0, _seen=None): |
There was a problem hiding this comment.
Do we really need this? Yeah it's probably the safest way to get the domain of the object, but I would expect that get_domain would always return the correct domain that the object is in. Maybe it doesn't matter since creating GenericRelationships typically never happen in a hot path
| read-only copy of every `Domain` row must also exist on every other configured `DATABASES` alias | ||
| so that per-process code (the router, `for_each_domain()`, `Domain.get_storage()`, etc.) can |
There was a problem hiding this comment.
Why should this be true? It doesn't seem like it should be. The only thing that I would expect that needs to exist on the satellite dbs is the default domain and the satellite domain.
2 participants
Initial PR for the multidb implementation. To test github actions. etc