feat: add EOL package archival and pruning tooling#191
Conversation
dbb17ef to
7786b8f
Compare
Add incremental APT/YUM archive migration scripts that move EOL distro packages to archive buckets, and pruning scripts that remove EOL Ruby version packages from still-supported distro repos. Scripts auto-detect EOL content by comparing repo state against config.yml and support --dry-run. Migration scripts are incremental — safe to re-run when future distros reach EOL. Includes dev-handbook runbook and updated architecture docs.
7786b8f to
e523f8a
Compare
- Wire up GCloudStorageLock in all four archive scripts so migrations and prunes mutually exclude with the publish workflow's pointer updates. The lock library was already imported but never used. - Wrap all four scripts in begin/ensure cleanup so a failure mid-run no longer leaves the imported GPG signing key on disk. - migrate-apt-to-archive: stop swallowing pool-merge errors via '|| true'. Real I/O/permission failures must surface; only no-clobber skips were ever benign. - migrate-apt-to-archive / migrate-yum-to-archive: comments claimed EOL detection compared against environments/, but the code compares against config.yml. Align the comments with the code. - archiving-eol-packages.md: APT explicit-distros example listed centos-8, which is RPM-only and would fail in the APT script. Replace with debian-10,ubuntu-20.04.
6c19add to
68b8e37
Compare
|
Are we planning on announcing the removal of EOL Ruby packages in some way so that users can archive them themselves if they are still unmigrated? |
FooBarWidget
left a comment
There was a problem hiding this comment.
Instead of pruning EOL packages, we should move them to the archive repos. EOL packages should still be installable.
abtreece
left a comment
There was a problem hiding this comment.
Agreed — installability of EOL Ruby packages on still-supported distros is a guarantee we shouldn't break. The current "pruning" approach is wrong for the same reason we archive EOL distros instead of deleting them.
I'll rework this:
prune-apt-packages.rb/prune-yum-packages.rb→ archive-move semantics (copy to archive bucket, then remove from live repo, idempotent, dry-run preserved). Rename toarchive-*so the tooling name matches what it does.- Update
dev-handbook/archiving-eol-packages.mdand the PR title/description to reflect archival, not pruning. - Coordinate with
fullstaq-ruby/infra#47on bucket layout.
One clarifier before I start: should EOL-Ruby-on-supported-distro packages land in the existing per-distro archive buckets (extending their scope so each archive bucket can host EOL versions for both EOL and still-supported distros), or do you want a separate archive layout that mirrors live distros? The first is simpler; the second keeps "EOL distro archive" and "EOL version archive" as distinct concepts. Either way the answer shapes infra#47, so I'd like to settle it before reworking the scripts.
Also re: @noahssarcastic's question above — with archive-instead-of-delete, no user action is required to retain installability, but I'll still add a heads-up to release notes when this ships.
infra#47 renames the archive buckets from `-repo-archive` to `-archive-repo` for naming consistency with the live buckets (see fullstaq-ruby/infra@c1814eb). Scripts here are bucket-name agnostic (they read `ENV['ARCHIVE_REPO_BUCKET_NAME']`), but the old names were baked into the runbook examples, the architecture doc, and the script header docstrings -- updated here so the runbook stays correct after both PRs merge. No code changes.
I don't know what "separate archive layout that mirrors live distros" means. Do you mean whether still-supported Ruby versions for still-supported distros should appear in the archive repo? The answer is no. The normal and archive repos should be non-intersecting sets. If the user wants to install either EOL'ed Ruby versions, or supported Ruby versions on EOL'd distros, then the user must add the archive repo. The normal repo is only for Ruby packages where both the Ruby version and the distro are still supported. If one of those two factors become EOL, then the corresponding packages should be archived. |
Summary
Context
Addresses #190 — CI disk space exhaustion caused by the ever-growing Aptly
state.tar.zst.Two-axis cleanup:
Migration scripts are incremental — safe to re-run when future distros reach EOL. Each run fetches the existing archive state, merges new content, and uploads as version N+1.
Retiring centos-8 from the active build matrix (and any other EOL distro) is intentionally out of scope for this PR and will be handled as a follow-up that edits
config.ymland regenerates workflows. This PR only delivers the tooling.Companion PR: fullstaq-ruby/infra#47 (GCS buckets, DNS, Caddy config)
Implementation plan:
context/plans/EOL-ARCHIVE-MIGRATION.md(Tasks 4-9)New scripts
archive/migrate-apt-to-archive.rbarchive/migrate-yum-to-archive.rbarchive/prune-apt-packages.rbarchive/prune-yum-packages.rbAll scripts support
--dry-runand auto-detect EOL content by comparing repo state againstconfig.yml.Test plan
eclint checkpasses--dry-runcorrectly identifies EOL distros/packagesfullstaq-ruby-X.Y*packages for non-active minor versions are removeddev-handbook/archiving-eol-packages.mdmatch actual script behavior