feat(run-task): add git sparse checkout support - #1039
Conversation
7f6bc32 to
29d0422
Compare
…ronment The patterns travel in `<REPO>_SPARSE_PATTERNS` as a JSON list of native git patterns, so the cloned revision does not need to contain the profile. A JSON list because generic-worker on Windows sets task environment variables through a batch `set` line, which cuts a value at its first newline. The option and the variable have to come together, and a checkout is required for either. Patterns go through stdin so a name starting with a dash is not taken for an option. A full task on a sparse cache drops the sparse checkout first. On the Mercurial path the option goes to robustcheckout as --sparseprofile. A sparse clone is blobless (--filter=blob:none) and defers its checkout until the patterns are set, so only files inside the profile are written or fetched. Git keeps that filter on the origin remote and a fetch only inherits it when it names that remote, so run-task fetches through origin whenever the head repository is the repository it cloned, and points origin at the task's repository first so a cached clone follows a repository that moved. git 2.49 keeps the filter for a fetch by matching URL as well, but that is not documented and the workers run other versions. Refs taskcluster#1025
Sparse and full tasks in a worker pool share one checkout cache, and this change decides what happens when a task finds a cache another kind of task left behind: the cache only ever grows. A full task on a sparse cache widens it to a full checkout, and every later task on that worker runs full. A sparse task on a full cache runs on it as is. A sparse task on a cache with another sparse profile appends its own patterns with `sparse-checkout add`, so the cache ends up covering both profiles. Appending is why every translated pattern has to be positive. Whatever grows the cache runs after the checkout of the target revision, so a widen or an addition fetches that revision's files and not the old revision's. A cache in cone mode is treated as foreign and widened for now. The next commit starts creating cone caches and takes them over. Refs taskcluster#1025
Git tests a no cone pattern against every index entry on every checkout. Cone mode accepts only directories, and in return git matches by prefix and keeps the index itself small with the sparse index. When every pattern is an anchored literal path, which is true for twelve of the eighteen profiles, run-task applies them with `set --cone --sparse-index`. Locally repackage-msi went from 15 to 1 second on a warm cache. A path to a file works as a cone directory because cone mode includes the files of every ancestor directory. The same rule means a cone checkout has a few hundred more files than the profile names, the files directly inside every ancestor up to the root. Never fewer, so tasks are unaffected. Mixing follows the widen only rule. A cone cache takes further cone profiles with `add`. A glob profile on a cone cache converts it to no cone mode and keeps every directory it had plus the files of their ancestors, listed with `git ls-tree` at the target revision, so nothing that was checked out disappears. Refs taskcluster#1025
29d0422 to
8c33f29
Compare
bhearsum
left a comment
bhearsum
left a comment
There was a problem hiding this comment.
I think I need to take a second pass on this with fresh brains; here's a few thoughts to start.
A full task widens a sparse cache.
Does this mean that tasks that use sparse profiles that run after a full checkout is cached are less performant than those that run with nothing cached, or after a different task that used a sparse profile?
Patterns go through stdin so a name starting with a dash is not taken for an
option.
This also avoids the possibility of hitting any limits on the length of arguments, presumably?
You're correct that with a split cache, a sparse task landing on a sparse cache is faster than it is with this shared cache, but overall, on average, this is still a net win. Workers get tasks without regard to whether they are sparse or full, so on any pool that has both kinds, most workers end up running both, and the two will always mix on their cache(s). Each approach pays for the mix somewhere. The split caches pay for it with a cold clone per kind. One cache pays for it with a sparse task after a full task having to do its fetch, checkout, and clean over the whole tree rather than just its profile. I tested this on try for both Linux and Windows, and here's some numbers:
Which means that the full to sparse case costs a couple seconds at most, but every cold clone we avoid is savings of up to an order of magnitude. So unless there's a ~10:1 ratio of sparse to full jobs in some pool, it's very likely that the one cache solution is better in aggregate.
It does! |
2 participants
Fixes #1025
run-task gains sparse checkouts for git clones. The task supplies native git patterns in
<REPO>_SPARSE_PATTERNSas a JSON list, and names the profile with--<repo>-sparse-profile. A sparse clone is blobless and applies the patterns before its first checkout, so only the profile's files are written or fetched.Checkout caches are shared between sparse and full tasks and only ever widen:
A profile made only of paths uses cone mode with the sparse index.
On the Firefox side this is supported by Bug 2071081, which translates the Mercurial sparse profiles into these patterns, and Bug 2071092, which keys checkout caches on the run-task a task actually runs. Delivering the patterns to git tasks lands after a release with this change is pinned.