worktree: transparent copy-on-write population via reflinks#2183
worktree: transparent copy-on-write population via reflinks#2183walter-zeromatter wants to merge 7 commits into
Conversation
Add a helper that clones a file using the FICLONE ioctl on Linux, creating the destination as a lightweight copy-on-write copy that shares extents with the source on filesystems that support it (e.g. btrfs, XFS, bcachefs). On other platforms the helper always fails with ENOSYS; callers are expected to fall back to a regular copy. Also add a test-tool command and a REFLINK test prerequisite so tests can probe whether the filesystem the test suite runs on supports reflinking. This will be used to speed up populating new worktrees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Walter Gray <walter@0m.dev>
When the GIT_WORKTREE_REFLINK_SOURCE environment variable points at the primary worktree, write_entry() tries to create each regular file as a copy-on-write clone (reflink) of the donor's file instead of inflating the blob from the object database. A file is only cloned when the donor's index has the same path with the same object id and mode, the donor file is stat-clean (racily clean entries are conservatively treated as dirty, and assume- unchanged and fsmonitor validity are ignored so that a real stat comparison happens), no content conversion (CRLF, ident, filter, working-tree-encoding) applies to the path, and the donor file size matches the blob size. Anything else falls through to the regular write path, so behavior on filesystems or platforms without reflink support is unchanged. Successfully cloned files flow into the existing refresh_cache handling, so their stat information is recorded in the new index and they are never re-hashed. Two trace2 counters, reflink/attempts and reflink/hits, provide observability and a test hook. This addresses the dirty-donor concern raised against the earlier opt-in --reflink proposal: cloning is decided per file from the donor index, never by blindly copying the donor tree. Note that a donor whose index was written in the same second as its files (e.g. immediately after a clone or commit) is racily clean, and its entries are conservatively skipped rather than cloned; a later patch will make "git worktree add" refresh the donor index before spawning the checkout so that an aged donor becomes eligible. The tests therefore age the donor files and refresh the index to make the donor deterministically non-racy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Walter Gray <walter@0m.dev>
Teach "git worktree add" to point the checkout it spawns at the primary worktree via GIT_WORKTREE_REFLINK_SOURCE, so that eligible files are created as copy-on-write clones on filesystems that support it. The feature is transparent: there is no command-line option, unsupported filesystems and platforms silently fall back to a regular checkout, and the working tree contents are identical either way. Before pointing the child checkout at the primary worktree, refresh the primary worktree's index (git update-index -q --refresh, non-fatal on failure) so that files committed or modified moments ago are not left "racily clean" and therefore skipped by the reflink logic in entry.c. A single configuration variable, worktree.useReflink (default true), is provided for users who want physically independent copies (shared extents also share on-disk corruption) or need to work around filesystem problems. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Walter Gray <walter@0m.dev>
try_reflink_entry() only reflinked a donor file when its path carried no content conversion at all (ca->crlf_action == CRLF_BINARY). In git.git's own tree that disqualifies the great majority of files, because its .gitattributes marks them "* text eol=lf", which convert_attrs() resolves to CRLF_TEXT_INPUT -- so a "git worktree add" reflinked only ~32% of the tree and kept paying a full copy for the rest. The checkout-direction (smudge) conversion for the input-style CRLF actions is the identity: crlf_to_worktree() returns early unless output_eol() is EOL_CRLF, which holds only for the "*_CRLF" actions, not for CRLF_TEXT_INPUT or CRLF_AUTO_INPUT. An LF blob therefore checks out byte-for-byte as its own content, so the worktree bytes of a fresh checkout equal the blob bytes. The clean direction only ever removes CR bytes (it never adds or substitutes), so a stat-clean donor whose on-disk size equals the blob size cannot have diverged from the blob -- the existing size==blob backstop makes a byte-divergent donor impossible. Admit CRLF_BINARY, CRLF_TEXT_INPUT, and CRLF_AUTO_INPUT via a small helper; ident, custom drv filters, and working-tree-encoding remain excluded because their smudge is not the identity. On git.git this lifts reflink eligibility from 1546/4779 (~32%) to 4773/4780 (~99.85%) of tracked files, cutting a "worktree add"'s exclusive (unshared) disk usage on btrfs from 17.33 MiB to ~12 KiB with no measurable time cost. Signed-off-by: Walter Gray <walter@0m.dev> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add GIT_WORKTREE_REFLINK_SOURCE_ENVIRONMENT to local_repo_env[] in environment.c so it is cleared when git crosses a repository boundary (e.g. into a submodule), matching how other internal repo-local vars are handled; the `worktree add` checkout child still receives it via explicit child_env, unaffected by this change. Also replace the empty-statement `if (run_command(&cp)) ;` idiom in builtin/worktree.c with a plain statement plus a leading comment. Along the way, fix mislabeled test suite names in the benchmark regression-sweep table (t2401/t2402/t2404/t2405). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Walter Gray <walter@0m.dev>
push_reflink_source_env() spawns "git update-index -q --refresh"
against the primary worktree's path (cp.dir), but did not clear the
repo-local environment variables (GIT_DIR, GIT_WORK_TREE, etc.) that
the parent process may have inherited or been given explicitly.
When "worktree add" is invoked as "git -C <linked-worktree> worktree
add ...", the parent process has GIT_DIR exported pointing at the
linked worktree's gitdir (.git/worktrees/<id>). The refresh child
inherits that GIT_DIR and, ignoring cp.dir, ends up refreshing the
linked worktree's index instead of the primary/donor's. The donor
index is therefore never refreshed, so every donor file fails the
stat-clean gate in the reflink checkout path and reflinking silently
never fires for worktrees created from a linked worktree.
This same stale-donor-index bug is the root cause of the intermittent
failure of t2408 test 11 ("worktree add from a linked worktree uses
the primary donor"): the test only passed when consecutive tests
happened to land within the same wall-clock second, which made the
stale cached stat data spuriously match the donor file's real stat
data.
Fix this by pushing local_repo_env onto the child's environment before
running it, the standard idiom (see trailer.c and odb.c) for making a
spawned git process rediscover its repository from cwd/-C rather than
inheriting the parent's repo-local environment.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Walter Gray <walter@0m.dev>
Close five test-coverage gaps in the reflink eligibility gate (try_reflink_entry() in entry.c) that were not yet pinned by any test: - First FICLONE failure disables further reflink attempts for the rest of the process (attempts stays at 1 across 3 eligible files), gated !REFLINK since it requires a filesystem where FICLONE genuinely fails. - A donor entry marked skip-worktree is excluded, even though the target worktree's own fresh entry has no such flag. - A donor entry with CE_VALID (assume-unchanged) and stale, same-size on-disk content is rejected because CE_MATCH_IGNORE_VALID forces a real stat compare instead of trusting the assume-unchanged bit; using same-size replacement content is deliberate, since a different-size replacement would also be rejected by the independent size==blob backstop and wouldn't isolate the CE_MATCH_IGNORE_VALID behavior (verified by toggling the flag locally and observing this test flip). - A donor index entry whose mode was flipped via "update-index --chmod" without touching the tree is rejected by the ce_mode comparison before any stat is even done. - Gitlink entries, which never reach try_reflink_entry() at all (it is only consulted for S_IFREG blobs), check out identically with or without an active reflink donor. No production code changes; all 20 t2408 tests pass on both btrfs (--root, 3x for stability) and ext4 (plain), with REFLINK/!REFLINK prereqs gating the right subset on each filesystem. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Walter Gray <walter@0m.dev>
Welcome to GitGitGadgetHi @walter-zeromatter, and welcome to GitGitGadget, the GitHub App to send patch series to the Git mailing list from GitHub Pull Requests. Please make sure that either:
You can CC potential reviewers by adding a footer to the PR description with the following syntax: NOTE: DO NOT copy/paste your CC list from a previous GGG PR's description, Also, it is a good idea to review the commit messages one last time, as the Git project expects them in a quite specific form:
It is in general a good idea to await the automated test ("Checks") in this Pull Request before contributing the patches, e.g. to avoid trivial issues such as unportable code. Contributing the patchesBefore you can contribute the patches, your GitHub username needs to be added to the list of permitted users. Any already-permitted user can do that, by adding a comment to your PR of the form Both the person who commented An alternative is the channel Once on the list of permitted usernames, you can contribute the patches to the Git mailing list by adding a PR comment If you want to see what email(s) would be sent for a After you submit, GitGitGadget will respond with another comment that contains the link to the cover letter mail in the Git mailing list archive. Please make sure to monitor the discussion in that thread and to address comments and suggestions (while the comments and suggestions will be mirrored into the PR by GitGitGadget, you will still want to reply via mail). If you do not want to subscribe to the Git mailing list just to be able to respond to a mail, you can download the mbox from the Git mailing list archive (click the curl -g --user "<EMailAddress>:<Password>" \
--url "imaps://imap.gmail.com/INBOX" -T /path/to/raw.txtTo iterate on your change, i.e. send a revised patch or patch series, you will first want to (force-)push to the same branch. You probably also want to modify your Pull Request description (or title). It is a good idea to summarize the revision by adding something like this to the cover letter (read: by editing the first comment on the PR, i.e. the PR description): To send a new iteration, just add another PR comment with the contents: Need help?New contributors who want advice are encouraged to join git-mentoring@googlegroups.com, where volunteers who regularly contribute to Git are willing to answer newbie questions, give advice, or otherwise provide mentoring to interested contributors. You must join in order to post or view messages, but anyone can join. You may also be able to find help in real time in the developer IRC channel, |
|
Note: I've got this in draft while I iterate on it & check CI state. I'll mark it as ready for review when I've actually gone through the code by hand. I'm using AI for a lot of the generation but will be reviewing every line before marking ready for anyone else to look at. |
1 participant
Populating a new worktree currently writes every file by inflating blobs from the object database, so N worktrees cost N full checkouts of disk and I/O. On filesystems with copy-on-write support (btrfs, XFS, bcachefs), the new worktree's files can instead be lightweight clones of the primary checkout's files.
This series teaches
git worktree addto do that transparently: no new command-line option, a singleworktree.useReflinkconfig (default true) as an escape hatch, and silent per-file fallback to the regular checkout path everywhere else — unsupported filesystems, other platforms, or any file that fails the eligibility gate. Output and results are byte-identical to an unpatched git in all cases.This follows up on the discussion around the earlier
--reflinkproposal (gitgitgadget#2317):entry.c:write_entry()) as a transparent optimization rather than an opt-in mode: if a clean file with the same content exists in the primary worktree, clone it; otherwise write from the object database.Because a freshly cloned or committed donor is racily clean in its entirety,
git worktree addrefreshes the primary worktree's index (update-index -q --refresh, non-fatal) before spawning the checkout, in an environment scrubbed of repo-local variables sogit -C <linked-worktree> worktree addrefreshes the right index.Input-style CRLF paths (
text eol=lf,text=autowithcore.autocrlf=input) are also eligible: their checkout-direction conversion is the identity, and since CRLF cleaning only ever removes bytes, the size==blob gate makes a byte-divergent donor impossible. This mattered in practice: git.git's own.gitattributeswould otherwise exclude ~68% of its files.Reflinked files flow into the normal
refresh_cachehandling, so their stat information lands in the new index and they are never re-hashed — unlike external tools (git-cow-worktree, coworktree) that reflink behind git's back and pay a full re-hash on the next status.Benchmark (btrfs, kernel 7.0.9, git.git itself, 4779 files):
worktree addallocates ~12 KiB of new data instead of ~48 MiB, with 4773/4780 files reflinked, at slightly better wall time than a regular checkout (no zlib inflation for cloned files).Platform support is Linux
FICLONEonly in this series; the wrapper is a stub returningENOSYSelsewhere. macOSclonefile()and Windows ReFS could be follow-ups behind the samereflink_file()interface. Files handled by parallel checkout (checkout.workers > 1) take the normal write path, correct but unaccelerated.Tests: new t2408 with a
REFLINKlazy prerequisite (and!REFLINKcases for the fallback path), covering the happy path via trace2 counters, dirty/racy/assume-unchanged/skip-worktree/mode-mismatch donors, conversion attributes, symlinks, gitlinks, executable bits, the kill switch,--no-checkout, and invocation from a linked worktree.🤖 Generated with Claude Code