connected: add incremental connectivity check - #2211
Open
spkrka wants to merge 5 commits into
Open
Conversation
Move the inline self-contained pack detection into a helper function. This makes check_connected() easier to follow and makes the detection logic available as a standalone helper. No functional change. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
spkrka
marked this pull request as ready for review
August 28, 2026 10:15
spkrka
force-pushed
the
tree-diff-connectivity-v1-clean
branch
9 times, most recently
from
September 1, 2026 14:45
b224ce7 to
326b1e4
Compare
The connectivity check uses rev-list to find commits that are not
reachable from local refs and then walks their object closure.
Commit traversal stops at the connectivity boundary, but the trees
and blobs reachable from that boundary still need to be walked so
they can be marked uninteresting. On repositories where the
boundary commits have large trees, this makes small incoming
changes expensive.
Add an alternative connectivity check that verifies incoming
commits incrementally against their parents.
The verifier walks a new commit's tree alongside its parent trees.
Previously verified entries are skipped; changed subtrees are
descended into recursively, and newly referenced blobs are checked
for existence in the object database.
For example, consider a commit changing one file under lib/:
Parent tree New tree
+-- src/ (aaa) +-- dev/ (aaa)
+-- lib/ (bbb) +-- lib/ (ccc)
+-- foo.c (ddd) +-- foo.c (ddd)
+-- bar.c (eee) +-- bar.c (fff)
1. Read the parent root and add its direct entries to the
verified set: aaa and bbb.
2. Walk the new root:
- aaa is already verified, so the subtree is skipped even
though it appears at a different path.
- ccc is new, so recurse into it using bbb as the parent
subtree.
3. Read bbb and add its direct entries ddd and eee to the
verified set.
4. Walk ccc:
- ddd is already verified.
- fff is new, so check that the blob exists.
Thus the changed lib/ subtree is the only subtree recursively
explored, and only the new bar.c blob needs an existence check.
The root trees still need to be read and scanned as comparison
bases.
New commits are processed with ancestors before descendants.
Parents outside the incoming commit set are reachable from
existing refs and form the initial trusted boundary. Once an
incoming commit has been verified, its tree can in turn be used
as a trusted base for its children.
Because object contents are immutable, the verified sets persist
across commits -- trees and blobs verified for one commit can be
reused for later ones (changes, reverts, subtree moves, merges).
Only objects derived from that trusted commit boundary or verified
while processing incoming commits enter the verified sets. Merely
having an unrelated object in the object database does not make it
trusted.
Missed reuse only costs extra work, not correctness. The parent-side
scan is intentionally greedy and does not find every reuse opportunity.
Pruning occurs
when an object is seen by the parent-side scan before it is
encountered on the child side. When that does not happen, the
subtree is verified again. For example, if a commit moves the
subtree aaa from x/y/ to y/, the parent-side scan never descends
into x/ and never sees aaa, so the child traversal verifies that
subtree recursively.
1. Collect and peel tips. Non-commit tips are verified
immediately; tips already covered by a self-contained pack
verified by index-pack are skipped.
2. Find the incoming commit boundary with
rev-list --stdin --not --all.
3. Process those commits in topological order and verify their
trees incrementally against their parents.
Gate the new algorithm behind transfer.connectivityCheck=incremental.
Fall back to the existing rev-list path for shallow fetches, partial
clones, replacement objects, and deepening fetches.
Benchmarks on a large repository (~3M commits, ~200K trees and
~500K blobs reachable from the tip), 1 new commit changing 1
file, measured with hyperfine --warmup 1:
With ~10K local refs:
rev-list: 1849 ms +/- 60 ms
incremental: 143 ms +/- 19 ms (12.9x faster)
With 1 local ref:
rev-list: 1791 ms +/- 72 ms
incremental: 17 ms +/- 1 ms (107x faster)
The benefit grows with tree closure size in these measurements:
linux.git (~100K reachable objects) shows ~2x; git.git (~5K
objects) shows no measurable difference.
Signed-off-by: Kristofer Karlsson <krka@spotify.com>
Teach the incremental connectivity check to handle shallow fetches. Shallow commits are treated as traversal roots with no parents, matching the boundary semantics of rev-list. Add parse_shallow_file_gently() to parse the temporary shallow file without dying on errors, and thread the resulting oidset through verify_new_commits and verify_commit_tree. Pass --shallow-file to the boundary-finding rev-list so it respects the shallow grafts. Remove the shallow_file guard from incremental_check_applicable() so incremental mode is now used for shallow fetches when configured. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
Teach the incremental connectivity check to handle partial clones. When a tree or blob cannot be read, check whether it is a promisor object before reporting an error. Promisor tips with missing targets are skipped during tag peeling. Pass --exclude-promisor-objects to the boundary-finding rev-list so promisor commits do not pollute the set of commits to verify. Remove the repo_has_promisor_remote() guard from incremental_check_applicable() so incremental mode is now used for partial clones when configured. Signed-off-by: Kristofer Karlsson <krka@spotify.com>
NOTE: This commit is an experimental proof of concept to illustrate
the optimization potential. It is not meant to be merged.
The incremental connectivity check shells out to rev-list
to find the boundary between incoming and locally-known commits.
With many local refs, this subprocess dominates the total cost even
though only a handful of new commits need to be identified.
Add a bespoke boundary walk in commit-reach.c that avoids spawning
rev-list entirely. This is based on the existing
paint_down_to_common() with some minor differences. The two sides
naturally represent the incoming tips and a subset of the old existing
tips. If the incoming tip side gets exhausted a valid boundary has
been found. The walk aborts and falls back to using rev-list if it
walks too far. This is a heuristic based on the commit-graph;
commits that exist in the commit-graph are typically reachable from
local tips. This is not required for correctness, since it is
always valid to fall back if the boundary cannot be proven.
Two things make this heuristic avoid work that would otherwise
slow it down:
1. For the happy path, most pushes have a nearby common
ancestor with one of the main local tips or the previously
seen tip. This is also the set of tips we use for walking,
instead of loading all tips -- which may be too many to
be efficient.
2. For the negative case where a boundary can not be found
this way, the walk quickly aborts as soon as it reaches
the commit-graph. Thus, the efficiency of this approach
depends on the commit-graph being somewhat up to date,
but it does not need to be perfectly up to date.
Gate this on the commit-graph having corrected commit dates (the
generation data v2 chunk), since v1 generation numbers can diverge
too far from topological depth. When no suitable commit-graph is
available, fall back to the rev-list path.
Benchmarks on a large repository (~3M commits, ~10K local refs),
1 new commit changing 1 file, measured with hyperfine --warmup 1:
rev-list: 1877 ms +/- 32 ms
incremental (rev-list boundary): 365 ms +/- 46 ms
incremental (paint boundary): 14 ms +/- 1 ms (134x faster)
Signed-off-by: Kristofer Karlsson <krka@spotify.com>
spkrka
force-pushed
the
tree-diff-connectivity-v1-clean
branch
from
September 1, 2026 18:03
326b1e4 to
452d6b9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This series implements the tree verification optimization described in
my recent RFC [1], gated behind transfer.connectivityCheck=incremental.
The current connectivity check uses a rev-list subprocess to perform
the object traversal. On repositories with large active trees this
can become expensive even for small fetches, because the tree/blob
closure at the connectivity boundary may be much larger than the
incoming change.
This series addresses the object-walk cost (step 2 from the RFC) by
introducing an incremental tree-diff approach. Incoming trees are
verified against their parents, recursively descending only into
entries that have changed. Parent trees still need to be
scanned as comparison bases. The boundary search (step 1) continues
to use rev-list; optimizing that is a natural follow-up.
When a tree entry has the same content in both
a new commit's tree and a trusted parent's tree, the
entire subtree is already verified. Only differing entries require
recursive verification. The verified set persists across commits, so
subtrees that have already been established as trusted can also be
reused across changes, reverts, moves, and merges.
Benchmarks on a large repository (~3M commits, ~200K trees and ~500K
blobs reachable from the tip). Scenario: 1 new commit, 1 file
changed, measured with hyperfine:
With ~10K local refs:
With 1 local ref:
In this benchmark the rev-list approach visits all ~700K tree and
blob objects while classifying the connectivity boundary. The
incremental verifier walks one new tree and checks one blob; the
corresponding parent tree is scanned as the comparison base. Tree
verification took 5 ms in both measurements.
With many local refs, the boundary search dominates the incremental
timings. Eliminating that subprocess and performing boundary discovery
in-process is the target for a follow-up.
On linux.git (~1.5M commits, ~100K reachable objects, ~1K refs):
On git.git (~80K commits, ~5K reachable objects, ~7K refs):
In these measurements, the benefit is largest on repositories with
large tree closures. linux.git's ~100K reachable objects produce
a clear 2.3x win, while git.git's ~5K objects show no measurable
difference.
Scaling with increasing numbers of incoming commits (git.git, no
alternates, single measurements using the RFC patch's in-process
boundary search):
In these measurements, the in-process paint walk falls back to the
rev-list boundary search at around 2000 incoming commits and above.
The implementation falls back to the rev-list path for deepening
fetches and repositories with active replacement objects.
One question around replacements: the existing rev-list connectivity
path follows replacement refs, while git prune explicitly disables
replacement refs before performing its reachability traversal. The
connectivity check ensures that refs do not point into incomplete
object graphs, while pruning ultimately operates on the underlying
object graph. It may therefore be worth discussing which replacement
semantics are intended here. I have left the existing behavior
unchanged in this series.
Open design point: the tree verifier introduces private _nofetch
variants of tree reading and tag peeling (read_tree_nofetch,
peel_to_non_tag_nofetch) to avoid triggering lazy promisor fetches
during verification. These duplicate logic from existing core
functions. Long term it would be cleaner to teach the existing
functions to accept a skip-fetch flag, but that touches many
callers and I would like mailing list feedback on whether that
direction is welcome before taking it on.
The series:
The RFC patch (sent separately as a reply to this cover letter)
shows that the boundary search can also be done in-process using
a paint_down_to_common()-like walk, reducing the total incremental
check from ~365 ms to ~14 ms on the large repository. It could
go in several directions (simplified, more generic with lazy ref
expansion, etc.) and is not meant to be merged as-is.
[1] https://lore.kernel.org/git/CAL71e4Nf=-zCrfN7ghEVGq11irajJhtdxYZgKe0Ycux0qs1ZvQ@mail.gmail.com/