Corgi-DDIR work - #795
Merged
frankmcsherry merged 77 commits intoJul 30, 2026
Merged
Conversation
…, value_id) A boundary where only integers cross: a storage backend presents each record as ((key_hash, value_id), time, diff) — integer proxies for data it keeps in its own layout — the operators own all the lattice/time logic over those integers, and the backend supplies value semantics via callbacks. Any columnar (or otherwise opaque-to-DD) value store can then reuse join and reduce without materializing values. - trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy columns, with from_unsorted (integer sort+consolidate with representative provenance) as the presentation-building helper. - operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the join_with_tactic and reduce_with_tactic seams (made pub here), and the backend traits: present-as-proxies (read), value callback with hash-minted output ids (write), materialize (egress). Reduce output ids are content hashes, so an output arrangement re-presents with the same ids downstream with no registry; pending interesting times are keyed by the stable key_hash across retires. The module doc carries the boundary contract and design notes (why value_id is not order-preserving; collision risk); each tactic and the in-memory reference backend (VecChunk arrangements, fnv hashes) sit in their own file under the module. - Tests: join and count/distinct/min reduces against the row operators over multi-round retracting inputs, and a scripted Product-time retire sequence exercising synthetic corrections and pending. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e, not a requirement value_id never outlives one computation (a join unit's presentations; one reduce retire, whose materialize resolves ids to real data before anything leaves). The actual contract is a per-computation bijection with value equality, plus within-retire agreement between the output presentation and minted ids. Content hashing discharges all of it statelessly (the reference backend's choice); exact schemes — dense ordinals from grouping, a per-retire value→id map — are equally valid and collision-free. Only key_hash must be a stable pure function of the key (cross-retire pending, changed-key filter), making the key side the irreducible collision exposure. Persisting ids into an output arrangement itself would force stable value ids; this design does not. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…worse than the cursor tactics Two regressions found by asking exactly that question, both fixed: - The reference reduce backend implemented the changed-key restriction by scanning every batch and filtering — O(trace) per retire where the cursor tactic seeks, O(delta·log). Presents now seek the changed keys (novel keys resolve from this retire's delta-sized input batches; pending keys from the retire that pended them, which is exactly what the persistent hash→key map retains — pruned to the changed set each retire, so it is bounded by the delta, and the per-retire value map is cleared). - The join interface had no restriction at all, forcing ANY backend to present the entire accumulated side per fresh batch — O(trace·log) per unit where the cursor join seeks. present0/present1 now take an optional sorted key-hash filter; the tactic presents the fresh side first and passes its key set for the accumulated side, the join analogue of reduce's changed-key restriction. The check is mechanical, not wall-clock: counting backend wrappers measure presented records — with 20k arranged keys and five single-key rounds, reduce presents 37 records and join 10, where the scanning versions present Ω(rounds·N) (join: 100,005) and fail the gate. What remains above the cursor tactics is a log-factor sort of delta-sized presentations, and per-(key, wave) rescans of a changed key's presented range where the row replayer consolidates progressively — same worst-case order, different constants. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cle fuzz The minimal harness for the framework, closed over itself: the tactics demand only BatchReader of the trace, and ProxyChunk is already a Chunk, so a batch of proxy chunks is the minimal arrangement — the proxy data IS the data, no separate chunk class needed. The identity backend makes values u64s with the identity as the id function: no hashing, no resolution machinery, no collision possibility, and materialize emits the proxy records verbatim (incidentally exercising ChunkBatch<ProxyChunk> as a real output batch). What remains under test is exactly the framework's own contribution — interesting-time discovery, desired-vs-current deltas, pending, held routing — fuzzed over Product-time grids: 300 random inputs retired through random diagonal frontiers (so synthetic joins arise inside and across intervals and must pend), driven through an emulation of the reduce driver protocol, and checked against a brute-force oracle at every grid point: the accumulated output must equal the reduction of the accumulated input, everywhere, for count/distinct/min-shaped reducers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An ignored benchmark (cargo test --release --test int_proxy -- --ignored --nocapture) compares full stacks — the stock row operators against chunk arrangements plus the proxy tactics over the reference backend — for a bulk load and a steady-state incremental phase (warmed past the post-load merge-amortization transient). The bench caught what the counting gates could not: the reference backend's hash→key map was pruned by retain (and the per-retire value map by clear), both of which keep the backing table — so after a million-key load, every retire walked a million-bucket table to visit one entry, ~130µs/round of pure capacity. shrink_to_fit after the prune and a fresh map per retire fix it. Steady-state single-key rounds after the fix, proxy vs row: reduce ~1.4x at every scale (flat from 10k to 1M keys — the delta- proportionality gates hold in wall clock too), join at parity below 1M (~1.0x). Bulk load carries the presentation layer's constant factor (per-record hashing, clones, by-hash sort): reduce 3-5x, join ~2x — the costs a columnar backend's bulk primitives are meant to attack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The longer-term goal for the boundary is fewer crossings into the backend's value logic (interpreted or columnar execution pays per-call overhead): each crossing should carry a list of keys and a longer bracketed list of value entries. ProxyReduceBackend gains reduce_many(keys, ends, input) — group_offsets -shaped brackets, one per key, each non-empty — returning concatenated per-key outputs with their own bracket ends. A default implementation loops the per-key reduce, so simple backends implement only that; backends with bulk value logic override reduce_many. The tactic now calls only reduce_many: retire's key-major loop becomes two passes — derive each changed key's active times, group the work into waves by time, then play the waves in ascending order (Ord extends the partial order, so a key's earlier deltas always precede a later time's reads) with at most one callback per wave, batching every key active at that time. The identity backend overrides reduce_many (asserting the bracket protocol from the backend's seat), so the grid-oracle fuzz exercises the batched path; the reference backend uses the default, so the row-comparison tests cover the loop. All gates and benches unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The desired output at a (key, time) moment is a function of the key's input accumulation at that time alone — no output-side state — so no time ordering constrains the batch: every moment of the retire can share one reduce_many call, a key contributing one bracket per active time. The order-sensitive part (subtracting the current output, which includes deltas emitted at earlier moments) is pure proxy-space arithmetic and moves to a separate pass that plays the moments in ascending time order. The bracket, not the key, is reduce_many's unit: keys may repeat. Contract docs updated accordingly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…linear The row suite's reduce_scaling/join_scaling shapes (one key, many distinct times, one batch) exposed quadratic behavior in both proxy tactics: reduce rescanned a key's full presented range per interesting time (and its interesting-time closure joined all pairs), and join cross-produced full matched histories pairwise. Measured: reduce 6.4s at scale 10k, 4.7x per doubling; join 9.8s at 10k, >90s at 20k. The robust versions are the cursor variants with integers in place of keys and values, as intended: - history.rs: IdHistory, the id-space ValueHistory — (value_id, time, diff) edits replayed in ascending time order into a buffer repeatedly advanced by the meet of the times still to come and consolidated. The advancement is the collapse that keeps a key with many distinct times linear: accumulations read the small buffer, never the raw history. The presentations serve as the fused per-key load. - reduce: discover_and_accumulate ports history_replay::compute — lazy interesting-time discovery (novel and pending seed; synthetics from joins with the advanced batch buffer and times_current) replaces the eager join-closure, and per-moment accumulation reads the advanced buffers. Phase B replays the output side per key over the discovered moments with the same machinery (suffix meets; emitted deltas advanced and consolidated). Still one batched reduce_many crossing per retire. - join: join_key ports JoinThinker::think — each side's edits replayed against the other side's advanced buffer (identical emitted times: t0 ∨ (t1 ∨ meet) = t0 ∨ t1), with the dead-simple cross product kept for small histories. proxy_reduce_scaling / proxy_join_scaling (scale 100k, the row tests' shapes) now pin this; scale 10k dropped from seconds to milliseconds and growth is ~4x per 4x scale. The grid-oracle fuzz over partially ordered times, the scripted pending test, the row comparisons, the delta-proportionality gates, and the steady-state bench all pass unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The row-vs-proxy comparison on the scaling shapes showed proxy reduce still superlinear (9x per 4x scale; timeout at 4M where row takes 0.5s). Profiling pinned it outside the tactic: the reference backend's materialize built ONE giant VecChunk and let the builder settle it, and settle's split path peels TARGET-sized pieces off the front with split_off, copying the remaining tail each iteration — O(m²/TARGET) in the batch size. Feeding the builder TARGET-sized chunks directly fixes it: 4M drops from >120s to 3.2s. With that, both operators are in the row implementations' complexity class on the scaling shapes (growth ~5x per 4x scale ≈ n·log n; the hot frames are the presentation sort and fnv hashing — constants, not structure): at scale 4M, join row 0.96s / proxy 3.6s, reduce row 0.48s / proxy 3.2s. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TimelyDataflow#778 Chunk split) A columnar differential-dataflow backend for DDIR whose native representation is corgi Value columns and whose scalar logic is corgi `eval_graph`, parallel to `backend::vec`. Arrangement: - CorgiChunk : Chunk (NOT NavigableChunk) — corgi columns + Vec time/diff, ordered (key,val) by corgi structural order then time. merge/extract/advance/ settle ported from the VecChunk reference; drives corgi's discrimination sort (sort_perm) + batched compare_idx, never per-pair. Rides TimelyDataflow#778's cursor-less Chunk path, so it gets the fueled/graded ChunkBatchMerger for free. - Tactics (Route B): cursor-less reduce (incremental key selection over the input delta); join via corgi `find` (find_ranges → equal-range merge-join, multi-record). as_collection reads columns directly. differential-dataflow proper: widen the JoinTactic/ReduceTactic/Fresh/ join_with_tactic/reduce_with_tactic seam from pub(crate) to pub, so an out-of-crate tactic can be implemented at all. (Candidate for upstreaming as the public extension point the TimelyDataflow#773 tactics were designed to be.) State: all 6 canonical programs match `vec`; 33 lib tests pass. reach ~1.9× slower than vec, compute-bound linear ~3× FASTER (columnar eval avoids the row backend's ~22% Value::cmp pointer-chasing). The remaining reach gap is entirely the reduce, still row-wise — the only operator not yet columnar. Notes / follow-ups: corgi_arrange.rs (old CorgiBatch impl) is superseded by corgi_chunk.rs and retained only as reference; scratch examples alongside the corgi_perf/corgi_progs/corgi_prof harnesses. Depends on frankmcsherry/wip corgi branch dd-arrange-api. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A standing harness that isolates each backend operator in a minimal DDIR
program, asserts corgi == vec, and reports the corgi/vec ratio against a
target. `PROG=<name> BACKEND=<corgi|vec>` loops one case for samply.
Baseline (n=100k linear / 20k arrange-y):
map8 0.30x ✓ (columnar compute, no row boundary)
filter 0.75x ✓
arrange 1.5x ✗ \
join 1.5x ✗ > every operator with a columns<->rows
reduce_distinct 2.0x ✗ / transcode boundary loses
reduce_count 2.0x ✗
reach 2.0x ✗
Corrects an earlier whole-program-profile conclusion ("only the reduce is
slow"): per-operator, arrange and join are independently ~1.5x — the shared
cost is the transcode boundary, not one operator. Beating vec means removing
row boundaries (chunk-native ingest, columnar operator emit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… vec
The columns↔rows transcodes at operator edges were self-inflicted, not
inherent. Removed all three:
- as_collection: read chunk columns straight into a CorgiContainer
(chunks_to_columns), no untranscode→from_updates round-trip.
- join emit: the projection already yields corgi columns; emit them via
give_container into a CapacityContainerBuilder<CorgiContainer> and drop the
JoinToCorgi unary — no untranscode + per-row give + re-transcode.
- arrange ingest: CorgiChunker (a ContainerBuilder) sort-consolidates each
input CorgiContainer's columns directly into CorgiChunks — replacing
ContainerChunker's drain-to-rows + VecMerger + row-based builder. It
ACCUMULATES to TARGET before consolidating, so it emits few large chunks,
not one tiny chunk per input container (else the columnar per-chunk set-up
dominates on many small batches).
Scorecard (corgi/vec): arrange 1.5x→0.9x (BEATS vec), join 1.5x→0.7x (BEATS
vec), reduce_distinct/count 2.0x→1.35x, reach 1.9x→1.45x. All 6 programs match
vec; 33 lib tests pass. Remaining reduce gap is the row-wise reducer (Rust over
Value rows) — next: columnar consolidate via corgi group→fold_add→filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
columnar_sum_by_key: the monoid Count/consolidate fold as pure corgi ops (group → map(fold_add) → filter via parse_ml + eval_graph), no Value rows. Signed diffs passed as raw two's-complement u64 bits; wrapping fold_add gives the correct i64 sum, `ne 0` the sign-agnostic zero-drop. Unit test: [(1,+1),(1,-1),(2,5),(3,9),(3,1)] -> [(2,5),(3,10)] (net-zero key dropped). Groundwork for the wave-based columnar reduce; not yet wired into the live reduce (which stays row-wise). Depends on corgi dd-arrange-api wrapping-Reduce. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…consolidation Count = Σ diffs ≤ t, and summing the consolidated per-value diffs equals summing the raw diffs — so for Count the consolidate_vals (a Value sort + clones) is pure waste. Sum diffs directly; no Value churn. reduce_count 1.39x→1.30x, all 6 programs match vec. Distinct/Min still consolidate per value (they reason about individual values) — that's the corgi two-level group→fold→filter path next; Collect stays row-wise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… tested) columnar_distinct_keys: the keys with a present value (net diff != 0, the nonzero simplification) as pure corgi ops — group by the (key,val) composite, fold_add raw diffs, drop net-zero, distinct keys of survivors. group-by-(key,val) is just group-by a projected composite; the nonzero test keeps everything on the raw-bit `ne 0` path (no signed encoding needed yet). Unit test: (1,10):+1,-1 → absent; (2,20),(2,21),(3,30) → present → keys [2,3]. Both monoid consolidate blocks now proven (sum for Count, two-level for Distinct). Remaining: the wave integration (feed many keys per wave, driven by the Rust time logic; reuse the existing delta/emit loop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…primitive Group a key column by identity, returning only integers: `perm` (row indices in group order) + `ends` (per-group exclusive end within perm). DD walks groups by index — group g is perm[ends[g-1]..ends[g]] — and drives its time/diff logic without ever seeing a key Value; keys and payloads stay columnar in corgi. This is the boundary model (corgi presents abstract int ids tied to times/diffs DD sees; DD hands back indices to include). Built on sort_perm + one batched compare_idx. Tested. Foundation for the path-2 columnar reduce (wave loop over group ranges). Note: cross-retire `pending` needs a STABLE key id (a u64 hash), since group indices are per-retire — converges with hashed-keys-as-u64. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oundary model Records: arrange & join now BEAT vec (transcode boundaries removed); multi-record primitives exposed; reduce consolidate blocks + group_offsets proven; and the design capture for a data-blind reduce tactic (framework presents as int-id/ time/diff, owns time navigation; backend supplies id-mapping + value callback; output ids are hashes — mint = hash the new value). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…flow#781) consumption plan Branch rebased onto int-proxy: the framework the SPIKE's boundary model called for now exists upstream (renamed ids -> int_proxy in review); the old in-branch copy is dropped. The hand-off section documents the current interfaces, the one-crossing-per-retire reduce_many contract, the assessment harness to replicate, the reduce-first plan, the traps already hit once (scan-presents, capacity leaks, giant-chunk settle), and the suitability questions the corgi agent should answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Implement ProxyReduceBackend for corgi and swap CorgiReduceTactic -> ProxyReduceTactic. The DD tactic owns all time/lattice logic over (key_hash, value_id, time, diff); the corgi backend supplies only: - hashing: key_hash/value_id via corgi::arrange::hash_rows over the corgi key/val COLUMNS (columnar, content-addressed; DD never hashes). - reduce_many: ONE crossing per retire over every (key,time) bracket. Count/Distinct are integer-only (no value untranscode); Min/Collect gather+untranscode the bracket values once, never per key. Output ids minted by hashing the produced value column (agrees with present_output). - present_input/present_output: read arrangement chunks, restrict to changed keys by columnar semijoin (novel whole, history filtered); delta-proportional. - materialize: resolve ids -> real (key,val) rows, seal a CorgiChunk batch. corgi_progs: all 6 match vec (scc min/recursion + unnest collect included). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the DValue resolution maps (HashMap<u64,DValue>) with corgi COLUMN pools + integer id->row-index maps. The real keys/values never leave corgi columns: - present_input/output register representative key/val columns into per-retire pools (key_index/val_index -> offsets into concatenated blocks), no untranscode. - reduce_many builds each reducer's output value COLUMN directly: Count -> u64 prim, Distinct -> Unit, Min -> gathered input rows (id reused from input), Collect -> List. Ids minted by hash_rows over the built column. No transcode. - materialize resolves proxy ids to columns by gather + columns_to_batch (new corgi_chunk helper), column-native egress (no from_rows transcode). The only residual untranscode is Min/Collect's value ORDERING (the reduction contract is DDIR Ord, which is not corgi's structural order); the chosen rows are still taken columnar. Scorecard corgi/vec: reduce_distinct 1.01x -> 0.92-0.96x (BEATS vec), reduce_count 1.03x -> 0.90-0.93x (BEATS vec), reach 1.10-1.22x -> 1.06-1.13x. corgi_progs: all 6 match vec; interactive lib tests green (36). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the last untranscode (Min/Collect value ordering) with one columnar corgi sort_blocks per retire: - Min: gather positive-diff candidates, segment by bracket, sort_blocks -> each bracket's argmin = perm[block_start]; output row taken columnar, reusing the input value_id. - Collect: segment entries by bracket, sort_blocks -> sorted run per bracket, expanded by diff into a List column. Uses corgi STRUCTURAL order, which equals DDIR Ord for the non-negative scalar/tuple values these reductions see (all 6 programs); diverges only for negative ints (unsigned leaf compare) and list-valued compares (length-first), neither of which arises here — documented as an order invariant. Needs corgi::arrange::sort_blocks (exposed on fm/corgi dd-arrange-api 9b41cdc, the segmented sort re-exported from ops::cmp::order). The reduce backend is now entirely transcode-free. corgi_progs: all 6 match vec; lib tests green (36); reduce still beats vec (distinct 0.92-0.95x, count 0.93-0.94x). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
key_index/val_index keys are already well-distributed hash_rows u64s, so siphash on every register/lookup was wasted (~7% of the reduce tactic in profiling). Pass the id through unchanged. reduce_count 0.93x->0.88x; distinct/reach within noise (their id-maps are small). Gate green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The proxy records sort by (key_hash, value_id, time); key_hash is a full 64-bit content hash, so one MSD counting pass on its high byte splits into 256 near-uniform buckets, each finished by a comparison sort on the full key. One linear pass replaces most of the n log n (falls back to the plain sort for n<512 or clustered hashes). Profiling: from_unsorted 20% -> 8% of the reduce, driftsort gone. Wall-clock ~flat (0.89->0.90x) — the reduce is allocation-bound (malloc 33-38%), so this just uncovers that as the next lever. Framework (int_proxy + reduce_reference) tests green; corgi_progs all 6 match vec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
retire allocated fresh Vecs + an IdHistory for EVERY changed key (58% of the reduce's Vec-growth in profiling, IdHistory another 17%). Hoist the transient buffers out of the per-key loops and clear/reload them (IdHistory::load already clears+refills, retaining capacity): rep, raw_moments (drain), and phase-B meets, out_replay, emitted. Pure capacity reuse, no semantic change. Scorecard: reduce_distinct/count 0.90->0.85-0.86x, reach 1.06->1.00x (parity) at n=4000. Allocation was the real lever (sort CPU wasn't). Framework tests (int_proxy + reduce_reference) green; corgi_progs all 6 match vec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…IdMap) Phase B allocated a fresh BTreeMap per (key, moment) for the desired-vs-current delta. Replace with one HashMap keyed by an identity hasher (value_ids are already u64 content hashes), hoisted above the loop and cleared per moment (retains capacity, no per-moment alloc/free). Iteration order is irrelevant — deltas are consolidated and the output batch is sorted by from_unsorted. Scorecard: reduce_count 0.86->0.85x, reach 1.06->0.98x at n=4000 (below parity). Framework tests (int_proxy + reduce_reference) green; corgi_progs all 6 == vec. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SCC is the sharp recursive case (nested fwd/bwd label propagation via min, negation, 4 joins). Findings: corgi is ~1.9-2.5x slower than vec and the gap GROWS with n (unlike reach, which reaches parity) — scc's nested-recursion / negation / min structure scales worse for corgi. And corgi DIVERGES from vec on larger random graphs (n>=1000: 1297 vs 1292) — a corgi-backend correctness bug (the proxy reduce tactic + reference backend pass the scc oracles in reduce_reference.rs, and it is NOT the min value-ordering — tested with a DValue-order argmin). Harness flags the mismatch instead of panicking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two debugging tools: - reduce_reference.rs: add reduce_int_proxy (int_proxy ProxyReduceTactic + Vec value backend over VecChunk, NO corgi) and scc_int_proxy_* tests that run SCC with only the reduce swapped to int_proxy, vs the cursor reduce. They PASS up to 50 nodes/120 edges/20 seeds — so the int_proxy reduce TACTIC is correct at SCC's nested-product-time depth. (reduce_reference's own scc_* compare cursor vs the *stock* reference tactic, never int_proxy — this fills that gap.) - interactive corgi_scc_min.rs: sweep+delta-debug an SCC corgi-vs-vec divergence to a minimal graph. Finds nodes=4, 5 edges [(0,0),(0,2),(1,3),(2,1),(2,2)]: corgi=3 vs vec=2. Conclusion: the divergence is in the CORGI backend, not the int_proxy work. The int_proxy tactic + Vec values reproduces SCC exactly; swapping in corgi is what breaks it. Next: narrow within corgi (reduce value backend vs join vs negate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ure) Four minimizer/trace harnesses (interactive examples): - corgi_scc_min: minimal SCC divergence = 4 nodes, [(0,0),(0,2),(1,3),(2,1),(2,2)]. - corgi_scc_trace: on that graph, export each SCC intermediate. Result: fwd::labels OK, trim_fwd OK, bwd::labels DIFF (node 2 -> label 1 in corgi vs 2 in vec), trim_bwd/scc DIFF. So the divergence enters at the backward min- reduce, structurally identical to the correct forward one. - corgi_cc_min: single-level fwd min-label propagation NEVER diverges (to 80 nodes) -> not single recursion. - corgi_bwd_min: two-pass fwd->trim->bwd with NO outer loop NEVER diverges -> not the chained nested scopes either. Conclusion: the bug REQUIRES SCC's outer recursion (the iterate wrapping fwd/bwd -> 3-deep product times, plus negation and feedback ). Symptom is a failed retraction (stale label survives). Suspects: corgi 3-level dynamic-time handling (leave_dynamic/results_in/enter_at) or negation under recursion. Not int_proxy (proven), not the reduce structure. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
corgi_scc_stream: emit the full (data,time,diff) stream of SCC intermediates via inspect, per backend; diff consolidated by (data,time) to find the first moment of divergence (DD is deterministic at all times). Result on the minimal 4-node graph: fwd::labels and trim_fwd streams AGREE exactly (the 16-vs-10 raw counts were +1/-1 that consolidate away). The bwd min-reduce: its INPUT (proposals+nodes) diverges only at inner-times 258/513, but its OUTPUT diverges at 257/512 — one tick earlier. So at time (0,[1,257]) the reduce input agrees while the output diverges (vec retracts node2->1, corgi does not). By the inputs-agree-output-differs test, the bug is IN the corgi min-reduce. The int_proxy tactic is proven on SCC (scc_int_proxy), so it's the CorgiReduceBackend (present/materialize/changed-key), not the tactic. The forward min-reduce (same code) is correct -> a data-specific edge case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Instrumented CorgiReduceBackend::present_input (env CORGI_DBG) to dump node-2's presented input on the minimal repro, and compared to vec's true input stream: for (node2, label1), vec has [0,257]+1 [0,258]+1 then [1,257]-1 [1,258]-1 (added round 0, retracted round 1 -> gone); corgi's present has [0,257]+1 [0,258]+1 [1,258]+1 -- the round-1 retractions are MISSING (plus a spurious +1). And corgi's input STREAM (bwdin) agrees with vec at [1,257] (both -1), so the retraction enters the arrange but present_input never returns it. => corgi loses a reduce-input update when arranging/presenting under nested times; the min then reads a non-retracted label1 and stays stale = 1. Not the min logic, not the int_proxy tactic (proven) -- the reduce-input arrangement (CorgiChunk) or present_input's changed-key/novel-history read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New native_scc_prof (fair enter_at compiled twin). Triptych at n=100k, self-time by module: allocator native 2.2% / vec 28.4% / corgi 27.2%; memmove native 0.7% / vec 2.8% / corgi 11.9%; timely progress tracking ~0.5% in BOTH backends. The 4.8x shared tax is BUFFER LIFECYCLE (allocation + copy churn), not progress tracking (kills lever 1b as a priority), not operator count (rung B), not edge time repr (rung A). vec's churn is per-row ir::Value boxes (cmp+clone+eq ~20% self); corgi's is column rebuilds (fresh Arc<Vec> per gather/merge/present). The suite ran on the SYSTEM allocator (only ddir_server/ddir_vec linked mimalloc). Adding mimalloc to all suite + prof binaries (one binary per benchmark: every column shares it; native columns move <5%, DDIR 12-36%): B1 250k: corgi 8.67 -> 7.07s (0.96x vec, 4.29x fair; was 5.2x fair) B2 150k: corgi 53.45 -> 42.94s (1.22x vec) 100k: 1.08x vec B3 1m: corgi 27.24 -> 19.79s (0.82x vec, 3.17x fair) B4 4m: corgi 20.09 -> 13.00s (0.45x vec, 2.96x fair) Gate 6/6 incl. fusion oracle; [checked] rows green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The pin moves from f556868 to the dd-arrange-api tip, which carries the four landed kernel additions (wip#7). Gate green; workspace passes -D warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
# Conflicts: # differential-dataflow/src/operators/int_proxy/history.rs # differential-dataflow/src/operators/int_proxy/join.rs # differential-dataflow/src/operators/int_proxy/reduce.rs # differential-dataflow/src/operators/mod.rs
Ten purpose-built binaries (four benchmarks + six profiling twins) collapse into one runner: interpreted programs live in examples/programs/*.ddp where they belong; the irreducibly compiled parts — input generators and the native twins, including the enter_at-fair SCC — register in the runner by bench name. BACKEND=/ITERS= select profiling mode (loop one column, a samply target), replacing the prof twins. Output formats, env protocols (N/SEED/CHECK/CHAIN), and all correctness assertions are unchanged; all four benches verified [checked] plus a prof-mode smoke; gate green; workspace passes -D warnings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eleven milestone/experiment binaries (rungs 0-4, M1-M4, Q3 of the backend build-up) whose questions are answered and whose coverage the survivors carry: the gate (corgi_progs) runs reach/scc and richer programs continuously, the scorecard owns per-operator triage, and ddir_bench owns end-to-end measurement and profiling. SPIKE.md narrates the arc they documented. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The .ddp extraction overwrote the pre-existing gate program with the benchmark's embedded SCC (different export shape, comments lost). The gate kept passing because the benchmark program is also a valid corgi==vec check — which is how it slipped through. scc.ddp is restored byte-identical; ddir_bench loads scc_bench.ddp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The five crate-root corgi_*.rs files become src/corgi/{container,chunk,join,
reduce,logic}.rs. No code change beyond the module paths: container is the
CorgiContainer on dataflow edges (was corgi_backend.rs), chunk the columnar
Chunk + arrangement plumbing, logic the DDIR-term compiler, join/reduce the
tactic bindings. backend/corgi.rs (the Backend impl, parallel to backend/vec.rs)
and col_times.rs (a generic columnar-time utility that imports no corgi) stay
where they are. Gate 6/6 both tactics; workspace passes -D warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Addresses review of the subsystem PR: - corgi_progs (the corgi==vec gate) moves from examples/ to a tests/ integration test (tests/corgi_backend.rs) — it is a gate, so its home is cargo test / CI, not examples/. Runs the six canonical .ddp programs, one #[test] each. - Remove the benchmark/measurement tooling that isn't a DDIR example: ddir_bench, corgi_scorecard, and the five benchmark-only programs (ast/ident/scc_bench/ scc_compound/scc_plain .ddp — the latter three the duplicate-SCC sprawl). These are dev tooling, not part of the library; they stay on the dev branch and can land separately if the later perf PRs want reproducible in-tree numbers. - Drop the working notes (SPIKE.md + two findings files, ~970 lines of markdown). examples/ is back to DDIR programs plus the pre-existing server/dump binaries. strongly_connected_at stays (a general algorithms/ addition, tested by tests/scc.rs). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
col_times exists to serve CorgiChunk (its own doc says so) and is used only by the corgi backend's modules — it belongs in src/corgi/, not at the crate root. No code change beyond the module path (crate::col_times -> crate::corgi::col_times). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drop the local-dev path note and the reference to a findings file that no longer exists; keep the git rev pin (explicit and reproducible) with a one-line purpose. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The header described an early milestone: Spine<Rc<CorgiBatch>> (the arrangement is a ChunkSpine<CorgiChunk> now), reduce 'awaits' a retire design (it is implemented via the tactics), and a 'this iteration ... = todo!()' note (nothing is todo). Rewrite it to describe the current, complete substrate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CC/Row/Upd/CTrace were defined ~110 lines below apply_ops, whose signature reads in terms of CC. Move them to just after the imports so the shorthands are in scope where the reader first meets them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Explain the empty enum: it is never a value, only a type carrying the Backend impl (selected via render_tree::<CorgiBackend>); the empty enum makes it unconstructable. Mirrors VecBackend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
strongly_connected_at was added for the benchmarks' fair-compiled SCC baseline; those benchmarks left this PR, orphaning it (tests/scc.rs was its only remaining user). Its scc_at tests exposed a PRE-EXISTING nondeterminism in propagate_at/ propagate_core (the rotated reduce_abelian propagation): scc_at_10_20_1000 is flaky under 3 workers (the plain scc test, which uses a local iterate+enter_at reachability, is deterministic). Rather than carry an orphaned API whose tests surface someone else's bug, revert scc.rs + tests/scc.rs to master-next. The API can return with the benchmark tooling once propagate_core's determinism is fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frankmcsherry
added a commit
to frankmcsherry/differential-dataflow
that referenced
this pull request
Jul 23, 2026
This PR's role is the protocol substrate for the Corgi/DDIR work (TimelyDataflow#795): the reshaped int-proxy tactics that its columnar backends port onto. Under that lens the SCC example's job — in-dataflow validation against a real backend — is superseded by the Corgi backend's differential gate (whole .ddp programs vs the vec reference), which is the stronger version of the same check. The example's findings stay on the record in its commits (1bc0752, ed19240): proxy joins at parity with join_core end-to-end, oracle-validated including under the TimelyDataflow#801 trigger shape at the regression suite's sizes. What remains in-tree is the harness's own property suite (tests/int_proxy.rs), which guards the contract the Corgi port will code against and is the one thing the downstream gate does not cover. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frankmcsherry
added a commit
to frankmcsherry/differential-dataflow
that referenced
this pull request
Jul 23, 2026
Two items from downstream (TimelyDataflow#795) review of the substrate: * retire's final zip would silently truncate a backend returning the wrong number of batches from finish(), shipping batches at the wrong held times. Now a hard assert (one batch per tile description), consistent with the harness's fail-loudly posture. * join_key's >=16x>=16 bilinear_wave path had no in-tree exercise — every key in the property suite was small, and the SCC example that covered the path was pruned. join_wave_path_matches_naive builds one deliberately fat key (20 presented records per side: spread-time insertions plus non-cancelling retractions, so consolidation keeps all of them) alongside a small key, so both join_key paths run in one unit, against the naive oracle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frankmcsherry
added a commit
to frankmcsherry/differential-dataflow
that referenced
this pull request
Jul 29, 2026
The last place the seam allocated per call. Every other buffer at the boundary is either caller-owned and appended to (produce's `out`) or round-tripped for reuse (next_window's `reuse`), and the harness's own staging is hoisted outside the window loop and cleared rather than reallocated. reduce_corrections still returned two fresh Vecs, once per round per window. Now it appends to `corrections` and `ends`, which the harness clears and keeps the capacity of. Both call sites consumed the result immediately and copied out of it, so nothing else changed. The one-end-per-key rule was already relied on by the indexing that follows; it is now stated in the doc and asserted, so a miscounting backend fails with a sentence rather than an index panic. Not expected to be measurable — the allocations are small against the data they carry. The point is that a backend author reading the two tactics now sees one convention for bulk calls instead of two, and the TimelyDataflow#795 port pays for the change once rather than twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MHrUxzFy5KKgcZMe5jrCJd
… tactic CorgiJoinBackend implements ProxyJoinBackend (TimelyDataflow#809): advance draws blocks of the key intersection as ((key, coordinate), time, diff) bridges; cross redeems matched coordinates directly against the instance's chunks (gather_lanes) and cuts TARGET_OUT-sized containers. run_unit and its whole-unit staging (the large-snapshot OOM) are gone; peak state is one block plus one container. Value tokens are canonical coordinates: the per-key merge across chunks gives equal values the coordinate of their least occurrence, so bridge consolidation cancels cross-batch churn (tokens are the unit of cancellation), while values themselves are never copied into the presentation — they are gathered once, from chunk storage into the projection input, for matched rows only. Group tokens are the key's own u64 when the key column flattens to a single 64-bit leaf lane (what DDIR Int keys and 1-tuples transcode to): chunk order is u64 order, so blocks resume by seeking from. Two regimes, mirroring the bespoke tactic's probe heuristic: a much-smaller side drives and the other is probed at the driver's keys (batched find_ranges); comparable sides are pulled and merged symmetrically (probing would cost n log n against the merge's n). Multi-lane keys fall back to a structural single-block walk (ordinal tokens). Gate green 6/6, debug (harness asserts live) and release. Bench vs the bespoke tactic: scc 3.71s vs 3.77s; ident@1M 21.79s vs 22.76s (0.92x vec); ast 0.83- 0.92x vec; scc-compound (2-lane keys, the structural fallback) 50.4s vs 46.5s, a known follow-up: the probe/merge split applies there too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
frankmcsherry
force-pushed
the
corgi-on-mn
branch
from
July 30, 2026 12:56
cbf7854 to
3236017
Compare
The bespoke tactic's presentation layer (SortedRun, flatten_batches, flatten_restricted) lost its last caller when run_unit was deleted, and the DrainContainer row path (containers drained to rows for a reused row-wise MergeBatcher) was never live: the corgi pipeline's one arrangement path is chunk-native (CorgiChunker/ChunkBatcher). sort_consolidate and concat_blocks stay — from_columns and the chunker still use them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hot-path TODOs as_collection emits one container per chunk — key/val columns clone by Arc bump, times materialize (the owned-time egress), diffs memcpy — instead of gather-concatenating every batch's chunks into one container; the concat bought nothing downstream, and chunks_to_columns goes with it. The vestigial row-update alias (Upd) leaves chunk.rs, taking the layer's last row-type import with it: nothing under src/corgi/ mentions DValue except logic.rs's transcode boundary. Comment hygiene: history-of-the-work narration (superseded implementations, reverted experiments, rung/phase references, stale type and document names) gives way to present-tense constraints; the fallback comments now say accurately that the gap is unwritten lowering in logic.rs, not expressiveness in corgi (which models sums and lists). Two TODOs mark the complete hot-path row-at-a-time residue: merge() awaits survey_groups (group-range Both; newer corgi revs export it) for a batched rewrite, and collect_present can memoize key hashes, batch its membership test, and move kept ranges via push_range. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nce debt Cross-reference CorgiContainer and the chunk's Inner as one payload that should become one type: the differences (row-mutable Vec<T> times on edges, sorted/consolidated/shared in the trace) each name the artifact that removes them, with a bulk-mutation time container dissolving the last. from_updates/ into_updates likewise declare their intended end-state — external ingest and inspection edges only — and each in-dataflow fallback caller names what retires it (the Case/Inject/List/Unary/Hash lowerings, bulk time mutation, columnar append, corgi list ops). Row-column round-trips inside the dataflow are debt, not design. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.