Record reproducible LDBC measurements and optimization gaps - #875
Draft
frankmcsherry wants to merge 3 commits into
Draft
frankmcsherry wants to merge 3 commits into
frankmcsherry wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The LDBC suite needs a reproducible performance baseline and a record of optimization opportunities that another session can refresh. Add
CURRENT.mdwith measurements of all 41 queries on both backends,GAPS.mdwith bounded investigation tasks, and exact data/build/run instructions inDATA.md,REFRESH.md, and the build receipt.The baseline uses unmodified master-next
229508ddon an M4 mini (16 GiB, four workers): three isolated SF0.003 catalogue trials, three concurrent IS1/IS3/IC11/BI11/BI18 trials, and one all-query tiny check. Seven complete compressed reports (~1.8 MB), their checksums, resolved Cargo.lock, and resource receipts are included. Query definitions, the timed runner, engine code and dependency pins are unchanged.readout.pyreports complete recorded cycles, setup and final retirement separately, and summarizes compatible repeated trials.compare.pyalso accepts compressed reports. The recipes document the fixed hash seed required by the current default parameter selection. Anyone can replace the readout and revise a gap assessment using new validated evidence; no growing historical archive or session-specific query driver is required.These are small-data lifecycle measurements, not official LDBC scores, SF1 capacity measurements, or 30-binding maintenance throughput. Each BI query has one standing binding, and the small update stream leaves many results unchanged. The gap register distinguishes observed code behavior from hypotheses about elapsed time.
Validation: all seven runs passed (8,818 same-plan answer checks including setup/warmup); repeated-trial comparisons passed for both timing recipes; the eight Python harness/catalogue tests passed; report checksums and documentation links verified. All runs stayed below the 2-GiB combined process-group cap with zero swap use. The concurrent panel's median full cycle was 10.093 ms on Vec and 11.777 ms on Corgi.