feat(conformance): engine conformance harness + adjudicated TiDB scoreboard (slice 1)#371
Merged
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ardening Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Walks corpus/tidb/pkg/parser/*_test.go and extracts every []testCase{src,
ok, restore} composite literal into CorpusEntry rows (positional and keyed
forms, raw-string and "a"+"b" concatenated literals). Non-literal srcs
skip with a reason instead of crashing. Short-gated smoke test guards
extraction health against the fetched corpus: 3838 entries (3192 accept /
645 reject / 1 skip), matching a whole-file AST cross-check of all 55
tables.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rows Non-composite []testCase elements now emit a provenance-carrying non_composite_element skip entry instead of vanishing, and the smoke test gains an independent flat whole-file element count (sees package-level tables too) asserted equal to the extracted-entry count — the review's out-of-band zero-loss check is now a regression gate for corpus tag bumps. Fixture adds the two most fidelity-sensitive literal shapes (interpreted escapes incl. \r\n and escaped quotes; multi-line raw string) with exact-bytes assertions, plus a bare-identifier element row. Documents the name-only syntactic match assumption on isTestCaseSlice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
omniTiDBVerdict runs one statement through the omni tidb parser and
maps the result to a Verdict. Panics are recovered into a reject with a
PANIC: marker — corpus statements must never crash the sweep, and
panics are themselves findings.
go mod tidy wires github.com/bytebase/omni via the existing replace;
the harness build compiles only tidb/{parser,ast} + stdlib (the go.sum
entries beyond that are sums for tidb/parser's testcontainers-based
test imports, not build deps).
Full-corpus containment probe (throwaway, deleted): 3838 entries =
2193 accept / 1644 reject / 0 PANIC / 1 skipped(no sql).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…st main First committed TiDB scoreboard (upstream lane, pre-adjudication). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…edup conflict guard - clusterRows: exemplar tie-break made total (shortest SQL, then lexicographic; first-row sentinel via Count==1 so an empty SQL cannot be stolen) — rendering no longer depends on row order - clusterRows: skip non-upstream lanes, matching the headline counts - buildRows: dropped duplicate with a conflicting upstream label flips the kept row to INDETERMINATE (duplicate_label_conflict); counted in run meta as duplicate_label_conflicts (zero in this corpus) - scoreboard: rename duplicate_count -> duplicates_dropped (wire name), add duplicate_label_conflicts line - truncate: rune-safe (back off to a rune boundary) - writeJSONL: buffered writer, flush + explicit close error propagation Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renamed counts lines, duplicate_label_conflicts (0), and 3 tied-cluster exemplars now resolved by the total tie-break. Counts unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
-adjudicate probes every non-agreeing row (GAP/OVER/INDETERMINATE) against the pinned pingcap/tidb:v8.5.5 container and reclassifies with the container as ground truth. Agreeing rows are untouched. Verdict lattice grounded in the vendored v8.5.5 parser source and verified against the live oracle: parse-reject = 1064/1149 plus the grammar-action and ast-validator abort codes (partition shape errors, ErrWrongUsage, ...) that TiDB emits for parse-time rejections with MySQL-compatible specific codes. Runtime collisions on those codes fail closed into INDETERMINATE label_container_disagree. Hazards handled: - DSN is normalized to multiStatements=true (multi-statement corpus rows would otherwise false-reject the whole batch) with bounded dial/read/write timeouts; a driver timeout is a non-MySQL error and lands fail-closed in INDETERMINATE infra_error. - Unsafe statements (first keyword SHUTDOWN/KILL/RESTART; the corpus literally contains them) are marked unsafe_to_adjudicate and never touch the container; if the container dies anyway, the sweep aborts naming the current and previous statement instead of poisoning the remaining rows. - duplicate_label_conflict rows drop their arbitrary first-seen label before adjudication so the container becomes sole truth. - Every row runs on a fresh connection (no idle reuse), so session state such as SET sql_mode cannot leak into later parse verdicts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Container-adjudicated run against pingcap/tidb:v8.5.5 (digest in meta): 1300 divergence rows probed in 8s. GAP 1141 -> 1133, OVER 159 -> 158, INDETERMINATE 0 -> 9 (3 unsafe_to_adjudicate KILL variants, 4 infra_error driver-side LOCAL-file rows, 2 label_container_disagree). OVER clusters collapse 101 -> 19 pre-lattice-fix, 45 after engine- message re-keying. Fresh-container reruns are class-deterministic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- adjudicateTiDB was at cognitive complexity 19 (house gate <=15) and is the clone-seam for the next engines: extract probeRow (Exec + infra fold + container-death abort) and adjudicationCandidates, invert the prepare guard to an early continue. Behavior identical; existing tests unchanged and green. - start_tidb.sh: export prefix on the TIDB_CONTAINER_DIGEST line (the documented copy-paste flow otherwise yields an unexported var and the scoreboard silently loses its provenance digest), and fail loudly if TiDB never listens within 60s. - Lattice comment: note the fail-closed collision net requires an upstream label (H3-cleared rows classify from the container alone). - Test pins: mid-batch "select 1; shutdown" stays first-statement-only (container-death abort is the backstop); 1102 lattice membership. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g engines) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1cd50d6d9a
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…probes
Two extraction/safety fixes from PR review:
- The extractor matched only []testCase{...} slice literals; the 12 bare
testCase{...} composite literals in append(...) calls were silently
omitted, violating the zero-loss invariant (a future labeled append-form
row would be lost without a SKIP trace). Add the bare-literal arm to the
extractor and mirror it in the countTestCaseElements cross-check; smoke
moves 3838->3850 entries, 1->13 skips (all 12 new rows are fmt.Sprintf
srcs, so non_literal SKIP).
- Extend the unsafe-to-adjudicate predicate with a SET PASSWORD arm
(first keyword SET + second keyword PASSWORD): the corpus contains
SET PASSWORD statements (parser_test.go:1386-1387), and probing one as
root would change the oracle's credentials mid-sweep. Other SET forms
stay adjudicable; ping-abort + disposable container remain the backstop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
go 1.26 -> go 1.25.7, exactly the omni root module: the harness uses no 1.26 features, and the higher directive blocked GOTOOLCHAIN=local users on the repo's 1.25 toolchain. go mod tidy -diff is empty. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The committed board recorded an intermediate branch commit as omni_sha — unreachable from main after a squash-merge. This branch never touches the parser, so the parser under test is exactly the merge-base. Document the rule (committed boards record a main-reachable omni_sha; harness-only branches use `git merge-base origin/main HEAD`) and regenerate the adjudicated board at f7991e5. Board delta vs the previous adjudicated run: SKIP 1 -> 13 and total 3781 -> 3793 (the 12 newly-captured append-form rows, all non_literal SKIPs); GAP/OVER/AGREE/INDETERMINATE counts and every cluster table are byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 78d6a96c92
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…scan, schemaless DSN, loopback bind Codex round 2. Connection-scope errors are not parse verdicts: a probed batch that mutates credentials (SELECT 1; SET PASSWORD) leaves the container alive, so the ping-abort never fires — instead every later fresh connection fails its handshake with 1045, which the classifier mapped to VerdictAccept: silent verdict poisoning. - classifier: 1045 -> VerdictNone (infra), routed to probeRow's ping-abort which names the culprit statement; 1049/1046/1044 stay statement-level "parsed" (test-pinned, reasoning in comments) - unsafeToAdjudicate: scan every statement in the batch, not just the first; the naive `;` split can only over-match (documented false-positive direction), never miss - SET arm: also unsafe on GLOBAL / PERSIST / @@global... / @@persist... (parse-affecting for every later probe, e.g. sql_mode ANSI_QUOTES); NAMES / sql_mode / @@session.x / @v stay adjudicable - DSN: no default schema (droppable by adjudicated DDL -> handshake 1049 poison); normalizeTiDBDSN forces DBName empty on user DSNs - start_tidb.sh: bind 127.0.0.1 only (passwordless root) Live-verified against the container: USE nonexistent -> 1049, unqualified name -> 1046, both classify parsed; full adjudicated re-sweep at merge-base f7991e5 regenerates the committed board byte-identical (only gitignored JSONL raw codes shift 1146 -> 1046). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7c8a4db59c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…obes Two unsafe-scan escapes, both container-verified on the pinned oracle: - /*! SET PASSWORD = 'x' */ and /*!40101 SET GLOBAL ... */ are EXECUTED by TiDB/MySQL but were stripped as comments before the deny-list ran. Executable-comment markers and their matching */ closers are now blanked first, so the content survives as scannable SQL; ordinary /* ... */ comments still strip. - CREATE/ALTER/DROP/RENAME USER can rewrite the identity every later fresh connection handshakes with (ALTER USER root IDENTIFIED BY 'x' verified: all later handshakes fail 1045 while the container stays up, so the ping-abort backstop never fires). GRANT/REVOKE stay adjudicable: verified unable to touch connection identity (user auto-create refused with 1410; GRANT ... IDENTIFIED BY on an existing user leaves credentials untouched). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Task 4 deferred the two []testErrMsgCase tables in parser_test.go (:5185, :5191); promote them. Semantics grounded in the corpus's RunErrMsgTest: a non-nil err demands terror.ErrorEqual against a real parse error (parse-reject), a nil err demands NoError (parse-accept) — the err value only pins the message upstream, so its non-nil-ness is the verdict. The extractor gains a third match arm (keyed and positional; nil ident or omitted keyed err = accept, any other expression = reject; non-literal src = provenance-carrying SKIP), and the zero-loss cross-check counts the same arm. Smoke movement: 3850 -> 3856 entries (+3 accepts, +3 rejects, skips unchanged): commentMsgCases 4 rows (1 accept / 3 rejects), funcCallMsgCases 2 rows (2 accepts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
harness/conformance is its own Go module, so the root go test ./... never compiles it — a compile break there shipped silently. One -short step after the root tests: hermetic (no corpus, no Docker) and fast; the container lane stays out of CI per the design doc. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full adjudicated sweep against the pinned oracle (1302 rows probed). Movement vs the previous board: - testErrMsgCase harvest (+6 rows, 1 duplicate): +3 AGREE_ACCEPT, +2 OVER — both the same omni bug class, over-accepting an unterminated /* block comment (container 1064); one joins the existing DELETE cluster, one opens OVER cluster 46 (`select 1/*`). - User-account DCL deny-list: 20 GAP rows (17 CREATE USER + 3 ALTER USER) now INDETERMINATE unsafe_to_adjudicate instead of being executed against the oracle; the 4 GRANT ... IDENTIFIED BY rows correctly stay adjudicable (GAP). - Executable-comment scan moved no corpus rows — prophylactic only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bc0ac2c4f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…mments) unsafeToAdjudicate's pre-pass was not comment-robust (one root cause, three bypasses): it neutralized only /*! executable markers while omni's own splitter (tidb/parser/split.go, Segment.Empty) also treats /*T! as executable TiDB SQL, and it stripped only LEADING ordinary comments while the server treats comments as whitespace anywhere — so /*T! SET GLOBAL ... */, SET /*c*/ PASSWORD = 'x', and CREATE /*c*/ USER ... all tokenized past the deny-list. Restructure into one normalization pipeline: (1) neutralize executable markers — /*! and /*T!, both grounded in split.go, plus optional version digits — keeping their content scannable; (2) strip ALL remaining ordinary comments (/* */ anywhere, -- and # to EOL) to a single space, skipping string literals and backtick identifiers so a comment opener inside a literal cannot swallow the real statements that follow; (3) THEN split on ; and scan each statement — so a ; inside an ordinary comment no longer fabricates a phantom segment, and commented-out unsafe text is correctly not flagged (the server never executes it). The now- redundant leading-comment strip in unsafeStatement is removed (classifyFamily's own leadingComment use is untouched). Board impact: zero — old-vs-new diff over all 3799 adjudicated rows shows no unsafe-status flips (the 29 corpus /*T! rows carry only DDL attributes), so the fix is purely prophylactic and the committed scoreboard stands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a999786114
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Round-5 review fixes for the unsafe-statement scan, each grounded on the pinned oracle (corpus v8.5.5 scanner source + live container probes): - /*T![feature_id,...] bracket groups are part of the executable-comment marker (scanFeatureIDs, corpus lexer.go:972-1015): blank a well-formed group along with the marker so the gated content - which the server executes, even when the comment is unterminated - reaches the keyword scan. Malformed groups stay visible instead: the oracle re-scans them as content whose leading bracket junk parse-errors the whole batch, so nothing in them executes. /*! takes only version digits, never a bracket group (corpus lexer.go:530-535). - Nested block comments: refuted against the oracle and pinned instead of changed. TiDB's scanner ends every block comment at the FIRST */ with no depth counter (corpus lexer.go:578-600; container-verified: the text after the first */ of an unterminated nested comment executes). omni's own lexer/splitter DO nest - an omni-vs-TiDB divergence the board measures; a nesting-aware strip would mirror omni instead of the oracle and wave a live KILL through. - SET CONFIG joins unsafeSetTarget: it mutates dynamic cluster/component configuration, which outlives the probe's session. The arm keys on the CONFIG keyword because the target may be a string literal. - probeRow: pin why whole-batch Exec is verdict-correct for multi-statement rows - TiDB parses the FULL batch before executing anything (verified live: CREATE TABLE t(a int); SELECT FROM -> 1064, not 1046, and the CREATE provably never executes). Board impact: the six upstream SET CONFIG rows flip GAP -> INDETERMINATE unsafe_to_adjudicate; no other flips. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pped Full adjudicated re-sweep against the pinned oracle at merge-base omni_sha. GAP 1113 -> 1107, INDETERMINATE 29 -> 35: the six SET CONFIG rows leave the probe path (unsafe_to_adjudicate) and the SET GAP cluster shrinks 14 -> 8. All other counts byte-identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 89869a5b06
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
…eutralizer, feature-gate union, multi-assignment SET) QUERY WATCH ADD/REMOVE persists runaway-query watch rules server-side; an ACTION KILL rule can kill later probes matching the watched SQL text/digest, so the sweep becomes order-dependent (container stays up, ping-abort never fires). Block the arm (first keyword QUERY + second WATCH). Make neutralizeExecutableComments string-aware: a `/*` inside a string literal is data, so it must skip quoted regions with the same skipQuoted helper stripComments uses. Otherwise `SELECT '/*'; /*! SET PASSWORD='x' */` reads the in-string `/*` as an ordinary comment and jumps past the real `/*!` marker, which stripComments then removes as ordinary -> the SET executes unseen. Feature-gate union scan: TiDB ignores the whole `/*T![feature] ... */` block when the feature id is unsupported and executes the SQL that follows. Without the version feature table, scan BOTH interpretations (content-visible and content-stripped) when a `/*T![` marker is present and flag unsafe if either trips. Plain `/*!`/`/*T!` digit forms execute unconditionally, so their single view stays exact. Multi-assignment SET: scan ALL comma-separated targets, not just the first, so `SET @v=1, @@GLOBAL.sql_mode='ANSI_QUOTES'` is caught. Comma split is naive (a comma inside a quoted value over-matches into a conservative skip). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
adjudicateTiDB now snapshots parse-affecting global state at sweep start and re-verifies it at sweep end (verifyNoDrift). Drift means a probed statement bypassed the deny-list and mutated shared state, so the board would be order-dependent: the sweep fails loudly, naming the drifted variable (before -> after) and instructing container recreation, instead of committing a possibly poisoned board. main.go already fatals on the adjudicateTiDB error before any JSONL/board write, so a tripped canary leaves no partial artifact. Snapshot list (all six confirmed present on v8.5.5): @@global.sql_mode, character_set_server, collation_server, default_week_format, sql_require_primary_key, tidb_skip_isolation_level_check — small and skewed to state that changes how a later row parses or what is accepted. An auth canary (fresh Ping) proves the root credentials still authenticate after the sweep. The deny-list is best-effort enumeration; the canary + 1045-abort + schemaless DSN + loopback bind + disposable container are the guarantees — a bypass is a detected loud failure, not silent corruption (README). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipped Regenerated on a fresh pinned container after the round-6 deny-list arms. The 14 QUERY WATCH GAP rows move from GAP to INDETERMINATE unsafe_to_adjudicate (GAP 1107 -> 1093, INDETERMINATE 35 -> 49); the 4 AGREE_REJECT QUERY WATCH rows are non-candidates and unchanged. Canary reported no drift end to end. omni_sha and container_digest unchanged (merge-base == committed sha). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…rites, trailing newline Codex round 7 (reporting accuracy + hygiene): - Scoreboard surfaces unexpanded generator sites: non_literal SKIP rows are loop-built runtime srcs each standing for multiple upstream cases, so the committed totals measure the static literal corpus, not full runtime-upstream coverage (expansion = static mini-interpreter, out of scope). README + PR body scope the zero-loss claim the same way. - Label-only runs no longer overwrite the committed adjudicated board: the board lands in gitignored out/tidb.scoreboard.md unless -adjudicate or the new -write-scoreboard (intentional label-only baseline) is set; decision pinned by TestWriteCommittedBoard. - renderScoreboard ends with exactly one trailing newline (git diff --check flagged a blank line at EOF on the committed board). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Full adjudicated sweep on a fresh pinned container (merge-base omni_sha f7991e5, digest unchanged): picks up the generator-site line (13) and the single-trailing-newline render. All counts and clusters byte-identical to the previous board — zero movement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 39add1599f
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Name the harvested shapes exactly ({src, ok} testCase / {src, err}
testErrMsgCase literals) in README and the extractor's top comment, and
enumerate the documented out-of-scope static shapes ([]string tables
labeled by consuming assertions, RunRestoreTest-style pairs, inline
Parse calls) as follow-up harvest candidates for the next corpus-tag
re-baseline. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9e720e9d7e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+372
to
+373
| c := s[i+2] | ||
| return c == ' ' || c == '\t' || c == '\n' || c == '\r' |
There was a problem hiding this comment.
When a row has -- followed by vertical-tab/form-feed before a newline, the pinned TiDB scanner treats that as a line comment because it uses unicode.IsSpace after the dashes, but this helper only recognizes four whitespace bytes. A candidate like --\v\nSET PASSWORD='x' is therefore considered safe here because the statement starts with -, while TiDB skips the comment and executes the password-changing SET, poisoning later fresh connections; mirror the oracle whitespace set before the unsafe scan.
Useful? React with 👍 / 👎.
| second, _ := nextKeyword(rest) | ||
| return second == "WATCH" | ||
| } | ||
| return false |
There was a problem hiding this comment.
When a divergent batch prepares and executes dynamic SQL in the same fresh session, neither PREPARE nor EXECUTE is treated as unsafe here, so a row such as SET @s='SET GLOBAL sql_mode="ANSI_QUOTES"'; PREPARE p FROM @s; EXECUTE p reaches the shared oracle and mutates the global parse mode while the scanner only sees safe top-level statements. That can make the following fresh-session probes order-dependent before the end canary trips, or evade it if the batch restores the value; reject PREPARE/EXECUTE batches or inspect the prepared text/session variable flow.
Useful? React with 👍 / 👎.
| } | ||
| sort.Slice(rows, func(i, j int) bool { return rows[i].StmtHash < rows[j].StmtHash }) | ||
| for i := range rows { | ||
| if err := enc.Encode(&rows[i]); err != nil { |
There was a problem hiding this comment.
When a corpus literal contains raw non-UTF-8 bytes, such as the pinned TiDB row select 111 as \xd6\xf7, encoding/json coerces the sql string to valid UTF-8 and writes replacement runes. The stmt_hash was computed over the original bytes and omni/container verdicts were produced from those bytes, so the JSONL artifact no longer contains a re-runnable statement and offline reclassification from the committed artifact can diverge; encode raw SQL bytes losslessly, for example with an escaped/base64 companion field.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Manager directive: full compatibility across engines, proactive rather than customer-request-driven — so we measure first. This PR lands the measuring instrument and the first measurement. The bar it tracks: GAP = 0 on the upstream lane at pinned engine versions (omni never falsely rejects SQL the engine accepts); OVER (over-accepts) is triaged, not chased; INDETERMINATE is surfaced, never silently dropped. Full design:
2026-07-07-omni-full-compatibility-conformance-design.mdin the workspace plans directory (not in this repo).What
harness/conformance/— a standalone Go module (replace => ../.., so it always sweeps the working-tree omni):fetch_corpus.shsparse-clonespingcap/tidbatv8.5.5into a gitignored dir (fetch-don't-vendor).go/astextractor harvests every pre-labeled{src, ok}testCase literal from the upstream parser tests — 3,838 statements, zero-loss over self-labeled test tables ({src, ok}/{src, err}; elements it cannot read statically become SKIP rows with provenance instead of vanishing; loop-generated runtime cases appear as unexpanded generator-site SKIP rows, counted on the board).-adjudicateprobes every non-agreeing row against a live pinned TiDB container: fresh session per row (session state would change how later statements parse); SHUTDOWN/KILL/RESTART variants never reach the shared oracle.scoreboards/tidb.md— its git history is the progress report) + gitignored full-provenance JSONL (out/tidb.jsonl, meta line first; the JSONL retains raw codes/messages/labels, so reclassification needs no container re-run).CI-invisible: the sweep is a binary, not a test; in-repo tests are unit tests plus a
-short-gated corpus smoke. Omni CI never touches Docker.First adjudicated TiDB scoreboard
3,838 harvested → 57 duplicates dropped → 3,781 statements, all divergences container-adjudicated:
Top GAP clusters (full ranked burn-down list = the GAP table in
scoreboards/tidb.md):trace beginSELECT SCHEMA();admin show ddl;create sequence seqALTER TABLE db.t RENAME db1.t1Caveat: the largest clusters are coarse family buckets, not single grammar items — omni's uniform "syntax error at or near …" text under-splits distinct gaps, so 71 clusters is a lower bound on distinct grammar work items; the JSONL supports finer slicing when scoping burn-down PRs.
INDETERMINATE 9 = 3 KILL variants (unsafe to execute against the shared oracle), 1 LOAD STATS + 3 PLAN REPLAYER local-file infra refusals (not parse verdicts), 2 label-vs-container disagreements.
Verdict lattice, not 1064-only
TiDB rejects at parse time with 26 distinct error codes, enumerated from the parser source in the pinned v8.5.5 corpus checkout (yacc 1064/1149,
AppendError/return-1 grammar-action aborts, ast-validator codes) and verified by live PREPARE-oracle probes. 47 label-reject rows carry non-1064 parse-abort codes — 44 through the specific-code arms, 3 through 1149. A 1064-only classifier would have mis-queued all 47 as label/container disagreements. Codes that also occur at runtime fail closed to INDETERMINATE, never a silently wrong class.Reviews
Every task went through a two-stage review (spec compliance + code quality) with container-grounded verification; determinism proven by byte-identical scoreboard re-sweeps against fresh containers.
Next
sql_mode/.optserver flags /ANSI_QUOTESchanges lexing);--errorannotations give a labeled reject-space.[]stringparser tables (TestTableSample-class — labels live in consuming assertions, needs per-site verification); bundle with the first corpus-tag re-baseline.🤖 Generated with Claude Code