Skip to content

Commit 22ae575

Browse files
icecrasher321claude
andcommitted
fix(secrets): make the reference prefilter exactly as tight as the authority
Review round 6. The gap between `{{` and the name accepted any non-word character, so `{{-API_KEY-}}` and `{{"API_KEY"}}` matched in SQL while `ENV_REF_PATTERN` rejects them. The previous commit called that free — "costs a candidate row and nothing else" — which was wrong: a candidate row is a slot under BLOCK_SCAN_LIMIT, so enough near-misses sorted earlier exhaust the cap before a genuine reference is read, and the tab reports a live key as unused. That is the same failure the tightening in round 1 was meant to remove, reintroduced by the round 4 loosening that fixed JSON-escaped whitespace. The gap now enumerates exactly the whitespace `\s` accepts, in each encoding it can arrive in: `[[:space:]]` for raw ASCII, `\\[tnrf]` and `\\u000[bB]` for the JSON escapes, and an explicit class for the Unicode spaces Postgres emits verbatim but `[[:space:]]` does not match. That class is generated from a code-point table rather than written literally. Writing it by hand put a run of invisible characters in the source — a reviewer cannot check them, and a formatter or editor can silently mangle them. The table is the readable form and `toPgEscape` renders it. Verified against the real jsonb rendering, 17 cases: raw space, tab, newline, carriage return, vertical tab, form feed, U+00A0, U+202F, U+3000 and an embedded reference all match; `{{-API_KEY-}}`, `{{"API_KEY"}}`, `{{API_KEY_TEST}}`, `{{MY_API_KEY}}`, prose and an across-braces span all do not. Every candidate the SQL admits is now a real occurrence, so the cap counts references and nothing else. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 18ef4a6 commit 22ae575

1 file changed

Lines changed: 41 additions & 15 deletions

File tree

  • apps/sim/lib/secrets/references

apps/sim/lib/secrets/references/scan.ts

Lines changed: 41 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -111,20 +111,47 @@ function referencesEnvKey(text: string, name: string): boolean {
111111
}
112112

113113
/**
114-
* One unit of what may sit between `{{` and the name.
114+
* Code points JS `\s` matches beyond ASCII, as inclusive ranges. Spelled out as numbers and
115+
* rendered to `\uXXXX` below rather than written literally, so the source stays readable ASCII
116+
* instead of carrying a run of invisible characters no reviewer could check.
117+
*/
118+
const UNICODE_SPACE_RANGES: ReadonlyArray<readonly [number, number]> = [
119+
[0x00a0, 0x00a0],
120+
[0x1680, 0x1680],
121+
[0x2000, 0x200a],
122+
[0x2028, 0x2029],
123+
[0x202f, 0x202f],
124+
[0x205f, 0x205f],
125+
[0x3000, 0x3000],
126+
[0xfeff, 0xfeff],
127+
]
128+
129+
/** A code point as the `\uXXXX` escape a Postgres regex understands. */
130+
function toPgEscape(codePoint: number): string {
131+
return `\\u${codePoint.toString(16).padStart(4, '0')}`
132+
}
133+
134+
const UNICODE_SPACE_CLASS = UNICODE_SPACE_RANGES.map(([low, high]) =>
135+
low === high ? toPgEscape(low) : `${toPgEscape(low)}-${toPgEscape(high)}`
136+
).join('')
137+
138+
/**
139+
* One unit of what may sit between `{{` and the name: exactly the whitespace
140+
* `ENV_REF_PATTERN` accepts, in each encoding it can arrive in.
115141
*
116-
* Three encodings have to be accepted at once, because the prefilter reads a `::text` rendering
117-
* of a JSON column and the two regex engines disagree about whitespace:
118-
* - raw characters, including every Unicode space (`ENV_REF_PATTERN`'s `\s` accepts U+00A0,
119-
* U+202F and U+3000, which Postgres `[[:space:]]` does not) — covered by `[^[:alnum:]_]`;
120-
* - JSON two-character escapes, since `jsonb::text` renders a real tab as the literal pair
121-
* `\` `t` and `t` is alphanumeric — covered by `\\[a-z]`;
122-
* - JSON `\uXXXX` escapes, which is how a vertical tab survives the same rendering.
142+
* The prefilter reads a `::text` rendering of a JSON column, and the two regex engines
143+
* disagree about whitespace, so all three forms are spelled out: `[[:space:]]` for raw ASCII;
144+
* the JSON escapes, since `jsonb::text` renders a real tab as the literal pair `\\` `t` and a
145+
* vertical tab as `\\u000b`; and the Unicode class above, which Postgres emits verbatim and
146+
* `[[:space:]]` does not match though JS `\\s` does.
123147
*
124-
* Excluding word characters rather than enumerating whitespace means no code-point list to drift
125-
* as `\s` evolves, while a longer key on either side (`_TEST`, `MY_`) still cannot be consumed.
148+
* Enumerating whitespace rather than excluding word characters costs a list to keep in step
149+
* with `\\s`, and buys a prefilter exactly as tight as the authority. A looser gap admitted
150+
* `{{-NAME-}}` and `{{"NAME"}}`, and those are not free: each occupies a row under
151+
* {@link BLOCK_SCAN_LIMIT}, so a workspace with enough of them sorted earlier would exhaust
152+
* the cap before a genuine reference was read — reporting a live key as unused.
126153
*/
127-
const REFERENCE_GAP = String.raw`(\\u[0-9a-fA-F]{4}|\\[a-z]|[^[:alnum:]_])`
154+
const REFERENCE_GAP = `([[:space:]]|\\\\[tnrf]|\\\\u000[bB]|[${UNICODE_SPACE_CLASS}])`
128155

129156
/**
130157
* Matches the name sitting inside `{{ }}` with only {@link REFERENCE_GAP} units between.
@@ -136,10 +163,9 @@ const REFERENCE_GAP = String.raw`(\\u[0-9a-fA-F]{4}|\\[a-z]|[^[:alnum:]_])`
136163
* row cap — so on a workspace with enough of them, genuine references sorted later were never
137164
* read at all.
138165
*
139-
* The gap can admit a non-reference like `{{-NAME-}}`; that costs a candidate row and nothing
140-
* else, because the scanners below re-check every candidate with `ENV_REF_PATTERN` and remain the
141-
* authority. Erring loose is deliberate — a false positive is a wasted read, a false negative is
142-
* this feature telling someone a live key is unused.
166+
* The gap accepts exactly the whitespace `ENV_REF_PATTERN` does, so a candidate row is always a
167+
* real occurrence and the cap counts only references. The scanners below still re-check each one
168+
* and remain the authority; this decides what is worth reading.
143169
*/
144170
function referencesKey(column: unknown, envKey: string) {
145171
return sql`${column} ~ ${`\\{\\{${REFERENCE_GAP}*${envKey}${REFERENCE_GAP}*\\}\\}`}`

0 commit comments

Comments
 (0)