Skip to content

feat(ai): extend the prompt injection defense to streamed suggestions - #668

Open
Reversean wants to merge 3 commits into
feat/ai-streamingfrom
fix/ai-stream-prompt-injection
Open

feat(ai): extend the prompt injection defense to streamed suggestions#668
Reversean wants to merge 3 commits into
feat/ai-streamingfrom
fix/ai-stream-prompt-injection

Conversation

@Reversean

@Reversean Reversean commented Jul 29, 2026

Copy link
Copy Markdown
Member

The one-shot path scans the finished answer before returning it. A streamed answer leaves the server while it is still being written, so a nonce split across two deltas would pass a per-delta check and reach the client.

The stream now runs through a guard that withholds the last nonce.length - 1 characters and scans them together with each new delta, releasing only text that can no longer begin the nonce:

delta 1  "...see marker 0123456789abcdef"  ->  out: "...see marker "   held: "0123456789abcdef"
delta 2  "0123456789abcdef tail"           ->  nonce found in held + delta, answer rejected

That length is the exact minimum: an occurrence spans nonce.length characters, so holding one less leaves it inside a single scanned window. The holdback is released at the end of a text block, since that text cannot be emitted under the next block's id, and its tail is kept as scanning context so that a nonce split across two blocks is still seen.

Detection is unchanged and shared with the one-shot path. guard and onReject are required on stream(): a stream cannot be checked after it has been sent, so an omitted guard would mean an unchecked answer.

The route serves text/plain, which carries no channel for a signal that is not content, so a rejected answer arrives as the prefix already sent, cut mid-word by the holdback, followed by the fallback message. Showing a clean failure instead needs toUIMessageStreamResponse here and a matching change in Garage, and is left for a follow-up.

@Reversean
Reversean force-pushed the fix/ai-stream-prompt-injection branch 2 times, most recently from a08550b to 18f7aa1 Compare July 29, 2026 16:21
@Reversean
Reversean force-pushed the fix/ai-stream-prompt-injection branch from 18f7aa1 to 571263c Compare July 29, 2026 17:22
@Reversean Reversean changed the title feat(ai): defend streamed Ask AI suggestions against prompt injection feat(ai): extend the prompt injection defense to streamed suggestions Jul 29, 2026
@Reversean
Reversean force-pushed the fix/ai-stream-prompt-injection branch from 571263c to e64c1a7 Compare July 29, 2026 17:50
@neSpecc
neSpecc requested a lite review from Copilot August 5, 2026 12:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Extends the existing prompt-injection / marker-leak defense from one-shot completions to streamed AI suggestions by introducing nonce-based spotlighting, deterministic leak detection, and a streaming “holdback” guard that prevents nonce substrings from crossing chunk boundaries.

Changes:

  • Add spotlighting markers + per-request random nonce in the prompt, and reject outputs that reproduce the nonce.
  • Add streaming leak defense via a holdback guard wired into streamText(...).experimental_transform.
  • Add an Express /integration/ai/stream route that proxies the UI message stream response; expand tests and shared test helpers accordingly.

Reviewed changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
test/services/askAi.test.ts Expands AIService tests to cover spotlighting, rejection reporting, and stream guard wiring.
test/services/askAi-spotlighting.test.ts Adds unit coverage for prompt marker wrapping and nonce generation/collision handling.
test/services/askAi-leak-detector.test.ts Adds unit tests for deterministic nonce leak detection behavior.
test/services/askAi-holdback.test.ts Adds unit tests for the streaming holdback guard behavior and edge cases.
test/integrations/vercel-ai.test.ts Adds tests for streamText usage and the guarded transform wiring/semantics.
test/integrations/github-routes.test.ts Refactors to reuse the new Express request helper.
test/integrations/ai-routes.test.ts Adds route-level tests for the new SSE proxy endpoint and auth/validation paths.
test/helpers/expressRequest.ts Introduces a reusable Express request/response harness that supports pipe() streaming.
src/services/types.ts Exports the Event type for reuse (e.g., AI service internals).
src/services/askAi/security/spotlighting.ts Implements nonce-based marker wrapping (buildEventPrompt) + system spotlighting instruction.
src/services/askAi/security/leakDetector.ts Adds isLeaked + fallback message constant for suggestion rejection.
src/services/askAi/security/holdback.ts Implements streaming holdback guard (createLeakGuard) to detect nonce across deltas.
src/services/askAi/inputs/eventSolving.ts Documents that this serialization is unwrapped/untrusted and must be wrapped via spotlighting.
src/services/ai.ts Updates AIService to use spotlighting + leak detection, and adds a streaming suggestion method.
src/integrations/vercel-ai/routes.ts Adds /integration/ai/stream proxy route for the UI message stream response.
src/integrations/vercel-ai/index.ts Adds vercelAIApi.stream and the guarded transform that applies the stream guard to deltas.
src/index.ts Registers the new AI assistant routes on the Express app.
src/directives/requireUserInWorkspace.ts Exports checkUserInWorkspaceByProjectId for reuse in the new route.
package.json Bumps package version.
.eslintrc.js Declares Fetch/Streams globals used in tests and streaming code.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/integrations/vercel-ai/index.ts
Comment thread src/integrations/vercel-ai/routes.ts
Comment thread src/integrations/vercel-ai/routes.ts Outdated
Reversean added a commit that referenced this pull request Aug 5, 2026
The stream route mapped any error from streamSuggestion to a 404
"Event not found", including failures unrelated to the event lookup
(e.g. a stream construction error). Only the exact "Event not found"
error is now reported as 404; anything else is forwarded to Express's
error handling.

Flagged by Copilot while reviewing #668, against code this PR added.
@Reversean
Reversean force-pushed the fix/ai-stream-prompt-injection branch from e64c1a7 to 983fded Compare August 5, 2026 14:12
@Reversean
Reversean force-pushed the fix/ai-stream-prompt-injection branch 2 times, most recently from 78359fa to 6b2ec10 Compare August 17, 2026 06:16
The one-shot path scans the finished answer before returning it. A streamed answer leaves the server while it is still being written, so that scan cannot be applied: a nonce split across two deltas passes a per-delta check untouched and reaches the client.

Run the stream through a guard that withholds the last nonce.length - 1 characters and scans them together with each new delta, releasing only text that can no longer begin the nonce. That length is the exact minimum, since an occurrence spans nonce.length characters and holding one less leaves it inside a single scanned window. The holdback is released at the end of a text block, because that text cannot be emitted under the next block's id, and its tail is kept as scanning context, without which a nonce split across two blocks would pass unseen.

The guard runs in experimental_transform, before the text stream is derived from the model's parts, so it holds whichever response method the route uses. stopStream is not used: it obliges the caller to synthesize finish chunks whose shape follows the SDK version, while suppressing the remaining text keeps the stream well formed.

A rejected answer reaches the client as the prefix already sent, cut at an arbitrary character, followed by the fallback message: a text delta cannot be retracted once written.
The guard inspects text deltas. Reasoning deltas carry model text built from the same untrusted payload and the UI message stream sends them by default, so they would reach the client without ever passing the nonce check.

Turn them off at the route. Guarding them instead would mean a second holdback over a second stream, for text nothing renders.
The guard appends the fallback message as another text delta, which the client cannot tell from the answer: it concatenates deltas, so the rejection lands glued to the truncated prefix already rendered.

Emit an error part instead. The UI message stream carries it on its own channel, so the client can drop what it has and show a failure. Text already sent still cannot be taken back, but the client now knows to discard it.
@Reversean
Reversean force-pushed the fix/ai-stream-prompt-injection branch from 6b2ec10 to acda192 Compare August 17, 2026 18:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants