fix(self-driving): scope the detected-tools scan to self-driving and filter it - #1028
Merged
gewenyu99 merged 1 commit intoJul 29, 2026
Conversation
…filter it The connected-tools scan was stashed under the warehouse program's shared `DETECTED_WAREHOUSE_SOURCES_KEY`. The integration program reads that key to build its prompt, and the integrate-run phase inherits a shallow copy of frameworkContext, so every self-driving run that installs PostHog first also rewrote the integration agent's prompt. Give self-driving its own key and accessor; nothing else reads it. The scan also returned matches from the whole warehouse catalog, so a routine repo led the issue-tracker ask with Postgres (matched on `DATABASE_URL`), Stripe and OpenAI. Filter to the kinds the inbox connects, and tag the detected count on every scan — including zero — so telemetry can tell "found nothing" from "never ran". Generated-By: PostHog Code Task-Id: df870021-4bdf-465b-ba4f-d486c37b97d3
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
gewenyu99
marked this pull request as ready for review
July 29, 2026 17:25
gewenyu99
merged commit Jul 29, 2026
9ac7981
into
tom/self-driving-detected-sources
14 checks passed
gewenyu99
added a commit
that referenced
this pull request
Jul 29, 2026
…e the leak test Two follow-ups to #1028, found by running the flow and reading the skill the ask actually comes from. The promoted-tools list was hand-written from a PR description rather than computed, so it was wrong both ways. Recomputed as the intersection of the options step 5 offers and the kinds this repo can detect: adds Shortcut, Honeybadger, Raygun, Kustomer, Plain, Canny, Productboard, Snyk and GoogleSearchConsole, all offered and all detectable, so all previously dropped on the floor; removes Intercom, which the ask never lists, so promoting it pointed the agent at a source it cannot create. The regression test guarding the frameworkContext fix is also missing from this branch — #1028 was merged at its first commit, so the test pushed afterwards was orphaned. Restored verbatim, and re-verified it fails when the shared-key write is reintroduced. Generated-By: PostHog Code Task-Id: df870021-4bdf-465b-ba4f-d486c37b97d3
gewenyu99
added a commit
that referenced
this pull request
Jul 29, 2026
Seven multi-line comment blocks from #1028 and #1031 collapsed to a single line each. Net -71 lines, no behaviour change. The rationale that was in them belongs in the PR descriptions, which already carry it: why the key is self-driving's own rather than the warehouse program's, how the promoted list was computed, and why the count tag fires at zero and at -1. Generated-By: PostHog Code Task-Id: df870021-4bdf-465b-ba4f-d486c37b97d3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Stacked on #1022 — review fixes, not a rework. Two defects in it:
The scan writes a key another program reads. #1022 stashes the result under the warehouse program's shared
DETECTED_WAREHOUSE_SOURCES_KEY.posthog-integrationreads that key to build its prompt (warehouseReportInstruction), andprepareRunSessionshallow-copiesframeworkContextinto the composed integrate-run sub-session — so every self-driving run that installs PostHog first silently gained a paragraph in the integration agent's prompt telling it to add a data-warehouse item to the setup report. That makes #1022 not inert. (Its outro is safe: a composed run returns beforebuildOutroData.)The list isn't filtered. The scanner matches the whole warehouse catalog, so a routine repo led the issue-tracker ask with Postgres (matched on
DATABASE_URL, which the registry itself flags as a deliberate over-match), Stripe and OpenAI — the wall of irrelevant options the ask was meant to lose.Why
The point of #1022 is that the ask should open with tools the user actually uses. Promoting a database and an LLM vendor to the top of an issue-tracker prompt works against that, and a scan that rewrites a neighbouring program's prompt is a coupling that will surprise whoever debugs the integration flow next.
Changes
SELF_DRIVING_DETECTED_TOOLS_KEY+ accessor. Nothing else reads it, so the blast radius is exactly the self-driving prompt.prepareRunSessionso the next person composing a program sees the inheritance rule before hitting it.Not touched: the
run-object → closure re-indent, and the prompt block's wording. Ordering and labels stay in the context-mill skill.Test plan
pnpm build && pnpm test && pnpm fix— 1668 tests pass, 0 lint errors. New/updated tests: the filter dropspg/stripeand keeps Sentry; the existing stash test now asserts what production wrote instead of hand-mirroring a value onto the session; a drift test pins the allowlist to the source registry.Not run end-to-end against a live self-driving run.
Notes for review
main) fixed the terminal overflow its Problem section cites, so what's left is choice overload, not rendering; and "safe to land in either order" doesn't hold while STEP 5's prompt text names buckets the current skill doesn't define.source_type:in the prompt as a mislabel. That was wrong —kindis documented as the PostHogsource_typeand self-driving's tools really are created as warehouse sources. No change made.SourceDetector, not a third copy. Worth reconciling against the skill's actual catalogue by someone who can read it.Created with PostHog Code