Skip to content

fix(cli): stop telemetry hangs and errors on blocked networks#5880

Merged
avallete merged 16 commits into
developfrom
pamela/cli-telemetry-network-resilience
Jul 21, 2026
Merged

fix(cli): stop telemetry hangs and errors on blocked networks#5880
avallete merged 16 commits into
developfrom
pamela/cli-telemetry-network-resilience

Conversation

@pamelachia

@pamelachia pamelachia commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

On networks that block or blackhole eu.posthog.com (corporate proxies, CI and Docker egress policies, air-gapped machines), telemetry breaks the CLI itself: stable v2.109.1 pays a 5s trailing hang and exits 1 with an UnknownError payload after every command, long-running commands print raw SDK stack traces to stderr, and the Go binary's shutdown wait is unbounded. The only workaround was supabase telemetry disable, which permanently removes those users from CLI analytics. This PR makes telemetry strictly fire-and-forget: a failed delivery never changes a command's output, exit code, or runtime beyond a 2s cap.

Changed:

  • Exit 1 after successful commands: the TS shutdown finalizer was Effect.promise(...).pipe(Effect.ignore), but Effect.promise turns the shutdown-deadline rejection into a defect that Effect.ignore cannot swallow, so a telemetry timeout failed the whole command. The rejection is now caught on the promise itself.
  • Unbounded and stacking exit delays: one shared 2s budget (EXIT_DELAY_CAP_MS) bounds each request and the entire _shutdown drain on the TS side, and the Go client gets the same 2s ShutdownTimeout (the SDK default waits indefinitely, including retries). The previous 5s shutdown deadline left room for two sequential 2s request timeouts (~4s exits).
  • Work outliving the deadline (review round 3): the SDK's shutdown deadline only stops the wait, it does not cancel work. The in-flight request kept running past scope release and the drain then started the queued request, keeping the runtime alive ~4s instead of 2s. A lifecycle AbortController now composes into every request's signal and fires when the deadline hits, so no request survives or starts after the scope closes. Covered by a unit regression asserting zero active requests one tick after release (red on the previous code) and an e2e test that runs the compiled binary against a TCP blackhole and asserts wall-clock process exit, empty stderr, and that telemetry actually reached the wire.
  • Stderr noise: Go routes SDK logs to the --debug logger; TS reports delivery failures as delivered inside the wrapped fetch, because posthog-node has no logger hook, only hardcoded console.error calls, so an infallible transport is the one supported way to get no retries, no output, and no stall.

Note: the exit bounds are caps, not floors. On healthy networks both shells return as soon as the flush completes (flushAt: 1 keeps the send overlapping command runtime), and fully detaching the send instead would race process exit and silently drop events on healthy networks. The accepted tradeoff is that deliveries slower than 2s are dropped; telemetry is best-effort, observable under --debug on the Go side only.

Linear

  • fixes GROWTH-1001

@pamelachia
pamelachia requested a review from a team as a code owner July 15, 2026 06:03
@pamelachia pamelachia self-assigned this Jul 15, 2026
@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Supabase CLI preview

npx --yes https://pkg.pr.new/supabase/cli/supabase@74a0e053158089fae8129f41d30300d2a1e0dbce

Preview package for commit 74a0e05.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a96c6f5826

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/cli/src/shared/telemetry/posthog-client.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e38cb95741

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread apps/cli-go/internal/telemetry/client.go
@avallete

Copy link
Copy Markdown
Member

@codex review

@pamelachia
pamelachia marked this pull request as draft July 15, 2026 11:21
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 77ce8398ee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@pamelachia
pamelachia marked this pull request as ready for review July 17, 2026 06:04
Comment thread apps/cli/src/shared/telemetry/posthog-client.ts Outdated

@avallete avallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Go-side timeout and debug-log routing in this PR look good and should stay. I don’t think we need to additionally coordinate one shared shutdown budget across TS parents and Go children: that would broaden the change into cross-runtime telemetry ownership for a transitional sidecar. Let’s keep the blocking scope focused on ensuring the TS client’s 2s shutdown cap also bounds actual Bun process exit.

Comment thread apps/cli/src/shared/telemetry/posthog-client.ts Outdated
@pamelachia

Copy link
Copy Markdown
Contributor Author

also if this doesn't work, and you have a preferred method @avallete that isn't how i did it, can i close this PR and I'll create a linear issue for CLI team to figure this out?

@avallete avallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good, thank's for all the adjustments you made through the review loops @pamelachia !

@avallete
avallete enabled auto-merge July 21, 2026 13:56
@avallete
avallete added this pull request to the merge queue Jul 21, 2026
Merged via the queue into develop with commit 3fa5a9d Jul 21, 2026
22 checks passed
@avallete
avallete deleted the pamela/cli-telemetry-network-resilience branch July 21, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants