Skip to content

fix(replays): use keyset pagination in delete_replays finder so it stops timing out Snuba - #121027

Open
strongs wants to merge 1 commit into
masterfrom
strongs/keyset-pagination-for-delete-replays
Open

fix(replays): use keyset pagination in delete_replays finder so it stops timing out Snuba#121027
strongs wants to merge 1 commit into
masterfrom
strongs/keyset-pagination-for-delete-replays

Conversation

@strongs

@strongs strongs commented Jul 31, 2026

Copy link
Copy Markdown
Member

Problem: the delete_replays GoCD custom job (getsentry.jobs.delete_replays.DeleteReplays, running since 2026-07-23 to clear Ramp's iOS/Android replays for a PII leak) times out Snuba reads and crashes. Symptom is SENTRY-5S8N: SnubaError: HTTPConnectionPool(host='snuba-api', port=80): Read timed out. (read timeout=30), referrer replays.scripts.delete_replays.

Root cause is deep-offset pagination in the finder. delete_replays pages with a growing OFFSET:

  1. _get_rows_matching_deletion_pattern runs a GROUP BY replay_id aggregation over the whole project + time window with set_limit(batch_size) / set_offset(offset).
  2. A growing OFFSET is not a cheap seek in ClickHouse. To return the rows at a given offset, it has to scan and aggregate every earlier row and then throw them away.
  3. So each page costs more than the last. Early pages return in well under 30s. Deep pages do not.
  4. Past a certain offset the per-page cost exceeds the 30s client read timeout in sentry/utils/snuba.py, the query is abandoned, and the job crashes. The notebook saw the same wall: "dry run failed before 1:34 @ offset 871,000."

There's a second, latent defect in the same query: it has no ORDER BY at all. Offset paging over an unordered result is non-deterministic, so ClickHouse can reorder rows between pages and the job can silently skip or double-process replays.

Fix: switch the finder to keyset (seek) pagination on replay_id.

@strongs
strongs requested a review from a team as a code owner July 31, 2026 23:20
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants