Repository navigation
fix(gcs): retry transient uploads before crashing consumer - #371
Merged
enochtangg merged 4 commits intoSep 21, 2026
Conversation
Retry GCS writer token/request failures with exponential backoff. Return RunTaskError::Other after max attempts or on HTTP 4xx so arroyo crashes the consumer instead of discarding the batch and committing. Co-Authored-By: Filippo Pacifici <fpacifici@sentry.io>
fpacifici
marked this pull request as ready for review
September 17, 2026 17:29
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
enochtangg
reviewed
Sep 17, 2026
enochtangg
left a comment
enochtangg
left a comment
Contributor
There was a problem hiding this comment.
Looks good, just two small comments
| assert!(StatusCode::UNAUTHORIZED.is_client_error()); | ||
| assert!(StatusCode::FORBIDDEN.is_client_error()); | ||
| assert!(StatusCode::NOT_FOUND.is_client_error()); | ||
| assert!(StatusCode::TOO_MANY_REQUESTS.is_client_error()); |
Contributor
There was a problem hiding this comment.
A 429 is classified as permanent. Does that mean a GCS rate-limit stop the consumer rather than retrying? I think in this case, we'd want to retry with exponential backoff?
| attempt | ||
| ); | ||
| let gcs_labels = vec![("source".to_string(), route_source.to_string())]; | ||
| metrics::histogram!(METRIC_SINK_GCS_WRITER_BYTES, &gcs_labels).record(bytes_len as f64); |
Contributor
There was a problem hiding this comment.
Should we also add a DD counter for retries and exhaustion?
enochtangg
approved these changes
Sep 18, 2026
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Transient GCS upload failures in
gcs_writerwere returned asRunTaskError::RetryableError. Arroyo logs that and drops the message, then later commits offsets past the failed parquet batch. That can permanently lose outcomes data.It seems arroyo is not actually retrying retryable errors.
https://github.com/getsentry/arroyo/blob/main/rust-arroyo/src/processing/strategies/run_task_in_threads.rs#L194-L196
It seems the task that raises a Retryable error is just ignored.
This adds a
retry_policyfunction that performs the retry when uploading fails for reasons that are not on theclient side. It applies exponential back off.
Fixes INC-2473.
Requested by Filippo Pacifici.
--
View Junior Session [Sentry]