ENG-115918: Use Retry-After header for rate limits, add retry to all AC endpoints#79
Merged
Merged
Conversation
…AC endpoints - Add retry_request() unified retry wrapper: 429 uses X-Rate-Limit-Retry-After-Seconds header (cap 300s), 5XX uses exponential backoff (5s base, 120s cap + jitter) - Add is_concurrent_limit_error() and get_retry_delay() helpers - Fix process() get-task: add 429 handling, fix '>500' to '>=500' - Refactor update_task(): replace recursive retry + fixed 2s sleep with retry_request() - Add retry to upload_response() (upload-result): was zero retry - Add retry to check_for_logs_fetch() (upload-logs): was zero retry - Add retry to get_s3_upload_url() (upload-url): was zero retry - Add retry to upload_s3() (S3 PUT): 5XX backoff only (max_retries=0)
dmeenaarmorcode
added a commit
that referenced
this pull request
Apr 10, 2026
…AC endpoints (#79) (#80) - Add retry_request() unified retry wrapper: 429 uses X-Rate-Limit-Retry-After-Seconds header (cap 300s), 5XX uses exponential backoff (5s base, 120s cap + jitter) - Add is_concurrent_limit_error() and get_retry_delay() helpers - Fix process() get-task: add 429 handling, fix '>500' to '>=500' - Refactor update_task(): replace recursive retry + fixed 2s sleep with retry_request() - Add retry to upload_response() (upload-result): was zero retry - Add retry to check_for_logs_fetch() (upload-logs): was zero retry - Add retry to get_s3_upload_url() (upload-url): was zero retry - Add retry to upload_s3() (S3 PUT): 5XX backoff only (max_retries=0)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes ENG-115918
Changes
New retry infrastructure
retry_request(func, max_retries=5, max_server_retries=3)— unified retry wrapperX-Rate-Limit-Retry-After-Secondsheader (capped at 300s), random 0-10s for concurrent limit errors, 2s fallbackmin(5×2ⁿ, 120s) + jittergevent.sleep()throughoutget_retry_delay(response)— extracts delay from response header/bodyis_concurrent_limit_error(response)— detects concurrent limit vs standard rate limitEndpoint fixes
process()get-task>500missed status 500>=500update_task()put-resultretry_request()with header delayupload_response()upload-resultretry_request()check_for_logs_fetch()upload-logsretry_request()get_s3_upload_url()upload-urlretry_request()upload_s3()retry_request(max_retries=0)(5XX only)Testing
python3 -c "import ast; ast.parse(...)"