Skip to content

Retry ALB CreateRule on PriorityInUse race condition#11

Merged
edwardrf merged 3 commits intomainfrom
copilot/fix-lambda-priority-issue
Mar 24, 2026
Merged

Retry ALB CreateRule on PriorityInUse race condition#11
edwardrf merged 3 commits intomainfrom
copilot/fix-lambda-priority-issue

Conversation

Copy link
Contributor

Copilot AI commented Mar 16, 2026

Concurrent Lambda invocations for different ACME domains each call GetNextAvailablePriority independently, causing both to claim the same priority. The second CreateRule call fails with PriorityInUse.

Changes

  • aws/alb/updatealb.go: Added createRuleWithRetry — on PriorityInUse, waits 1s, re-fetches the next available priority (which now correctly skips the priority claimed by the competing invocation), and retries up to 10 times
  • Refactored AddListenerStaticRule and AddListenerTriggerTargetGroupRule to use createRuleWithRetry
func createRuleWithRetry(ctx context.Context, svc *elbv2.Client, listenerArn string, input *elbv2.CreateRuleInput) error {
    for i := 0; ; i++ {
        if _, err := svc.CreateRule(ctx, input); err != nil {
            var apiErr smithy.APIError
            if errors.As(err, &apiErr) && apiErr.ErrorCode() == "PriorityInUse" {
                if i >= maxPriorityRetries {
                    return fmt.Errorf("failed to create rule after %d retries: %w", maxPriorityRetries, err)
                }
                log.Printf("Priority %d is in use, retrying (%d/%d)...", *input.Priority, i+1, maxPriorityRetries)
                time.Sleep(time.Second)
                priority, err := GetNextAvailablePriority(ctx, listenerArn)
                // ...
                input.Priority = ptr.Int32(priority)
                continue
            }
            return err
        }
        return nil
    }
}
Original prompt

This section details on the original issue you should resolve

<issue_title>Race: multiple lambda invocations try to create rule with priority 1</issue_title>
<issue_description>Two ACME domains result in two separate lambda invocations (until we do #5), each creating a rule starting from priority 1, causing one lambda to fail. We could handle the priority failure and retry.

2026-03-16T21:13:25.506Z cd ** error: sdk-v2/provider2.go:572: sdk.helper_schema: invoking Lambda Function (acme-lambda-0d3a9fa): {"errorMessage":"failed to create listener static rule: operation error Elastic Load Balancing v2: CreateRule, https response error StatusCode: 400, RequestID: 24b62462-04f5-4e6b-ba9c-f954daf18fdd, PriorityInUse: Priority '1' is currently in use","errorType":"wrapError"}: provider=aws@6.83.2

Workaround: rerunning the deployment should fix it.
</issue_description>

<agent_instructions>Handle the PriorityInUse error message and retry creation of ListenerRule</agent_instructions>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits March 17, 2026 00:02
Co-authored-by: lionello <591860+lionello@users.noreply.github.com>
Co-authored-by: lionello <591860+lionello@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix race condition for lambda invocations creating rules Retry ALB CreateRule on PriorityInUse race condition Mar 17, 2026
Copilot AI requested a review from lionello March 17, 2026 00:08
@lionello lionello marked this pull request as ready for review March 17, 2026 01:35
@lionello lionello requested a review from edwardrf March 17, 2026 20:46
Copy link
Collaborator

@edwardrf edwardrf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@edwardrf edwardrf merged commit 4c35231 into main Mar 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race: multiple lambda invocations try to create rule with priority 1

3 participants