Skip to content

ECS baseline build hangs: scoped-session bypasses test boto3 mock + watchdog doesn't reap + late πŸ‘€ ACKΒ #615

Description

@isadeks

Problem

On the ECS Fargate compute substrate, the agent's pre-agent baseline build (mise run build in the cloned repo) can hang silently β€” the agent unit-test suite stalls for 40+ min, up to the build-verify ceiling, with no output.

Root cause

The ECS agent task def sets AGENT_SESSION_ROLE_ARN (per-session IAM scoping, #209). When it's set, aws_session.get_session() resolves a scoped session and tenant_client(...) returns session.client(...) β€” which bypasses a @patch("boto3.client") mock. A mocked test (e.g. test_attachments) then makes a real S3 get_object that blocks forever on the ECS network (no egress) in a socket read that pytest-timeout's SIGALRM (main-thread only) can't interrupt.

agent/tests/conftest.py already resets the session cache per test, but that is insufficient: with AGENT_SESSION_ROLE_ARN still in the environment, a cold get_session() just re-resolves scoped and the mock is bypassed again.

Separately, the session-level faulthandler.dump_traceback_later(exit=True) watchdog does not reap the hang: pytest's faulthandler_timeout re-arms faulthandler's single internal timer per-test without exit=True, cancelling the exit timer β€” so a hang dumps but never exits.

Secondary UX gap (same fix)

react_task_started (πŸ‘€ + Backlogβ†’In Progress) fires after setup_repo() (the baseline build). On a large repo that build is many minutes, so a Linear/Jira task shows no feedback at all during it β€” the issue looks dead.

Fix

  1. Add AGENT_SESSION_ROLE_ARN to the _AGENT_ENV_VARS scrub list in conftest.py so every test resolves the unscoped path where boto3.client mocks intercept. + regression test.
  2. Replace/augment the faulthandler watchdog with an independent daemon-thread reaper that dumps all stacks and os._exit(1)s at a fixed deadline β€” pytest cannot clobber it; a blocked socket read releases the GIL so the timer thread runs.
  3. Move the πŸ‘€ / In-Progress ACK (+ channel token resolution + start comment) to before setup_repo() so a large-repo task shows immediate feedback. As a side benefit a setup-phase failure then has a πŸ‘€ for the existing crash handler to swap to ❌.

Evidence

Diagnosed + fixed + deployed on the linear-vercel line and live-verified on dev (ECS substrate): the agent suite that hung 55 min now runs test_attachments in ~1s and the whole 1328-test suite completes in ~12s; the πŸ‘€ appears ~3s after task start instead of after the multi-minute baseline. This PR ports that fix to main.

Acceptance

  • Agent test suite resolves unscoped even when AGENT_SESSION_ROLE_ARN is set β†’ no real S3 call, no hang.
  • A genuine hang is reaped (non-zero exit) within the deadline instead of stalling to the build ceiling.
  • πŸ‘€ / In-Progress appears within seconds of task admission regardless of baseline-build duration.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

approvedWhen an issue has been approved and readybugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions