ecs-agent disconnected, ConnectionExpired: Reconnect to continue

[nathanielks]

Hey team!

ECS is complaining that it's lost connection with the agent. I was just curious if y'all have seen these errors before:

In the ECS console:

service docker-demo-app was unable to place a task because no container instance met all of its requirements. The closest matching container-instance af2a4711-2b8b-420a-8699-66623a6fa9c9 encountered error "AGENT". For more information, see the Troubleshooting section.

From docker log:

[WARN] Connection closed for a valid reason: websocket: close 1000 (normal): ConnectionExpired: Reconnect to continue

[nathanielks]

service ecs-agent status:

Process: 1705 ExecStartPre=/usr/bin/docker pull amazon/amazon-ecs-agent:v1.14.1 (code=exited, status=0/SUCCESS)
Process: 1681 ExecStartPre=/usr/bin/docker rm ecs-agent (code=exited, status=1/FAILURE)
Process: 1668 ExecStartPre=/usr/bin/docker kill ecs-agent (code=exited, status=1/FAILURE)
Process: 1665 ExecStartPre=/bin/mkdir -p /var/log/ecs (code=exited, status=0/SUCCESS)
Process: 1662 ExecStartPre=/bin/mkdir -p /var/lib/ecs/data (code=exited, status=0/SUCCESS)

[achille-roussel]

We've seen this on older versions of the ECS agent. Does the agent not reconnect automatically after losing the connection?

[nathanielks]

@achille-roussel correct, the logs are littered with those ConnectionExpired warnings every 30 seconds or so. We're running 1.14.1. I'm going to try downgrading and see if that helps at all.

[nathanielks]

This ticket suggests the agent disconnects due to inactivity. With this being a test stack, it's super inactive and I suppose it's disconnecting because of that. aws/amazon-ecs-agent#730.