Skip to content

hackney_conn leak: a body truncated by the peer parks an unpooled connection in closed forever (bypasses #903's finish_sync_request/3) #918

Description

@lambadalambda

Sibling of #902, on the path that #903's finish_sync_request/3 does not cover.

When the peer closes the connection mid-body, read_full_body/2 returns with socket = undefined, so receiving({call, From}, body, ...) goes straight to closed and never reaches finish_sync_request/3:

%% hackney_conn.erl (4.7.2)
{ok, Body, #conn_data{socket = undefined} = NewData} ->
    %% Socket was closed during body read (e.g., no Content-Length)
    {next_state, closed, NewData, [{reply, From, {ok, Body}}]};

closed(enter) only arms closed_grace_expired -> {stop, normal} for pooled connections, and an unpooled conn's owner is hackney_conn_sup (connect_direct/4 and start_conn_with_socket_internal/5 pass no owner), so the owner-DOWN clause never fires either. The process parks forever, pinning every refc binary it read.

The caller cannot clean up: hackney 4 ignores with_body and always returns the body directly (hackney.erl:1318), so a sync request never yields a ref — and the result is {ok, 200, Headers, Body}, a success.

This is what #902 predicted but did not fix:

a finite timeout just moves the zombie from connected to closed

Reproduction

Any server that declares a Content-Length and then closes early. Real-world trigger: a reverse proxy that aborts a large response mid-stream, fetched through an HTTP CONNECT proxy (tunnels are always no_reuse and never pooled).

%% server: declare 100 MB, send 8 MB, close
{ok, L} = gen_tcp:listen(0, [binary, {active, false}, {reuseaddr, true}]),
{ok, Port} = inet:port(L),
spawn(fun() ->
    {ok, S} = gen_tcp:accept(L),
    _ = gen_tcp:recv(S, 0, 5000),
    gen_tcp:send(S, "HTTP/1.1 200 OK\r\nContent-Length: 100000000\r\n\r\n"),
    gen_tcp:send(S, binary:copy(<<"x">>, 8 * 1024 * 1024)),
    gen_tcp:close(S)
end),

Url = "http://127.0.0.1:" ++ integer_to_list(Port) ++ "/x",
{ok, 200, _H, Body} = hackney:request(get, Url, [], <<>>, [{pool, false}]),
byte_size(Body).   %% 8388608 -- success, despite truncation

Observed on 4.7.2, via a Pleroma instance whose media proxy sits behind Squid:

hackney_conn children before: 0
RESULT ok status=200 body_bytes=8388608
hackney_conn children after: 1
  #PID<0.7145.0> state=:closed binary=8618039

Still present after 60 s idle. hackney:close(Pid) frees it instantly. The identical request without the proxy (pooled) leaves 0 connections, because the pool's grace timer reaps it.

In production this was one leaked process per proxied media fetch, each holding tens of MB, until the node OOMed.

Suggested fix

Give the truncated path the same treatment #903 gave the completed one: consult the reuse policy instead of unconditionally parking in closed. PR follows.

receiving({call, From}, stream_body, ...) has the same {done, #conn_data{socket = undefined}} clause and needs it too.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions