[Bug]: RFC 8446 violation: wolfSSL sends `unsupported_extension` instead of `illegal_parameter` for a bad extension in a HelloRetryRequest

[aeyno]

Version

wolfssl 5.9.1

Description

A wolfSSL TLS client receiving a ServerHello whose Random field equals the
HelloRetryRequest magic value (RFC 8446 §4.1.3) and whose extensions include a
recognised extension that is not permitted in a HelloRetryRequest sends an
unsupported_extension alert (code 110) instead of the illegal_parameter alert
(code 47) required by RFC 8446 §4.2.

OpenSSL 3.4.0 sends illegal_parameter for the same input.

Impact

RFC violation. Any TLS 1.3 peer relying on the alert code to distinguish a
malformed HelloRetryRequest from an unsolicited-extension error will misclassify
the failure. Additionally, a conforming peer that expects illegal_parameter on
malformed HRR messages may log or handle the alert incorrectly, complicating
diagnostics and error recovery.

RFC 8446 violation

RFC 8446 §4.2 requires:

"If an implementation receives an extension which it recognizes and which is not specified for the message in which it appears, it MUST abort the handshake with an "illegal_parameter" alert."

A ServerHello carrying the HRR magic Random must be treated as a
HelloRetryRequest. The extensions permitted in a HelloRetryRequest are listed
in RFC 8446 Table 4: supported_versions, cookie, key_share (group only), and
server_supported_groups. A ServerName extension is not in
this set; it is therefore not specified for this message. Per §4.2, the correct
alert upon receiving it is illegal_parameter (47).

wolfSSL sends unsupported_extension (110) instead.

Reproduction steps

Start the Python server below, which waits for one connection, echoes back a
ServerHello carrying the HRR magic Random and a single ServerName extension,
then prints the alert the client sends:

import socket
import struct

HOST = "0.0.0.0"
PORT = 3000

# RFC 8446 §4.1.3: ServerHello Random value that MUST cause the client to
# treat the message as a HelloRetryRequest.
HRR_MAGIC = bytes.fromhex(
    "CF21AD74E59A6111BE1D8C021E65B891C2A211167ABB8C5E079E09E2C8A8339C"
)

ALERT_DESC = {
    0:   "close_notify",
    10:  "unexpected_message",
    40:  "handshake_failure",
    47:  "illegal_parameter",
    50:  "decode_error",
    70:  "protocol_version",
    109: "missing_extension",
    110: "unsupported_extension",
}


def extract_session_id(ch: bytes) -> bytes:
    """Extract the session_id from a raw TLS ClientHello record."""
    # Record header (5) + Handshake header (4) + legacy_version (2) + Random (32) = 43
    off = 43
    if len(ch) <= off:
        return b""
    sid_len = ch[off]
    return ch[off + 1 : off + 1 + sid_len]


def make_malformed_hrr(session_id: bytes) -> bytes:
    """
    Build a ServerHello with the HRR magic Random and a ServerName extension.

    Per RFC 8446 §4.1.3 the client MUST process this as a HelloRetryRequest.
    The ServerName extension (0x0000) is not in the permitted HRR extension list
    (RFC 8446 Table 4), so RFC 8446 §4.2 requires the client to abort with
    illegal_parameter (47).
    """
    # ServerName ACK: type=0x0000, length=0 (4-byte extension, empty payload)
    sni_ext = struct.pack(">HH", 0x0000, 0)
    exts = struct.pack(">H", len(sni_ext)) + sni_ext

    body  = b"\x03\x03"                        # legacy_version = TLS 1.2 (0x0303)
    body += HRR_MAGIC                           # Random = HRR magic (32 bytes)
    body += bytes([len(session_id)]) + session_id
    body += b"\xc0\x2b"                         # TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    body += b"\x00"                             # compression = null
    body += exts                                # extensions

    hs = b"\x02" + struct.pack(">I", len(body))[1:] + body
    return b"\x16\x03\x03" + struct.pack(">H", len(hs)) + hs


def parse_alerts(data: bytes) -> list[str]:
    msgs = []
    i = 0
    while i + 5 <= len(data):
        rec_t = data[i]
        rec_l = struct.unpack(">H", data[i + 3 : i + 5])[0]
        body  = data[i + 5 : i + 5 + rec_l]
        i    += 5 + rec_l
        if rec_t == 0x15 and len(body) >= 2:
            level = "fatal" if body[0] == 2 else "warning"
            desc  = ALERT_DESC.get(body[1], body[1])
            msgs.append(f"Alert({level},{desc})")
    return msgs


with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as srv:
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind((HOST, PORT))
    srv.listen(1)
    print(f"[*] Listening on {HOST}:{PORT} ...")

    conn, addr = srv.accept()
    with conn:
        print(f"[+] Connection from {addr}")
        ch = conn.recv(4096)
        print(f"[>] Received ClientHello ({len(ch)} bytes)")

        sid = extract_session_id(ch)
        hrr = make_malformed_hrr(sid)
        conn.sendall(hrr)
        print("[<] Sent malformed HelloRetryRequest (HRR magic Random + ServerName extension)")

        conn.settimeout(3)
        data = b""
        try:
            while True:
                chunk = conn.recv(4096)
                if not chunk:
                    break
                data += chunk
        except socket.timeout:
            pass

        alerts = parse_alerts(data)
        result = ", ".join(alerts) if alerts else f"raw: {data[:20].hex()}"
        print(f"[>] Client response: {result}")

Then connect with a wolfSSL client that advertises both TLS 1.2 and TLS 1.3
(the bug does not manifest if the client is restricted to TLS 1.3 only):

./build/examples/client/client -p 3000

Expected behavior (RFC 8446 §4.2): the client recognises the HRR
magic Random, treats the message as a HelloRetryRequest, finds the ServerName
extension not permitted in that context, and aborts with illegal_parameter (47).

Acknowledgements

This bug was found thanks to the tlspuffin fuzzer designed and developed by the tlspuffin team:

• Nataël Baffou - Engineer, Inria, France
• Olivier Demengeon - Engineer, Inria, France
• Tom Gouville - PhD student, Inria, France
• Lucca Hirschi - Researcher, Inria, France
• Steve Kremer - Researcher, Inria, France