Bound CKS extension allocation in TLSX_CKS_Parse

[embhorn]

Description

The TLS 1.3 CKS (Certificate Key Selection) extension parser allocated a heap buffer sized directly from the attacker-controlled extension length, with no upper bound other than the word16 type limit.

Fixes zd22167

Testing

`test_wolfSSL_dual_alg_cks_parse_ext

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

This PR mitigates a potential denial-of-service issue in the TLS 1.3 CKS (Certificate Key Selection) extension parser by preventing attacker-controlled extension lengths from driving unbounded heap allocations.

Changes:

• Introduces WOLFSSL_MAX_CKS_SIGSPEC_SZ to define a semantic maximum size for the CKS signature specifier list.
• Adds an explicit upper-bound check in TLSX_CKS_Parse() to reject oversized lists before allocation.
• Adds an API test to validate rejection/acceptance behavior at boundary conditions and exposes TLSX_CKS_Parse for test visibility.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
wolfssl/ssl.h	Adds a max-size constant for CKS signature specifier lists.
wolfssl/internal.h	Makes TLSX_CKS_Parse test-visible for API tests.
src/tls.c	Enforces a maximum CKS list length to bound heap allocation.
tests/api/test_ssl_ext.h	Registers the new CKS parser test.
tests/api/test_ssl_ext.c	Adds coverage for CKS parsing length/value validation and regression behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +82 B (+0.0%, 200,744 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m4-crypto-only

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +82 B (+0.1%, 174,992 B / 262,144 B, total: 67% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata +88 B, .text +64 B (+0.0%, 771,708 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pkcs7

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +82 B, .text +64 B (+0.1%, 213,140 B / 262,144 B, total: 81% used)

gcc-arm-cortex-m4-pq

• FLASH: .rodata +88 B (+0.0%, 279,888 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .rodata +88 B (+0.0%, 325,616 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-tls13

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +82 B (+0.0%, 236,570 B / 262,144 B, total: 90% used)

gcc-arm-cortex-m7

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +82 B (+0.0%, 200,680 B / 262,144 B, total: 77% used)

gcc-arm-cortex-m7-pq

• FLASH: .rodata +88 B (+0.0%, 280,464 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m7-tls13

• FLASH: .rodata.CSWTCH.1 +8 B, .rodata.str1.1 +82 B (+0.0%, 236,634 B / 262,144 B, total: 90% used)

linuxkm-standard

• Data: __patchable_function_entries -8 B (-0.0%, 48,592 B)

stm32-sim-stm32h753

• FLASH: .text +92 B (+0.1%, 183,828 B / 2,097,152 B, total: 9% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• linuxkm-pie

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10953

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[embhorn]

retest this please