Fix DTLS 1.3 extSz out-of-bounds and word16 truncation on oversized certificate chains

[gasbytes]

Description

Fix DTLS 1.3 extSz out-of-bounds and word16 truncation on oversized certificate chains.

Fixes zd#21435

Testing

added test_dtls13_oversized_cert_chain that exercises the appropriate code pathss

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[rizlik]

move it just after outputSz is computed

[rizlik]

move it just after sendSz is computed

[programsurf]

Thanks for the fix, @gasbytes. I'm the original reporter (zd#21435). Reviewed the changes:

• tls13.c: sendSz/i bounds check before Dtls13HandshakeSend cast — addresses the root cause. - dtls13.c:1045: outputSz check in Dtls13SendFragmentedInternal — correct. - dtls13.c:1639: sendSz check in Dtls13RtxSendBuffered — correct. - extIdx + 1 < MAX_CERT_EXTENSIONS off-by-one fix — good catch. - Test covers the 70KB+ chain scenario.

LGTM. Thanks for the thorough fix and test.

[gasbytes]

retest this please

[douzzer]

only failing test is something in the multi-test PRB (logs lost)

[douzzer]

[clang-tidy-asn-template-sp-all-small-stack] [51 of 52] [5a573e0c76]
    configure with clang-tidy...   real 0m12.984s  user 0m7.858s  sys 0m6.142s
    build.../tmp/tmp.4346_28411/wolfssl_test_workdir.32702/wolfssl/tests/api/test_dtls.c:2840:9: warning: the value returned by this function should not be disregarded; neglecting it may lead to errors [bugprone-unused-return-value]
2840 |         XFSEEK(f, 0, XSEEK_END);
|         ^
./wolfssl/wolfcrypt/wc_port.h:1165:24: note: expanded from macro 'XFSEEK'
1165 |     #define XFSEEK     fseek
|                        ^
/tmp/tmp.4346_28411/wolfssl_test_workdir.32702/wolfssl/tests/api/test_dtls.c:2840:9: note: cast the expression to void to silence this warning
2840 |         XFSEEK(f, 0, XSEEK_END);
|         ^
./wolfssl/wolfcrypt/wc_port.h:1165:24: note: expanded from macro 'XFSEEK'
1165 |     #define XFSEEK     fseek
|                        ^
/tmp/tmp.4346_28411/wolfssl_test_workdir.32702/wolfssl/tests/api/test_dtls.c:2842:9: warning: the value returned by this function should not be disregarded; neglecting it may lead to errors [bugprone-unused-return-value]
2842 |         XFSEEK(f, 0, XSEEK_SET);
|         ^
./wolfssl/wolfcrypt/wc_port.h:1165:24: note: expanded from macro 'XFSEEK'
1165 |     #define XFSEEK     fseek
|                        ^
/tmp/tmp.4346_28411/wolfssl_test_workdir.32702/wolfssl/tests/api/test_dtls.c:2842:9: note: cast the expression to void to silence this warning
2842 |         XFSEEK(f, 0, XSEEK_SET);
|         ^
./wolfssl/wolfcrypt/wc_port.h:1165:24: note: expanded from macro 'XFSEEK'
1165 |     #define XFSEEK     fseek
|                        ^
Suppressed 41 warnings (32 in non-user code, 9 NOLINT).
   real 0m14.533s  user 3m43.584s  sys 0m12.986s
    clang-tidy reported defects in 1 file(s).

[gasbytes]

Jenkins retest this please (logs lost)