[jackctj117]

Pick, label and check RSA-PSS signature schemes by the key's OID in TLS 1.2 and 1.3, honour the key's PSS parameters, and accept rsa_pss_pss_* in a TLS 1.2 CertificateVerify. Wire rsapss.test into make check and add a unit test.

Fixes #11988

Description

Please describe the scope of the fix or feature addition.

Fixes zd#

Testing

How did you test?

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[github-actions]

MemBrowse Memory Report · Explore what changed →

gcc-arm-cortex-m4-dtls13

• FLASH: .text -64 B (-0.0%, 196,440 B / 1,048,576 B, total: 19% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata +48 B, .text +768 B (+0.1%, 802,932 B / 1,048,576 B, total: 77% used)
• RAM: .data +16 B (+0.0%, 121,788 B / 262,144 B, total: 46% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +448 B (+0.1%, 313,872 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .text +512 B (+0.1%, 344,464 B / 1,048,576 B, total: 33% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +384 B (+0.2%, 250,530 B / 262,144 B, total: 96% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +448 B (+0.1%, 314,704 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +384 B (+0.2%, 250,530 B / 262,144 B, total: 96% used)

stm32-sim-stm32h753

• FLASH: .text +440 B (+0.2%, 204,908 B / 2,097,152 B, total: 10% used)

zephyr-frdm_rw612

• FLASH: datas +16 B, log_const_area -8 B, rodata +36 B, text +11,712 B (+4.2%, 294,220 B / 67,108,864 B, total: 0% used)
• RAM: bss +49 B, datas +16 B (+0.1%, 58,423 B / 983,040 B, total: 6% used)

No memory changes detected for:

• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7

---

_{Catch memory regressions in CI with MemBrowse. Free for open source.}

[Copilot]

🟡 Changes recommended

RSA-PSS handling can bypass key restrictions and ignores MGF constraints, while new tests fail on supported OpenSSL versions and hash configurations.

7 open issues

• Handle older OpenSSL RSA-PSS summary formats · New
• Keep checking RSA-PSS restrictions after selecting a candidate · New
• Preserve and enforce the key's MGF restriction · New
• Gate test cases and fixtures on enabled hash algorithms · New
• Treat PSS saltLength as a minimum, not an exact value · New
• Guard SHA-384 cases when WOLFSSL_SHA384 is disabled · New
• Use the matching 3072-bit root for client verification · New

What changed in this PR

This PR addresses #11988 by aligning wolfSSL’s RSA-PSS signature selection and validation with certificate key types across TLS 1.2 and 1.3, including shared DTLS paths.

Changes:

• Distinguishes rsa_pss_rsae_* from rsa_pss_pss_* during negotiation, encoding, and verification.
• Carries certificate PSS hash and salt parameters into TLS state for compatibility checks.
• Adds API regression tests, expands OpenSSL interoperability cases, and registers rsapss.test with make check.

File	Description
wolfssl/​wolfcrypt/​asn.h	Adds decoded PSS key parameter fields.
wolfssl/​internal.h	Adds TLS PSS state and a peer-validation declaration.
wolfcrypt/​src/​asn.c	Retains parsed PSS hash and salt parameters.
tests/​api.c	Adds RSA-PSS negotiation and mutual-authentication tests.
src/​tls13.c	Updates TLS 1.3 scheme encoding and peer validation.
src/​ssl_load.c	Copies certificate PSS parameters into TLS state.
src/​internal.c	Updates scheme selection, signing, and verification.
scripts/​rsapss.test	Expands TLS/DTLS interoperability and rejection cases.
scripts/​include.am	Registers RSA-PSS interoperability tests.

_{Review effort: Balanced}

[jackctj117]

@claude review

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #12004

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): scripts/include.am, scripts/rsapss.test
Coverage: 6 of 7 in-scope changed file(s) opened by the reviewer; not opened: wolfssl/wolfcrypt/asn.h

Findings: 3
3 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Review tier: Max

[jackctj117]

Jenkins retest please. .

[jackctj117]

Jenkins retest this please

[Copilot]

🟡 Changes recommended

Unresolved TLS 1.2 RSA-PSS client-authentication flaws can select unoffered or key-incompatible signature schemes, while missing SHA-256 guards cause new tests to fail in supported configurations.

1 open issue

• Skip RSA-PSS success tests in NO_SHA256 builds · New

7 resolved since last review

• Preserve and enforce the key's MGF restriction
• Keep checking RSA-PSS restrictions after selecting a candidate
• Handle older OpenSSL RSA-PSS summary formats
• Use the matching 3072-bit root for client verification
• Guard SHA-384 cases when WOLFSSL_SHA384 is disabled
• Treat PSS saltLength as a minimum, not an exact value
• Gate test cases and fixtures on enabled hash algorithms

_{Review effort: Balanced}

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #12004

Scan targets checked: wolfcrypt-src, wolfcrypt-bugs, wolfssl-src, wolfssl-bugs
Not reviewed (no scan target covers them): scripts/include.am, scripts/rsapss.test
Coverage: 5 of 8 in-scope changed file(s) opened by the reviewer; not opened: src/ssl.c, tests/api.c, wolfssl/wolfcrypt/asn.h

Fenrir result: Approved ✅

No new issues found in the changed files.

Advisory only — this automated result does not count as a GitHub approval.

Review tier: Lite

Fenrir's latest completed scan found no issues; clearing the prior automated change request.