crypto: add mgf1Hash for RSA-OAEP

[adamjmcgrath]

crypto.publicEncrypt() and crypto.privateDecrypt() expose only the oaepHash option, which sets both the OAEP message digest and the MGF1 hash.

As a result, Node cannot decrypt (or produce) an RSA-OAEP ciphertext where the OAEP digest and the MGF1 hash differ — e.g. OAEP digest = SHA-256, MGF1 = SHA-1.

OpenSSL exposes the two digests independently via EVP_PKEY_CTX_set_rsa_oaep_md and EVP_PKEY_CTX_set_rsa_mgf1_md so this PR adds an mgf1Hash option that, when set, calls EVP_PKEY_CTX_set_rsa_mgf1_md independently of oaepHash.

This is required to implement http://www.w3.org/2001/04/xmlenc#rsa-oaep-mgf1p where the message digest can be configurable (e.g. SHA-256) but the mask generation function must be fixed to SHA-1.

[panva]

Thank you Adam

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/75560/

[codecov]

Codecov Report

❌ Patch coverage is 71.42857% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.31%. Comparing base (27d6cfa) to head (0a04e0b).
⚠️ Report is 50 commits behind head on main.

Files with missing lines	Patch %	Lines
src/crypto/crypto_cipher.cc	55.55%	0 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #65073      +/-   ##
==========================================
- Coverage   90.31%   90.31%   -0.01%     
==========================================
  Files         759      759              
  Lines      248258   248269      +11     
  Branches    46854    46856       +2     
==========================================
- Hits       224225   224220       -5     
+ Misses      15465    15461       -4     
- Partials     8568     8588      +20     

Files with missing lines	Coverage Δ
lib/internal/crypto/cipher.js	98.01% <100.00%> (+0.01%)	⬆️
src/crypto/crypto_cipher.h	68.47% <ø> (ø)
src/crypto/crypto_rsa.cc	64.21% <100.00%> (+0.12%)	⬆️
src/crypto/crypto_cipher.cc	76.51% <55.55%> (-0.41%)	⬇️

... and 30 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/75581/

[avivkeller]

Commit Queue failed

- Loading data for nodejs/node/pull/65073
✔  Done loading data for nodejs/node/pull/65073
----------------------------------- PR info ------------------------------------
Title      crypto: add mgf1Hash for RSA-OAEP (#65073)
   ⚠  Could not retrieve the email or name of the PR author's from user's GitHub profile!
Branch     adamjmcgrath:add-mgf1hash -> nodejs:main
Labels     crypto, test, author ready, needs-ci, commit-queue, commit-queue-squash, lts-watch-v24.x
Commits    3
 - crypto: add mgf1Hash for RSA-OAEP
 - crypto: fix lint and tests
 - test: update test
Committers 1
 - Adam Mcgrath <adam.mcgrath@okta.com>
PR-URL: https://github.com/nodejs/node/pull/65073
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/65073
Reviewed-By: Filip Skokan <panva.ip@gmail.com>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Thu, 06 Aug 2026 08:23:40 GMT
   ✔  Approvals: 1
   ✔  - Filip Skokan (@panva) (TSC): https://github.com/nodejs/node/pull/65073#pullrequestreview-4875510214
   ✘  This PR needs to wait 82 more hours to land (or 0 minutes if there is one more approval)
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2026-08-06T17:55:09Z: https://ci.nodejs.org/job/node-test-pull-request/75581/
- Querying data for job/node-test-pull-request/75581/
✔  Build data downloaded
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /Users/avivkeller/Documents/projects/nodejs/node/.ncu

/nodejs/node/actions/runs/

[avivkeller]

Landed in 11423b1