Java Decrypt with NodeJS Encrypt padding and oaepHash

I need to decrypt some information coming from an external service built with NodeJS. This service ask for a RSA (2048) public key in pem format in base64, in order to encrypt the information.

I am creating the key pair in Java:

    KeyPairGenerator kpg = KeyPairGenerator.getInstance("RSA");
    kpg.initialize(2048);
    KeyPair kp = kpg.generateKeyPair();
    PublicKey pubkey = kp.getPublic();
    PrivateKey privkey = kp.getPrivate();

    String pemPublicString = "-----BEGIN PUBLIC KEY-----n";
    pemPublicString = pemPublicString+Base64.getEncoder().encodeToString(pubkey.getEncoded())+"n";
    pemPublicString = pemPublicString+"-----END PUBLIC KEY-----n";
    
    String pemPrivateString = "-----BEGIN RSA PRIVATE KEY-----n";
    pemPrivateString = pemPrivateString+Base64.getEncoder().encodeToString(privkey.getEncoded())+"n";
    pemPrivateString = pemPrivateString+"-----END RSA PRIVATE KEY-----n";
    
    //Send to node js service
    String base64publickey = Base64.getEncoder().encodeToString(pemPublicString.getBytes());

    //Store for decrypting
    String base64privatekey = Base64.getEncoder().encodeToString(pemPrivateString.getBytes());

The external service is encrypting the information as follows and is returning the bytes:

  crypto.publicEncrypt(
  {
    key: publicKey,
    padding: crypto.constants.RSA_PKCS1_OAEP_PADDING,
    oaepHash: "sha256",
  },
    dataToEncrypt
  );

I am trying to decrypt the information in Java as follows:

    public String decrypt(String payload, String privateKey){
      byte [] ciphertext = payload.getBytes(StandardCharsets.UTF_8);
      Cipher oaepFromInit = Cipher.getInstance("RSA/ECB/OAEPPadding");
      OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256", "MGF1", new 
      MGF1ParameterSpec("SHA-1"), PSource.PSpecified.DEFAULT);
      oaepFromInit.init(Cipher.DECRYPT_MODE, getRSAPrivateKeyFrom(privateKey), oaepParams);
      byte[] pt = oaepFromInit.doFinal(ciphertext);
      return new String(pt, StandardCharsets.UTF_8);
    }

    private PrivateKey getRSAPrivateKeyFrom(String content) {
      byte[] decodedBytes = Base64.getDecoder().decode(content);
      String decodedString = new String(decodedBytes);
      Security.addProvider(new BouncyCastleProvider());
      PEMParser pemParser = new PEMParser(new StringReader(decodedString));
      JcaPEMKeyConverter converter = new JcaPEMKeyConverter().setProvider("BC");
      Object object = pemParser.readObject();
      PrivateKey k = converter.getPrivateKey((PrivateKeyInfo) object);
      return k;
   }

Now I am getting a BadPadding Exception, any idea of what could be the problem? Thanks in advance.

Answer

The posted code does not show how the NodeJS code exports the ciphertext. The following line in decrypt() of the Java code:

 byte[] ciphertext = payload.getBytes(StandardCharsets.UTF_8);

implies that you have used a Utf-8 encoding. This is a common mistake that corrupts the ciphertext (see here). Instead of Utf-8, apply a binary-to-text enncoding, such as Base64.

The export of the ciphertext would then be done in NodeJS with:

var chiphertextBase64 = ciphertext.toString('base64');

and the import in Java with:

import java.util.Base64;
...
byte[] ciphertext = Base64.getDecoder().decode(payload);  

---

In the NodeJS code, OAEP (RSAES-OAEP) is specified as padding. crypto.publicEncrypt() applies with the parameter oaepHash the same digest for both, the OAEP and MGF1 digest. oaepHash: "sha256" thus specifies SHA256 for both digests.
In contrast, Java allows separate (and different) specification of OAEP and MGF1 digests. While SHA256 is used for the OAEP digest in decrypt(), SHA1 is applied for the MGF1 digest. The latter is inconsistent with the NodeJS code and needs to be changed to SHA256:

OAEPParameterSpec oaepParams = new OAEPParameterSpec("SHA-256", "MGF1", new MGF1ParameterSpec("SHA-256"), PSource.PSpecified.DEFAULT);

This bug was already suspected in the 1st comment by Maarten Bodewes.

---

In the posted code, the PEM encoded pemPublicString and pemPrivateString keys are Base64 encoded as base64publickey and base64privatekey respectively. This is not necessary because the body of a PEM encoded key is already Base64 encoded and header and footer consist of ASCII characters. A second Base64 encoding thus brings no advantage, but the disadvantage that the key data is unnecessarily enlarged (Base64 overhead: 33%, see here). On the other hand, if the service expects a double Base64 encoded public key, you must comply.

When generating keys, the posted Java code implicitly uses the PKCS#8 format for the private key. You can verify this with privkey.getFormat(), which will output PKCS#8. The associated header for PKCS#8 is -----BEGIN PRIVATE KEY----- and Footer -----END PRIVATE KEY-----. Both the header and footer you are currently using belong to a PEM encoded PKCS#1 private key and are thus inconsistent with the key data. This issue has already been addressed in the 2nd comment by Maarten Bodewes.

By the way, a PKCS#8 key can easily be imported without BouncyCastle using PKCS8EncodedKeySpec. For this only header, footer and line breaks have to be removed and the rest has to be Base64 decoded (DER encoding), s. e.g. here.