RSA decryption according to the private key reports an error

My code using RSA on the front end：

const rsa = new JSEncrypt();
rsa.setPublicKey(k);
const resultText = rsa.encrypt("violet");
console.log(resultText);

My code using RSA in the backend：

byte[] inputByte = org.apache.commons.codec.binary.Base64.decodeBase64(str.getBytes("UTF-8"));
byte[] decoded = org.apache.commons.codec.binary.Base64.decodeBase64(privateKey);
PrivateKey priKey = KeyFactory.getInstance("RSA").generatePrivate(new 
PKCS8EncodedKeySpec(decoded));
Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.DECRYPT_MODE,priKey);
String outStr=new String(cipher.doFinal(inputByte));
return outStr;

PublicKey like this:

    -----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA13gYdCmOjR9yqQD7ldzG
ZXabSon6SiLceCK6vRXf4NMbF+EQke0vRpqU3IZ/S1pFdvoQswQabsA4zf0WACVT
iaGIhWDlPu3mecri8rYtmOSfd8GCE0vEgFNvSD6IXRLPeLCB+i7WENBa4fCEtW8W
Hzdas96CLiESbjSAruRasQXP2OLqEA2GU83/069vh8uRKzui+yw0aAXZFyFyFRFa
lxYltFadVpz3+kBplvpzuj82t4fc3yCRbrpeRyTyX1sz0ULSxx/k3/p1OuJtIq9Y
9uN0G4gxhcDFJ4L41uXOln5CPapk7tlsYobhhvxYHw1rrweY+06hrQ7r0Hblv2nH
GQIDAQAB
-----END PUBLIC KEY-----

PrivateKey like this:

    -----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA13gYdCmOjR9yqQD7ldzGZXabSon6SiLceCK6vRXf4NMbF+EQ
ke0vRpqU3IZ/S1pFdvoQswQabsA4zf0WACVTiaGIhWDlPu3mecri8rYtmOSfd8GC
E0vEgFNvSD6IXRLPeLCB+i7WENBa4fCEtW8WHzdas96CLiESbjSAruRasQXP2OLq
EA2GU83/069vh8uRKzui+yw0aAXZFyFyFRFalxYltFadVpz3+kBplvpzuj82t4fc
3yCRbrpeRyTyX1sz0ULSxx/k3/p1OuJtIq9Y9uN0G4gxhcDFJ4L41uXOln5CPapk
7tlsYobhhvxYHw1rrweY+06hrQ7r0Hblv2nHGQIDAQABAoIBAAyqFmXde294BblB
QYhRbafRDNeYvIlW+zZkdC1g98OzJMiGhf7NvhWcSFud3CWFrMeNcyXSe+s+iRgy
Y/SmPP6969RLGa5VNVK7RhOV+aAe7/COAyM3NNmGDehlJIaz8FXbqggWcKaUWIMn
K+WuHdK/4ijoTyZ+8yJfG6Er8tisryLQ9io9+op9g/ZWzaUKgu934/cDxUt70bfm
x+ZEPi6YfkJ1uOpXnnadDyw2RUDcvCM3fK3KF5fqM7SJAXY9b1pmLr+Ccn1qkT9G
I+QHidEsGfJciX5AoHnlIMLPMVIPKBbq4GwC/Ngb41LprNJWlPR38N2ySjky/Jyt
159XWHECgYEA9lx2KfFmyLyVjnkIF3JI50mSZAw4YPBBqB27UInacvxXbjfVURht
xK60GB9OkDbFdeNh89x86Gfwvm5bTq4W8YSH4Obd5Fg8XjTuaicTi03CSfF5SdJn
JLLOUmlqP75gkbEPNUoOfqhqq6IbyJVB3egyL90cd2/wCdJOVLEUly8CgYEA3+Y4
lNdl0xedkDNkWsCyyA4iPSUzVxsuch9hW/VGBwzga8rtiNllpKifVvctQOEu/KUe
vVQRF78ojZaMGT33l6TivnUL54z9Lo9uWghoG8TqMfnG34pFPe3R+zvGP87Hrozw
1EUhiMT198SlB/YHrgGGGlJbG+rlm5GIx3lEdDcCgYA4RSw0LlA0v2ZFkX14pYDj
WxmVwnjKI3ZLqObU4XfE1cA+i4AssrC3wNOfwt7V77ywTYxc/9qD1uHVDS3LzdWt
uoCyrOi3tDOtrNdb5asAIXWkIAR9CRLH/hNEHZHIF3rFLDT2DgE7iso6g59m9DiE
L/nulsleunGQPLnpfDzgvwKBgDRV5Q3tl3CTSZJGYQPRnTikDR7LzkdjJCUq7qAH
IhpNyTuJEKL3ZgnqHGzAlERhHpGRqzDIMMKjPUBzW0YfNPuuYA3y4Bh83UV/42SK
KIOtMK0D3JeuA2dparbWDw4lMIm9iiGkEyWcHH6Q6Z6FxN/InWcTrxZEfu0xRI6T
6wtbAoGAfl5dW9LNoaNfQbgIq+PJKZm9f1frza55mFTJgo3ravdb3GmzWVHN8xRf
nLKyKyNLqbkT35IGc39NkALJLxT5RibkAZLiUiwqdMF63sgODbA9AGTmhe+JHS+V
hBmFnCyp6UiN9E4ZAWcZQILa0rRMftMFngAJ3El0ZP+HziRnNzs=
-----END RSA PRIVATE KEY-----

Bu-t, when i do the java code decryption, it reported such an error：

java.security.InvalidKeyException: IOException : DerInputStream.getLength(): lengthTag=111, too big.

How can i solve this problem ?

Answer

1. You are decoding wrong. PEM format has a dash-BEGIN line identifying the type of data, a block of base64 encoding the data, and a dash-END line. The BEGIN and END lines are part of the format, but they do not contain base64-encoded data; only the lines in between contain the base64-encoded data. You are apparently passing the whole thing, including the BEGIN and END lines, to commons.codec.Base64, which results in decoding a bunch of garbage before and after the actual data. That garbage isn’t valid ASN.1 DER, so when Java tries to parse it as DER it fails.

2. Plus your data is not a PKCS8-clear privatekey. The PEM type ‘RSA PRIVATE KEY’ is an OpenSSL-defined format that contains a ‘traditional’ or ‘legacy’ format, namely the PKCS1 representation of the private key. This is not PKCS8, which is the only key format Java supports natively; that’s why the spec class is named PKCS8EncodedKeySpec, because it is a key spec encoded as PKCS8 and more specifically PKCS8-clear. If you fix the above problem by removing the BEGIN and END lines before base64-decoding, Java can parse the result as DER, but not as a PKCS8-clear key; you get a different exception about ‘algid parse error, not a sequence’. To fix this there are 5 approaches:

• change whatever process you use to initially generate the keypair so it generates PKCS8, not OpenSSL-legacy PKCS1. Especially since you need anyway to replace the keypair you compromised by publishing it, as 207421 said. You give no clue what that process is or was, so I can’t give any details.

• convert your generated privatekey, or a copy, to PKCS8-clear. This is not programming or development and offtopic, but if you have or get OpenSSL (on the same or any accessible and secure system), you can do

  openssl pkey -in oldfile -out newfile   # 1.0.0 up only, but older is now rare
  # or
  openssl pkcs8 -topk8 -nocrypt -in oldfile -out newfile   # even ancient versions
  

Once you have a PKCS8-clear file, just remove the BEGIN and END lines and base64-decode what is left, and pass that to KeyFactory as PKCS8EncodedKeySpec as you already do.

• use https://www.bouncycastle.org . The ‘bcpkix’ jar has (Java) code to read a large range of OpenSSL-supported PEM formats, including the RSA-PKCS1 private key format you have. There are lots of existing Qs about this; just search for PEMParser and JcaPEMKeyConverter.

• convert it yourself. Decode the body of the file you have, after removing the BEGIN and END lines, to get the PKCS1 key, then build the PKCS8 format for that key, and then pass it to KeyFactory as PKCS8EncodedKeySpec. See answers by Noa Resare and Jean-Alexis Aufauvre on Getting RSA private key from PEM BASE64 Encoded private key file or mine in Java: Convert DKIM private key from RSA to DER for JavaMail .

• do it entirely yourself. Decode the file you have without BEGIN/END to get PCKS1, parse that as DER following e.g. RFC8447, and build RSAPrivateCrtKeySpec. Some other As on the Q I linked just above do this. However, this requires either: using undocumented internal sun.* classes, which used to work in Java (hence the existing answers) but which ‘modular’ Java versions (9 up) since 2017 have steadily made more difficult or impossible; using BouncyCastle which has documented (and good) support for ASN.1 — but then it’s easier to use bcpkix for the whole job as above; or writing your own ASN.1 parsing, which is a good deal of work.

PS: encrypting text with RSA is usually a bad design; it’s not suited for that. But that’s not really a programming issue and doesn’t belong here.