SSL for JMX with RMI

We have a Java application which has had a JConsole connection with password authentication for a while. In improving the security of this, we are trying to encrypt the connection made from JConsole to the application.

Up until now, we have launched our application with the following launch command:

java -Dcom.sun.management.jmxremote 
     -Dcom.sun.management.jmxremote.port=1099 
     -Dcom.sun.management.jmxremote.rmi.port=1099 
     -Dcom.sun.management.jmxremote.authenticate=true 
     -Dcom.sun.management.jmxremote.password.file=jmx.password 
     -Dcom.sun.management.jmxremote.access.file=jmx.access 
     -Dcom.sun.management.jmxremote.ssl=false
     -jar MyApplication.jar

With this, we can flawlessly access the JMX methods of MyApplication via both JConsole, jmxterm, and other Java applications. In JConsole and jmxterm, we can use both hostname:1099 and service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi without issues. From the Java applications, we always use service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi, again without issues. Our application has no code-based setup of the JMX endpoint (we exposes some methods and attributes, but we did not touch the registry and socket factories).

Now we are trying to set up SSL between our application, and all other parties, following www.cleantutorials.com/jconsole/jconsole-ssl-with-password-authentication. Doing this, we have a keystore and truststore for both MyApplication and whoever the client connection to the JMX methods is. We use

java -Dcom.sun.management.jmxremote 
     -Dcom.sun.management.jmxremote.port=1099 
     -Dcom.sun.management.jmxremote.rmi.port=1099 
     -Dcom.sun.management.jmxremote.authenticate=true 
     -Dcom.sun.management.jmxremote.password.file=jmx.password 
     -Dcom.sun.management.jmxremote.access.file=jmx.access 
     -Dcom.sun.management.jmxremote.ssl=true 
     -Dcom.sun.management.jmxremote.ssl.need.client.auth=true 
     -Dcom.sun.management.jmxremote.registry.ssl=true 
     -Djava.rmi.server.hostname=hostname 
     -Djavax.net.ssl.keyStore=server-jmx-keystore 
     -Djavax.net.ssl.keyStorePassword=password 
     -Djavax.net.ssl.trustStore=server-jmx-truststore 
     -Djavax.net.ssl.trustStorePassword=password 
     -jar MyApplication.jar

After this, almost all our connections fail. The only one succeeding, is via JConsole (adding the client keystore and truststores to the launch config), and only using hostname:1099. Using the address service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi no longer works, not via JConsole, not via jmxterm, and not via other applications.

We have tried about any combination of launch settings we could think of, but nothing that we find anywhere seems to work. The error we see when trying to connect from e.g. jmxterm is:

java.rmi.ConnectIOException: non-JRMP server at remote endpoint

(I can provide the full stack if needed).

We’re a bit at a loss on how to continue, what we can do to make all connections that used to work, now work. What should we do to enable connecting with service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi-like connection strings via SSL?

If relevant, this application is using OpenJDK 11.0.5, other applications where we might need this run on OpenJDK 8.

Edit

Debugging both the JConsole client and the backend side, it seems that the protocol that the client is trying to establish is not known in the SSL context. On the backend, we have the following error:

javax.net.ssl|DEBUG|20|RMI TCP Connection(1)|2021-12-28 10:04:04.265 CET|null:-1|Raw read (
  0000: 4A 52 4D 49 00
                JRMI.
)
javax.net.ssl|ERROR|20|RMI TCP Connection(1)|2021-12-28 10:04:04.267 CET|null:-1|Fatal (UNEXPECTED_MESSAGE): Unsupported or unrecognized SSL message (
"throwable" : {
  javax.net.ssl.SSLException: Unsupported or unrecognized SSL message
          at java.base/sun.security.ssl.SSLSocketInputRecord.handleUnknownRecord(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketInputRecord.decode(Unknown Source)
          at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(Unknown Source)
          at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source)
          at java.base/java.io.BufferedInputStream.fill(Unknown Source)
          at java.base/java.io.BufferedInputStream.read(Unknown Source)
          at java.base/java.io.DataInputStream.readInt(Unknown Source)
          at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
          at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(Unknown Source)
          at java.base/java.security.AccessController.doPrivileged(Native Method)
          at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at java.base/java.lang.Thread.run(Unknown Source)}
)

After which the backend closes the connection.

Based on some tutorials online, it should be possible to get the SSL connection working using the service-based URL, but we can’t get it to work.

Answer

After a long search, with a lot of debugging, trial and error, we came to the conclusion that there is no out-of-the-box solution in Spring (Boot) to enable SSL with an RMI registry and a JMX connection server. This had to be configured manually. We used the following Spring configuration class that did the trick:

@Configuration
@EnableMBeanExport
public class JMXConfig {

    private static final Log LOG = LogFactory.getLog(JMXConfig .class);

    @Value("${jmx.registry.port:1098}")
    private Integer registryPort;
    @Value("${jmx.rmi.port:1099}")
    private Integer rmiPort;

    @Bean
    public RmiRegistryFactoryBean rmiRegistry() {
        final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
        rmiRegistryFactoryBean.setPort(rmiPort);
        rmiRegistryFactoryBean.setAlwaysCreate(true);

        LOG.info("Creating RMI registry on port " + rmiRegistryFactoryBean.getPort());
        return rmiRegistryFactoryBean;
    }

    @Bean
    @DependsOn("rmiRegistry")
    public ConnectorServerFactoryBean connectorServerFactoryBean() throws MalformedObjectNameException {
        String rmiHost = getHost();
        String serviceURL = serviceURL(rmiHost);
        LOG.info("Creating JMX connection for URL " + serviceURL);

        final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
        connectorServerFactoryBean.setObjectName("connector:name=rmi");
        connectorServerFactoryBean.setEnvironmentMap(createRmiEnvironment(rmiHost));
        connectorServerFactoryBean.setServiceUrl(serviceURL);
        return connectorServerFactoryBean;
    }

    private String getHost() {
        try {
            InetAddress localMachine = InetAddress.getLocalHost();
            return localMachine.getCanonicalHostName();
        } catch (UnknownHostException e) {
            LOG.warn("Unable to get hostname, using localhost", e);
            return "localhost";
        }
    }

    private String serviceURL(String rmiHost) {
        return format("service:jmx:rmi://%s:%s/jndi/rmi://%s:%s/jmxrmi", rmiHost, registryPort, rmiHost, rmiPort);
    }

    private Map<String, Object> createRmiEnvironment(String rmiHost) {
        final Map<String, Object> rmiEnvironment = new HashMap<>();
        rmiEnvironment.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.rmi.registry.RegistryContextFactory");
        rmiEnvironment.put(Context.PROVIDER_URL, "rmi://" + rmiHost + ":" + rmiPort);
        rmiEnvironment.put(RMIConnectorServer.RMI_CLIENT_SOCKET_FACTORY_ATTRIBUTE, new SslRMIClientSocketFactory());
        rmiEnvironment.put(RMIConnectorServer.RMI_SERVER_SOCKET_FACTORY_ATTRIBUTE, new SslRMIServerSocketFactory());
        return rmiEnvironment;
    }
}

This enables SSL using the connection details service:jmx:rmi:///jndi/rmi://hostname:1099/jmxrmi. To make it work, you need to add a keystore/password to your backend, and a truststore/password to your frontend (as in the tutorial).