Setup SSL for Solr Cluster with internode and Client-to-node encryption

This is a step by step instruction and a live example to show how to setup SSL for Solr cluster with internode and Client-to-node encryption and how to use dsetool or https to create/reload Solr core and run Solr queries. 

1. Prepare the certificates and keystore, truststore by following the documentation in Preparing server certificates.

Note: it is required to set a valid dname/CN value to the IP address, FQDN or a wildcard DNS record such as  *.example.com of the node in the certificate in order for the dsetool or https to connect to.

• On node 0:

Generate a private and public key pair on each node of the cluster, create a new .keystore:

keytool -genkey -keyalg RSA -alias node0 -keystore .keystore -storepass tomcat -keypass tomcat -dname "CN=104.197.35.79, OU=None, O=None, L=None, C=None"

Export the public part of the certificate from the keystore:

keytool -export -alias node0 -file node0.cer -keystore .keystore

Add the node0.cer certificate to the truststore:

keytool -import -v -trustcacerts -alias node0 -file node0.cer -keystore .truststore

• Scp the node0.cer to each node in the cluster, and import it into the truststore on each node. 

2. Turn on Client-node and internode encryption in cassandra.yaml by following documentation Client-to-node Encryption. 

3. Using dsetool utility to manage the core:

• Create a core:

dsetool -h <IP address or hostname> create_core <keyspace>.<table> generateResources=true

Note: The IP address/hostname passed in -h should match the dname in certificate.

• Reload a core

dsetool -h <IP address or hostname> reload_core <keyspace>.<table> [options]

To upload a schema or Solr configuration change, provide the new schema.xml and solrconfig.xml in the command. Typically, a change to the Solr schema requires a full re-indexing.

4. Using Curl and HTTPS to manage the core.

When using Curl and HTTPS to access a SSL encrypted cluster, it is required to pass the server certificate to the command. When client authentication is turned on, ("require_client_auth=true" in cassandra.yaml), you would need to provide the client certificate files and this enables 2-way SSL authentication. The certificate files must be PEM format.

• Download the server certificate:

openssl s_client -showcerts –connect <Server IP address>:8983 > server_cert.pem

Note: The server IP address should match the dname in the certificate.

When there are multiple nodes in the cluster and each node has individual certificate, you could concatenate the certificates into one server cert PEM file. Then copy the PEM file to all of the nodes. 

• 2-way SSL: In order to generate the PEM files for client certificate and key, create a p12 keystore from the .keystore on the node, then use openssl to export a pem certificate and key from the p12 keystore:

  keytool -importkeystore -srckeystore .keystore -destkeystore node0.p12 -deststoretype PKCS12 -srcstorepass tomcat -deststorepass tomcat
  openssl pkcs12 -in node0.p12 -nokeys -out node0.cer.pem -passin pass:tomcat
  openssl pkcs12 -in node0.p12 -nodes -nocerts -out node0.key.pem -passin pass:tomcat

Note: The PEM files generated here can be used to setup cqlshrc as well. Please refer to Using cqlsh with SSL encryption for more information.

• Create a core: 

1-way SSL ("require_client_auth=false" in cassandra.yaml):

curl --cacert <path to server_cert.pem>/server_sert.pem "https://<server IP address or hostname>:8983/solr/admin/cores?action=CREATE&name=<keyspace>.<table>"

2-way SSL (require_client_auth=true" in cassandra.yaml):

You would need to provide client certificate and key file into the command. In our example, they are node0.cer.pem and node0.key.pem for node0 or node1.cer.pem and node1.key.pem for node1.

curl --cacert <path to server_cert.pem>/server_sert.pem --cert <path to client cer file>/node0.cer.pem --key <path to client key file>/node0.key.pem "https://<server IP address or hostname>:8983/solr/admin/cores?action=CREATE&name=<keyspace>.<table>"

• Upload the Solr resources:

1-way SSL:

curl --cacert<path to server_cert.pem>/server_sert.pem https://<server IP address or hostname>:8983/solr/resource/<keyspace>.<table>/solrconfig.xml --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'

2-way SSL:

curl --cacert<path to server_cert.pem>/server_sert.pem --cert <path to client cer file>/node0.cer.pem --key <path to client key file>/node0.key.pem https://<server IP address or hostname>:8983/solr/resource/<keyspace>.<table>/solrconfig.xml --data-binary @solrconfig.xml -H 'Content-type:text/xml; charset=utf-8'

• Reload a core:

1-way SSL:

curl --cacert<path to server_cert.pem>/server_sert.pem "https://<server IP address or hostname>:8983/solr/admin/cores?action=RELOAD&name=<keyspace>.<table>"

2-way SSL:

curl --cacert<path to server_cert.pem>/server_sert.pem --cert <path to client cer file>/node0.cer.pem --key <path to client key file>/node0.key.pem "https://<server IP address or hostname>:8983/solr/admin/cores?action=RELOAD&name=<keyspace>.<table>" 

5. Common Error:

curl: (52) Empty reply from server

The error occurs when using HTTP to access a SSL encrypted cluster.

Solution: Use HTTPS instead.

IOException occured when talking to server at: https://localhost:8983/solr
Caused by: javax.net.ssl.SSLException: hostname in certificate didn't match: <localhost> != <199.223.234.58>

The error occurs when the IP address or hostname used in dsetool -h doesn't match the dname in certificate

Solution: Provide the IP address or hostname available in the node certificates and the truststore.

curl: (35) Unknown SSL protocol error in connection to 104.197.35.79:8983

The error occurs when "require_client_auth=true", but the client certificate and key PEM files are not provided.

Solution: Provide the client certificate and key PEM files by using --cert <path to client cert>/node1.cer.pem --key <path to client key>/node1.key.pem 

curl: (60) SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
More details here: http://curl.haxx.se/docs/sslcerts.html

The error occurs when the server certificate is not provided in the curl request. 

Solution: Provide the server certificate in PEM file by using --cacert <path to server certificate>/server_cert.pem

It is not uncommon that you might have existing certificates which are used by applications, but the dname was set to a generic term. In order to use the cluster as a Solr cluster, it is recommended to regenerate the certificates which include an accessible IP address or hostname in dname. A temporary workaround is using -k in curl command to bypass the verification of the certificate. But it is not recommended due to security reason.