Summary
This article discusses an issue with restoring data on clusters where passwords in configuration files are encrypted.
Applies to
- OpsCenter 6.7.0
- OpsCenter 6.5.3 or earlier
- OpsCenter 6.1
Symptoms
Attempts to restore data via OpsCenter fails with UnrecoverableKeyException
. Here is an example stack trace from OpsCenter 6.1.2:
mykeyspace: Unknown error when running sstableloader. \ If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. \ Please report the error to DataStax support:: Unknown error when running sstableloader. If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. Please report the error to DataStax support: java.lang.RuntimeException: Could not create SSL Context. at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:280) at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83) at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:38) Caused by: java.io.IOException: Error creating the initializing the SSL Context at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:201) at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:276) ... 7 more Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785) at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) at java.security.KeyStore.load(KeyStore.java:1445) at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:171) ... 8 more Caused by: java.security.UnrecoverableKeyException: Password verification failed at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:783) ... 13 more
Cause
OpsCenter uses the sstableloader
utility to perform restores. The UnrecoverableKeyException
is caused by a bug with sstableloader
not decrypting the keystore password in cassandra.yaml
(DSP-13492). Because it uses the password in its encrypted form, the JavaKeyStore
class reports an IOException
with the message "Keystore was tampered with, or password was incorrect".
Solution
DSP-13492 was fixed in DataStax Enterprise 5.1.11, 6.0.1, and is included in 6.7.0.
In the special case where all the following are in use:
- internode encryption (server_encryption_options.internode_encryption),
- client/server encryption (client_encryption_options.enabled), and
- configuration encryption (config_encryption_active),
then it will also be necessary to apply the fix for DSP-17536 included in DataStax Enterprise 5.1.12, 6.0.5, and 6.7.1.
OPSC-12312 was fixed in OpsCenter 6.5.4 and 6.7.1.
To get the latest fixes and improvements, upgrade to the latest version of OpsCenter.
See also
Support KB - sstableloader fails with "UnrecoverableKeyException: Password verification failed"