Summary
This article discusses an issue with restoring data on clusters where passwords in configuration files are encrypted.
Applies to
- OpsCenter 6.7.0
- OpsCenter 6.5.3 or earlier
- OpsCenter 6.1
Symptoms
Attempts to restore data via OpsCenter fails with UnrecoverableKeyException. Here is an example stack trace from OpsCenter 6.1.2:
mykeyspace: Unknown error when running sstableloader. \
If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. \
Please report the error to DataStax support:: Unknown error when running sstableloader. If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. Please report the error to DataStax support:
java.lang.RuntimeException: Could not create SSL Context.
at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:280)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:38)
Caused by: java.io.IOException: Error creating the initializing the SSL Context
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:201)
at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:276)
... 7 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:171)
... 8 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:783)
... 13 more
Cause
OpsCenter uses the sstableloader utility to perform restores. The UnrecoverableKeyException is caused by a bug with sstableloader not decrypting the keystore password in cassandra.yaml (DSP-13492). Because it uses the password in its encrypted form, the JavaKeyStore class reports an IOException with the message "Keystore was tampered with, or password was incorrect".
Solution
DSP-13492 was fixed in DataStax Enterprise 5.1.11, 6.0.1, and is included in 6.7.0.
In the special case where all the following are in use:
- internode encryption (server_encryption_options.internode_encryption),
- client/server encryption (client_encryption_options.enabled), and
- configuration encryption (config_encryption_active),
then it will also be necessary to apply the fix for DSP-17536 included in DataStax Enterprise 5.1.12, 6.0.5, and 6.7.1.
OPSC-12312 was fixed in OpsCenter 6.5.4 and 6.7.1.
To get the latest fixes and improvements, upgrade to the latest version of OpsCenter.
See also
Support KB - sstableloader fails with "UnrecoverableKeyException: Password verification failed"