OpsCenter restore fails with "UnrecoverableKeyException: Password verification failed"

Summary

This article discusses an issue with restoring data on clusters where passwords in configuration files are encrypted.

Applies to

• OpsCenter 6.7.0
• OpsCenter 6.5.3 or earlier
• OpsCenter 6.1 

Symptoms

Attempts to restore data via OpsCenter fails with UnrecoverableKeyException. Here is an example stack trace from OpsCenter 6.1.2:

mykeyspace: Unknown error when running sstableloader. \
  If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. \
  Please report the error to DataStax support:: Unknown error when running sstableloader. If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. Please report the error to DataStax support:
java.lang.RuntimeException: Could not create SSL Context.
        at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:280)
        at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
        at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:60)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:38)
Caused by: java.io.IOException: Error creating the initializing the SSL Context
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:201)
        at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:276)
        ... 7 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:171)
        ... 8 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:783)
        ... 13 more

Cause

OpsCenter uses the sstableloader utility to perform restores. The UnrecoverableKeyException is caused by a bug with sstableloader not decrypting the keystore password in cassandra.yaml (DSP-13492). Because it uses the password in its encrypted form, the JavaKeyStore class reports an IOException with the message "Keystore was tampered with, or password was incorrect".

Solution

DSP-13492 was fixed in DataStax Enterprise 5.1.11, 6.0.1, and is included in 6.7.0.

In the special case where all the following are in use:

• internode encryption (server_encryption_options.internode_encryption),
• client/server encryption (client_encryption_options.enabled), and
• configuration encryption (config_encryption_active),

then it will also be necessary to apply the fix for DSP-17536 included in DataStax Enterprise 5.1.12, 6.0.5, and 6.7.1.

OPSC-12312 was fixed in OpsCenter 6.5.4 and 6.7.1.

To get the latest fixes and improvements, upgrade to the latest version of OpsCenter.

See also

Support KB - sstableloader fails with "UnrecoverableKeyException: Password verification failed"