sstableloader fails with "UnrecoverableKeyException: Password verification failed"

Summary

This article discusses an issue with bulk-loading data on clusters where passwords in configuration files are encrypted.

Applies to

• DataStax Enterprise 6.7.0
• DataStax Enterprise 6.0.0
• DataStax Enterprise 5.1.11 or earlier
• DataStax Enterprise 5.0.14 or earlier

Symptoms

Attempts to run sstableloader on a node fails with UnrecoverableKeyException. Here is an example output:

mykeyspace: Unknown error when running sstableloader. \
  If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. \
  Please report the error to DataStax support:: Unknown error when running sstableloader. If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. Please report the error to DataStax support:
java.lang.RuntimeException: Could not create SSL Context.
        at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:280)
        at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
        at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:60)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:38)
Caused by: java.io.IOException: Error creating the initializing the SSL Context
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:201)
        at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:276)
        ... 7 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:171)
        ... 8 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:783)
        ... 13 more

Cause

The UnrecoverableKeyException is due to a bug with sstableloader, not decrypting the keystore password in cassandra.yaml (DSP-13492). Since sstableloader uses the password in its encrypted form, the JavaKeyStore class reports an IOException with the message "Keystore was tampered with, or password was incorrect".

Solution

DSP-13492 was fixed in DataStax Enterprise 5.1.11, 6.0.1, and is included in 6.7.0.

In the special case where all the following are in use:

• internode encryption (server_encryption_options.internode_encryption)
• client/server encryption (client_encryption_options.enabled)
• configuration encryption (config_encryption_active)

You must also apply the fix for DSP-17536 included in DataStax Enterprise 5.1.12, 6.0.5, and 6.7.1.

To get the latest fixes and improvements, upgrade to the latest version of DataStax Enterprise 5.1, 6.0 or 6.7.

See also

Support KB - OpsCenter restore fails with "UnrecoverableKeyException: Password verification failed"