Summary
This article discusses an issue with bulk-loading data on clusters where passwords in configuration files are encrypted.
Applies to
- DataStax Enterprise 6.7.0
- DataStax Enterprise 6.0.0
- DataStax Enterprise 5.1.11 or earlier
- DataStax Enterprise 5.0.14 or earlier
Symptoms
Attempts to run sstableloader on a node fails with UnrecoverableKeyException. Here is an example output:
mykeyspace: Unknown error when running sstableloader. \
If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. \
Please report the error to DataStax support:: Unknown error when running sstableloader. If this error is related to sstableloader running out of memory, please try increasing the setting sstableloader_max_heap_size. Please report the error to DataStax support:
java.lang.RuntimeException: Could not create SSL Context.
at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:280)
at org.apache.cassandra.tools.BulkLoader.load(BulkLoader.java:83)
at org.apache.cassandra.tools.BulkLoader.main(BulkLoader.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.datastax.bdp.tools.ShellToolWrapper.main(ShellToolWrapper.java:38)
Caused by: java.io.IOException: Error creating the initializing the SSL Context
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:201)
at org.apache.cassandra.tools.BulkLoader.buildSSLOptions(BulkLoader.java:276)
... 7 more
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:785)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:171)
... 8 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:783)
... 13 more
Cause
The UnrecoverableKeyException is due to a bug with sstableloader, not decrypting the keystore password in cassandra.yaml (DSP-13492). Since sstableloader uses the password in its encrypted form, the JavaKeyStore class reports an IOException with the message "Keystore was tampered with, or password was incorrect".
Solution
DSP-13492 was fixed in DataStax Enterprise 5.1.11, 6.0.1, and is included in 6.7.0.
In the special case where all the following are in use:
- internode encryption (server_encryption_options.internode_encryption)
- client/server encryption (client_encryption_options.enabled)
- configuration encryption (config_encryption_active)
You must also apply the fix for DSP-17536 included in DataStax Enterprise 5.1.12, 6.0.5, and 6.7.1.
To get the latest fixes and improvements, upgrade to the latest version of DataStax Enterprise 5.1, 6.0 or 6.7.
See also
Support KB - OpsCenter restore fails with "UnrecoverableKeyException: Password verification failed"