Summary
This article discusses a scenario where DSE fails to start with yaml parsing error, but the error is actually a JCE issue.
Symptoms
Datastax Enterprise (DSE) fails to start:
[user@ip ~]$ sudo service dse restart OKtarting DSE daemon : dse [ OK ] DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features) WARNING: Timed out while waiting for DSE to start. The service may have started though. Please check your logs
Applies to
- All versions of DSE and DDAC
Symptoms
- DSE cannot be started
- According to the system or debug log, the cassandra.yaml file cannot be parsed:
INFO [main] 2019-12-27 14:02:06,381 YamlConfigurationLoader.java:89 - Configuration location: file:/etc/dse/cassandra/cassandra.yaml DEBUG [main] 2019-12-27 14:02:06,381 YamlConfigurationLoader.java:110 - Loading settings from file:/etc/dse/cassandra/cassandra.yaml ERROR [main] 2019-12-27 14:02:06,503 DseModule.java:100 - {}. Exiting... com.google.inject.CreationException: Unable to create injector, see the following errors: 1) An exception was caught and reported. Message: Invalid yaml: file:/etc/dse/cassandra/cassandra.yaml Error: null; Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=java.lang.reflect.InvocationTargetException; in 'reader', line 9, column 1: cluster_name: test ^ at com.datastax.bdp.DseModule.configure(Unknown Source) 1 error at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)
Cause
The yaml parser exception handling hides the actual issue leading to this problem.
In this case, the issue was related to permissions on the US_export_policy.jar java cryptographic file:
[user@ip ~]$ locate US_export_policy.jar | xargs ls -ltr -rw------- 1 root root 538 Jan 1 2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar -rw------- 1 root root 538 Jan 1 2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/limited/US_export_policy.jar
To troubleshoot the issue, run a native Java tool such as sstabledump to bypass the yaml parser that fails at startup and hides the root cause:
[user@ip ~]$ sstabledump /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/aa-12-bti-Data.db Exception in thread "main" java.lang.ExceptionInInitializerError at javax.crypto.JceSecurityManager.<clinit>(JceSecurityManager.java:65) ... Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:93) ... 27 more Caused by: java.security.PrivilegedActionException: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar (Permission denied) at java.security.AccessController.doPrivileged(Native Method)
Solution
Allow non-root users to read the security files. For example:
chmod 644 ProblematicFile
See an example based on OpenJDK. Path will vary based on JDK used and version:
[user@ip ~]$ locate US_export_policy.jar | xargs sudo chmod 644 [user@ip ~]$ locate US_export_policy.jar | xargs ls -ltr -rw-r--r-- 1 root root 538 Jan 1 2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar -rw-r--r-- 1 root root 538 Jan 1 2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/limited/US_export_policy.jar
Troubleshooting side note:
During the investigation, it was also confirmed the cassandra.yaml file was not in the supported Unix file format. In this instance, the file was in Windows format, which is not supported.
To verify and set the file format:
1. Open your file in Vim and in normal mode type:
:set ff?
to see what the file format is.
2. If the file format is DOS, then type :set fileformat=unix
to change the file format to Unix and save the file.
3. When done, quit the text editor and restart DSE.