DSE cannot start with yaml parsing error

Summary

This article discusses a scenario where DSE fails to start with yaml parsing error, but the error is actually a JCE issue.

Symptoms

Datastax Enterprise (DSE) fails to start:

[user@ip ~]$ sudo service dse restart
 OKtarting DSE daemon : dse                                [  OK  ]
DSE daemon starting with only Cassandra enabled (edit /etc/default/dse to enable other features)
WARNING: Timed out while waiting for DSE to start. The service may have started though. Please check your logs

Applies to

• All versions of DSE and DDAC

Symptoms

• DSE cannot be started
• According to the system or debug log, the cassandra.yaml file cannot be parsed:

INFO  [main] 2019-12-27 14:02:06,381  YamlConfigurationLoader.java:89 - Configuration location: file:/etc/dse/cassandra/cassandra.yaml
DEBUG [main] 2019-12-27 14:02:06,381  YamlConfigurationLoader.java:110 - Loading settings from file:/etc/dse/cassandra/cassandra.yaml
ERROR [main] 2019-12-27 14:02:06,503  DseModule.java:100 - {}. Exiting...
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) An exception was caught and reported. Message: Invalid yaml: file:/etc/dse/cassandra/cassandra.yaml
 Error: null; Can't construct a java object for tag:yaml.org,2002:org.apache.cassandra.config.Config; exception=java.lang.reflect.InvocationTargetException;  in 'reader', line 9, column 1:
    cluster_name: test
    ^
  at com.datastax.bdp.DseModule.configure(Unknown Source)

1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466)

Cause

The yaml parser exception handling hides the actual issue leading to this problem.
In this case, the issue was related to permissions on the US_export_policy.jar java cryptographic file:

[user@ip ~]$ locate US_export_policy.jar | xargs ls -ltr
-rw------- 1 root root 538 Jan  1  2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar
-rw------- 1 root root 538 Jan  1  2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/limited/US_export_policy.jar

To troubleshoot the issue, run a native Java tool such as sstabledump to bypass the yaml parser that fails at startup and hides the root cause:

[user@ip ~]$ sstabledump /var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/aa-12-bti-Data.db 
Exception in thread "main" java.lang.ExceptionInInitializerError
        at javax.crypto.JceSecurityManager.<clinit>(JceSecurityManager.java:65)
...
Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism
        at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:93)
        ... 27 more
Caused by: java.security.PrivilegedActionException: java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar (Permission denied)
        at java.security.AccessController.doPrivileged(Native Method)

Solution

Allow non-root users to read the security files. For example:
chmod 644 ProblematicFile
See an example based on OpenJDK. Path will vary based on JDK used and version:

[user@ip ~]$ locate US_export_policy.jar | xargs sudo chmod 644
[user@ip ~]$ locate US_export_policy.jar | xargs ls -ltr
-rw-r--r-- 1 root root 538 Jan  1  2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/unlimited/US_export_policy.jar
-rw-r--r-- 1 root root 538 Jan  1  2014 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre/lib/security/policy/limited/US_export_policy.jar

Troubleshooting side note:

During the investigation, it was also confirmed the cassandra.yaml file was not in the supported Unix file format. In this instance, the file was in Windows format, which is not supported.

To verify and set the file format:

1. Open your file in Vim and in normal mode type: 

:set ff? to see what the file format is.

2. If the file format is DOS, then type :set fileformat=unix to change the file format to Unix and save the file.

3. When done, quit the text editor and restart DSE.