Summary

When using Kerberos authentication for DSE, if the hostname is not derived by DSE to match the configured service principal hostnames, then the login to cqlsh and other clients will fail

Symptoms

After validating the following:

• listprincs shows all the correct relevant service principals
• The user is able to kinit from the DSE node
• The keytab has the correct permissions for the DSE user (usually cassandra)
• The keytab has the correct service principals as configured in the KDC server
• The dse.yaml has all the relevant correct Kerberos configuration

Using the --debug switch the following kind of error may be observed

$ cqlsh 10.1.2.3 --debug 
Using CQL driver: <module 'cassandra' from '/usr/share/dse/cassandra/lib/cassandra-driver-internal-only-3.6.0.zip/cassandra-driver-3.6.0/cassandra/__init__.py'> 
Using connect timeout: 5 seconds 
Using 'utf-8' encoding 
Using DSEGSSAPIAuthProvider(service=cassandra, qops=auth) 
This will only be used if the server requests kerberos authentication 
Connection error: ('Unable to connect to any servers', {'10.1.2.3': AuthenticationFailed('Failed to authenticate to 10.1.2.3: Error from server: code=0000 [Server error] message="java.lang.RuntimeException: javax.security.auth.login.LoginException: Unable to obtain password from user\n"',)})

Cause

Normally the dse.yaml will have the following configuration for the Kerberos service principals (where <REALM>is your required Kerberos realm)

kerberos_options:
  keytab: /etc/dse/dse.keytab
  service_principal: dse/_HOST@<REALM>
  http_principal: HTTP/_HOST@<REALM>
  qop: auth  

Note the _HOST placeholder above. This will normally be derived from a reverse DNS lookup of the broadcast ip address to give a fully qualified domain name (fqdn) of the node . So for example nodeA19.datastax.com would have a service principal of dse/nodeA19.datastax.com@REALM1.DATASTAX.ORG however if the fqdn isn’t derived correctly then it could mean the service princpal passed up to the KDC server is wrong.

Workaround

In such cases the dse.yaml can be configured with the fqdn instead of using the placeholder, for example:

kerberos_options:
  keytab: /etc/dse/dse.keytab
  service_principal: dse/nodeA19.datastax.com@REALM1.DATASTAX.ORG
  http_principal: HTTP/nodeA19.datastax.com@REALM1.DATASTAX.ORG
  qop: auth  

Solution

Ensure your service principals match the expected fqdn of the node. Typically on most Linux distros nslookup <ip address> will give you that.