Summary
When using Kerberos authentication for DSE, if the hostname is not derived by DSE to match the configured service principal hostnames, then the login to cqlsh
and other clients will fail
Symptoms
After validating the following:
listprincs
shows all the correct relevant service principals- The user is able to
kinit
from the DSE node - The
keytab
has the correct permissions for the DSE user (usuallycassandra
) - The keytab has the correct service principals as configured in the KDC server
- The
dse.yaml
has all the relevant correct Kerberos configuration
Using the --debug
switch the following kind of error may be observed
$ cqlsh 10.1.2.3 --debug
Using CQL driver: <module 'cassandra' from '/usr/share/dse/cassandra/lib/cassandra-driver-internal-only-3.6.0.zip/cassandra-driver-3.6.0/cassandra/__init__.py'>
Using connect timeout: 5 seconds
Using 'utf-8' encoding
Using DSEGSSAPIAuthProvider(service=cassandra, qops=auth)
This will only be used if the server requests kerberos authentication
Connection error: ('Unable to connect to any servers', {'10.1.2.3': AuthenticationFailed('Failed to authenticate to 10.1.2.3: Error from server: code=0000 [Server error] message="java.lang.RuntimeException: javax.security.auth.login.LoginException: Unable to obtain password from user\n"',)})
Cause
Normally the dse.yaml
will have the following configuration for the Kerberos service principals (where <REALM>
is your required Kerberos realm)
kerberos_options:
keytab: /etc/dse/dse.keytab
service_principal: dse/_HOST@<REALM>
http_principal: HTTP/_HOST@<REALM>
qop: auth
Note the _HOST
placeholder above. This will normally be derived from a reverse DNS lookup of the broadcast ip address to give a fully qualified domain name (fqdn) of the node . So for example nodeA19.datastax.com
would have a service principal of dse/nodeA19.datastax.com@REALM1.DATASTAX.ORG
however if the fqdn isn’t derived correctly then it could mean the service princpal passed up to the KDC server is wrong.
Workaround
In such cases the dse.yaml
can be configured with the fqdn instead of using the placeholder, for example:
kerberos_options:
keytab: /etc/dse/dse.keytab
service_principal: dse/nodeA19.datastax.com@REALM1.DATASTAX.ORG
http_principal: HTTP/nodeA19.datastax.com@REALM1.DATASTAX.ORG
qop: auth
Solution
Ensure your service principals match the expected fqdn of the node. Typically on most Linux distros nslookup <ip address>
will give you that.