Attempts to connect to a Kerberos-enabled cluster running on CentOS 7 fails with "LoginException: Message stream modified (41)"

Summary

This article discusses an authentication issue on Kerberos-enabled clusters running on CentOS 7 (and related Linux distributions).

Applies to

• CentOS 7.x
• RedHat Enterprise Linux (RHEL) 7.x
• Oracle Enterprise Linux (OEL) 7.x

Symptom

Attempts to connect to a cluster with Kerberos authentication enabled running on CentOS 7.x fail with an authentication exception. For example, a failed cqlsh connection generates the following error in the system.log:

ERROR [IOThread-0] 2020-03-04 05:42:32,069  DseAuthenticator.java:778 - Error obtaining subject for server identity
javax.security.auth.login.LoginException: Message stream modified (41)
    at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
    at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at com.datastax.bdp.transport.server.KerberosServerUtils.loginServer(KerberosServerUtils.java:36)
    at com.datastax.bdp.cassandra.auth.DseAuthenticator$GSSAPISaslNegotiator.(DseAuthenticator.java:774)
    at com.datastax.bdp.cassandra.auth.DseAuthenticator.getSaslNegotiatorForScheme(DseAuthenticator.java:682)
    at com.datastax.bdp.cassandra.auth.DseAuthenticator.access$400(DseAuthenticator.java:100)
    at com.datastax.bdp.cassandra.auth.DseAuthenticator$UnifiedSaslNegotiator.evaluateResponse(DseAuthenticator.java:513)
    ...
Caused by: sun.security.krb5.internal.KrbApErrException: Message stream modified (41)
    at sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:101)
    at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:159)
    at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
    at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308)
    at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447)
    at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780)
    ... 31 common frames omitted

Cause

The issue occurs due to an unknown problem with Kerberos on CentOS 7.x and related Linux distributions including RHEL 7.x and OEL 7.x. The problem was reported by other users and is being tracked as a CentOS bug #17000. Neither the root cause nor a fix in CentOS or RHEL was available at the time of writing.

Workaround

Users have reported that removing the renew_lifetime parameter from the Kerberos configuration file /etc/krb5.conf allows applications to authenticate successfully. For example, remove the line:

 renew_lifetime = 7d

from the [libdefaults] section:

[libdefaults]
 default_realm = TEST.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

Solution

This is not a DataStax Enterprise issue. Please contact your Linux operating system vendor to obtain a fixed version of your Linux distribution.

See also

Dev Community - https://community.datastax.com/questions/2846/