Summary
This article discussed a situation where users who try to access the Spark Master web interface on an upgraded cluster are prompted to enter credentials to login.
Applies to
- DataStax Enterprise 6.7
- DataStax Enterprise 6.0
- DataStax Enterprise 5.1
Symptom
After upgrading an Analytics node, users who try to access the Spark Master web interface (for example http://10.101.32.168:7080/) are prompted with a login box. Versions earlier than DSE 5.1 did not require authentication. Below is an example of a sign in dialog box:
If users do not provide credentials, a HTTP 401 unauthorized error occurs with a message like:
Problem accessing /. Reason:
Authentication is required to access this resource
Providing incorrect credentials will return the following error:
Problem accessing /. Reason:
Authentication failed
Cause
A new security feature was added in DSE 5.1.0 which leverages DSE Advanced Security to secure cluster components and services. If DSE Unified Authentication is enabled with LDAP or internal authentication, users will be prompted for login credentials when accessing the Spark Master web interface. See DSP-11311 in the 5.1.0 DSE Analytics changes and enhancements section of the release notes.
Note that Kerberos authentication is not supported in the Spark web interface.
Workaround
Spark web interface basic HTTP authentication supports only LDAP and internal authentication. For clusters which use only Kerberos, disable the Spark web UI authentication by following the steps below.
STEP 1 - On the first DSE Analytics node, modify the Spark daemon configuration file. For DSE package installations:
$ vi /etc/dse/spark/spark-daemon-defaults.conf
For DSE tarball installations:
$ vi <path_to_installation>/resources/spark/conf/spark-daemon-defaults.conf
STEP 2 - Disable the SparkUIAuthFilter by commenting out (add a # at the start) or removing the following line:
spark.ui.filters com.datastax.bdp.auth.SparkUIAuthFilter
STEP 3 - Restart DSE on the node for the change to take effect.
If the authentication filter has been disabled correctly, there should not be an entry like this in the logs:
INFO [dispatcher-event-loop-1] 2019-07-28 16:20:18,438 Logging.scala:54 - \ Adding filter: com.datastax.bdp.auth.SparkUIAuthFilter
STEP 4 - Repeat the steps above on the next node until all DSE Analytics node have been done.