When authentication is enabled, completed jobs information cannot be viewed using Spark Worker UI

Summary

When authentication and authorization are turned on, stdout and stderr logs from Spark Worker UI (http://hostname:7081/) cannot be viewed for Finished Drivers/applications. This issue persists even with correct Spark authentication for the current user.

Applies to

• DataStax Enterprise 6.0.4, 6.0.5, 6.0.6
• DataStax Enterprise 6.7.1, 6.7.2

Symptoms

Using the Spark Web UI or the Spark History Server web UI to view executor logs (stdout and/or stderr) of completed applications and drivers, you are unable to view the log contents and this error occurs:

HTTP ERROR 403 
Problem accessing /logPage/. Reason: you are not allowed to access this resource

Cause

This is a known issue in DSE 6.0.4, DSE 6.0.5, and DSE 6.0.6 (Internal DSP-17854 - Unable to access stderr/stdout from Spark Worker UI for Finished Drivers jobs).

Even when a user has the required permissions, the reference to the driver or executor in the table is removed when the job is completed and causes all requests to view information on the finished jobs to fail. 

Workaround

View the logs on the file system of the Spark Worker nodes instead using the Spark Web UI.  The UI reads the logs from:

$SPARK_WORKER_DIR/<appID or driverID>/<executor id>/<stdout or stderr>

By default, the SPARK_WORKER_DIR is /var/lib/spark/worker.

Solution

Upgrade to DSE 6.0.7 or later or DSE 6.7.3 or later.   

And be sure the user has at least one of the following permissions:

GRANT DESCRIBE ON ANY SUBMISSION TO role_name;

Or

GRANT DESCRIBE ON ANY SUBMISSION IN WORKPOOL 'datacenter_name.*' TO role_name;

Or

GRANT DESCRIBE ON SUBMISSION id IN WORKPOOL 'datacenter_name.*' TO role_name;