Summary
This article discusses an issue with download links on the Spark History Server web user interface.
Applies to
- DataStax Enterprise 6.7.x
- DataStax Enterprise 6.0.x
Symptoms
The Spark History Server allows access to event logs via a web-based user interface when the functionality is enabled. When browsing the Spark history server UI (by default http://dse_node_IP:18080) then clicking on the download button of one of the completed applications (Spark job) returns a "no such app" error.
Here is an example screenshot of the web UI on a node running DSE 6.7.2:
In this example, clicking on the download button for app ID app-20190308134458-0000 returns the following error:
no such app: app-20190308134458-0000
Cause
There is an issue with open source Apache Spark 2.2 where the API for the History Server generates an invalid URL for the download link of event logs (SPARK-20517). The generated URL incorrectly includes the string /null in the path.
In the example above, the URL for the download link is:
http://10.1.2.3:18080/api/v1/applications/app-20190308134458-0000/null/logs
Notice that the URL contains /null.
SPARK-20517 has been fixed in open source Apache Spark but was not merged into any Spark 2.2.x release and is not available in any of the Spark versions included with DSE.
Workaround
Manually update the URL on the browser and remove the /null in the path.
For the example above, the following URL will allow users to download the event logs for app ID app-20190308134458-0000:
http://10.1.2.3:18080/api/v1/applications/app-20190308134458-0000/logs
Solution
The DSE fix for SPARK-20517 is tracked with DSE (internal ID DSP-17883).
Click the "Follow" button on this article to receive updates to track future DSE releases that might contain the fix.