DataStax Help Center

Hadoop map reduce job fails with: No such file or directory

Summary

On Hive, customer is able to execute:

hive> select * from <table>;

However, the following will fail

hive> select count(*) from <table>;

With the following exception on /tmp/mapreducelocal/<job id>/syslog :

2014-06-02 19:09:49,832 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.Error: Failed to create temporary file for jnidispatch library: java.io.IOException: No such file or directory
    at com.sun.jna.Native.loadNativeLibraryFromJar(Native.java:765)
    at com.sun.jna.Native.loadNativeLibrary(Native.java:699)
    at com.sun.jna.Native.<clinit>(Native.java:114)
    at org.apache.hadoop.io.filestreaming.FileStreaming.<clinit>(FileStreaming.java:19)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189)
    at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:184)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:256)
    at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:237)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
    at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:435)
    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:315)
    at org.apache.hadoop.mapred.JobLocalizer.writeLocalJobFile(JobLocalizer.java:558)
    at org.apache.hadoop.mapred.Child.localizeTask(Child.java:354)
    at org.apache.hadoop.mapred.Child.main(Child.java:231)

 And Cassandra system log showing:

FATAL [IPC Server handler 8 on 58471] 2014-06-02 19:09:49,832 TaskTracker.java (line 3557) Task: attempt_201406021902_0001_m_000004_3 - Killed : Failed to create temporary file for jnidispatch library: java.io.IOException: No such file or directory

Cause

There are 3 possible causes for this issue:

- Insufficient permissions on /tmp/mapreducelocal directory for the user running the job

- Duplicated jna.jar on the Cassandra CLASSPATH

- Old version of JNA library

Solution

- Grant permissions to the user running hive to /tmp/mapreducelocal

- Verify from the Cassandra system log @ /var/log/cassandra/system.log that jna.jar is not duplicated with different paths to the same file, for example

/usr/lib/jna.jar
/usr/share/java/jna.jar

   In this case, move /usr/lib/jna.jar out of the way

- Make sure JNA package is installed as per our guidelines, we require at least JNA 3.2.7

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

  • Avatar
    José Martínez Poblete

    Created from T10551

  • Avatar
    Dreddy

    was running into this exact issue, thanks for the pointer

Powered by Zendesk