33.0k views
1 vote
How to resolve Hadoop warning ""WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable""?

User Vader B
by
3.2k points

1 Answer

3 votes

Answer:

This warning message indicates that the Hadoop native libraries, which provide better performance for some operations, are not available on your platform. This is typically not a problem, as Hadoop will automatically fall back to using pure Java implementations for these operations. However, if you would like to remove the warning message, you can try one of the following solutions:

1. Install the Hadoop native libraries for your platform. These are typically included in the Hadoop distribution, but may need to be installed separately. Consult the Hadoop documentation for more information.

2. Add the following line to the top of your Hadoop configuration file (usually named "hadoop-env.sh"):

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=<path to native libraries>"

This will tell Hadoop to use the specified path to search for the native libraries, and should prevent the warning message from appearing.

3. If you don't need the performance benefits provided by the native libraries, you can disable them by setting the following property in your Hadoop configuration file:

hadoop.native.lib = false

This will prevent Hadoop from trying to load the native libraries, and will suppress the warning message. However, you may see a performance degradation for certain operations.

User Pindare
by
3.4k points