How To Install Hadoop On Mac

Building Hadoop native libraries on Mac in 2022

TL;DR to exist found at the end

Recently I came into a situation that I "needed" Hadoop native libraries. Well, when I say "needed", I mean I was but getting fed up past the constant warnings like this one:

            WARN util.NativeCodeLoader: Unable to load native-hadoop library              for              your platform... using builtin-java classes where applicable                      

So I thought I would build my own Hadoop native libraries. How difficult tin can it be, correct? Honest answer? Less than an hour if you don't have a tutorial. Fifteen minutes if y'all do and most of that is compilation time. In my search, I establish out a lot of tutorials and guides were either outdated or didn't offer everything needed for a full compilation and installation and that is why I wrote my own which I tested on two independent Macs, thus it should be "tested enough".

Why do it

At that place was no real world issue I was hoping to solve. I just had a few minutes on my hands and I used them to learn something new. But I did read that there are cases of speed improvements which is skilful if yous are developing or testing something locally because local machines tend to be irksome and whatsoever improvement is more welcome. Another thing is I did see two random articles a while back saying they did have some issues with the Java libraries, but chances of some of you having the aforementioned issues are really modest.

Dependencies

First of all, we demand to install the dependencies for the build and I am including links so yous can check what you are going to install exactly:

• gcc
• autoconf
• automake
• libtool
• cmake
• snappy
• gzip
• bzip2
• zlib
• wget
• openssl i.0
  • 1.1 on Brew has an issue. More in the comments section. Thanks to @imasli
• protobuf 2.five.0

(Delight note I am skipping maven, java and others that I think you would already have. If I am wrong, tell me and let's update the article. Likewise equally Hadoop installation. There is a beautiful article about Hadoop installation on Mac by Zhang Hao here.)

For the installation of most of these, I will be using Homebrew. It's a good tool, has a one-liner installation and a very brusk boilerplate time to exist productive with information technology. Equally the link provides everything you need I am skipping the installation here.

If you are not using Homebrew for the first time, update and upgrade your tools. If you are using it for some time already and would like to keep some things with the current version, utilize brew pivot like this.

                          # Update              brew update brew upgrade              # Then the installation              brew              install              wget gcc autoconf automake libtool cmake snappy              gzip              bzip2 zlib openssl                      

As you could take noticed 1 of those dependencies listed is missing from the list above. Yep! It is a protobuf that has been deprecated and can't be hands installed from Homebrew. So let's build our own. Information technology's cleaner that way and much more fun and so it sounds. We volition get-go need to get it from GitHub and unarchive it somewhere. You can delete it right subsequently, and then you don't demand a special folder construction.

            wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-ii.five.0.tar.gz              tar              -xzf              protobuf-2.five.0.tar.gz              cd              protobuf-2.5.0                      

So comes the process of building and making certain everything went smoothly. It takes some time and I advise you lot to run it step by step to run across and know what is happening. Some warnings here and there are normal then yous tin skip those.

            ./configure make brand check brand              install              # And just to check if everything is ok.              # This should print libprotoc 2.5.0              protoc              --version                      

OpenSSL setup

Now, linking OpenSSL libraries by hand as Homebrew refuses to link OpenSSL and the compiler needs them. This is a known characteristic and needs to be done by running ln.

                          cd              /usr/local/include              ln              -due south              ../opt/openssl/include/openssl              .                      

This volition solve an mistake that looks something like the caption below.

                          [              exec              ]              --              Configuring incomplete, errors occurred!              [              exec              ]              Run into too /Users/user/github/hadoop/hadoop-tools/hadoop-pipes/target/native/CMakeCMake Error at /usr/local/Cellar/cmake/iii.14.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:137              (message):              [              exec              ]              Could NOT find OpenSSL, try to              gear up              the path to OpenSSL root folder              in              the              [              exec              ]              system variable OPENSSL_ROOT_DIR              (missing: OPENSSL_INCLUDE_DIR)              [              exec              ]              Call Stack              (nearly contempo call offset):              [              exec              ]              /usr/local/Cellar/cmake/3.14.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:378              (_FPHSA_FAILURE_MESSAGE)              [              exec              ]              /usr/local/Cellar/cmake/3.14.3/share/cmake/Modules/FindOpenSSL.cmake:413              (find_package_handle_stFiles/CMakeOutput.log.              [              exec              ]              andard_args)              [              exec              ]              CMakeLists.txt:20              (find_package)              [              exec              ]              [              exec              ]                      

Building native libraries

And finally! The building of the libraries. Again, this will create a folder that you can delete in the stop. Hither is probably the first identify you volition demand to modify something and that is the version of Hadoop you lot will be using.

            git clone https://github.com/apache/hadoop.git              cd              hadoop              # Change the version as needed              git checkout branch-<VERSION>              # And merely package.              mvn package              -Pdist,native              -DskipTests              -Dtar              # Later build, move your newly created libraries.              cp              -R              hadoop-dist/target/hadoop-<VERSION>/lib              $HADOOP_HOME                      

Setting up environment variables

Now the critical part, making your shell meet the libraries. I don't know what kind of shell you are using, nevertheless, put this into your beat out profile (.bashrc, .zshrc, etc.):

                          export                            HADOOP_OPTS              =              "-Djava.library.path=              ${              HADOOP_HOME              }              /lib/native"              export                            LD_LIBRARY_PATH              =              $LD_LIBRARY_PATH:${              HADOOP_HOME              }/lib/native              export                            JAVA_LIBRARY_PATH              =              $JAVA_LIBRARY_PATH:${              HADOOP_HOME              }/lib/native                      

This will signal all the libraries to the correct path and will make everything fall right into place. The final matter that we need is merely to check if everything is ok (and by everything I mean near everything, because bzip is acting upward and I still accept not establish a manner to solve, when I practice I will update this).

            hadoop checknative              -a              #The output should be something similar this.              19/05/17 19:00:14 WARN bzip2.Bzip2Factory: Failed to load/initialize native-bzip2 library organization-native, volition use pure-Java version 19/05/17 19:00:14 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop:              true              /usr/local/Cellar/hadoop/2.vii.v/lib/native/libhadoop.dylib zlib:              truthful              /usr/lib/libz.1.dylib snappy:              true              /usr/local/lib/libsnappy.i.dylib lz4:              true              revision:99 bzip2:              false              openssl:              true              /usr/lib/libcrypto.35.dylib 19/05/17 nineteen:00:14 INFO util.ExitUtil: Exiting with status 1                      

Afterword

Hopefully, everything is running smoothly and you no longer get those warnings and if I helped even one person with this I am glad. Considering if in that location is no added value for the reader, then information technology is just me talking to my wall. On the other mitt, if you lot did find some issues in the lawmaking or the article, please practice tell me and I will fix everything I am capable of.

TL;DR

This is just a step by step shell script extracted from the upper text.

Source: https://dev.to/zejnilovic/building-hadoop-native-libraries-on-mac-in-2019-1iee

Posted by: looneysamet1997.blogspot.com

Share this post