Menu
- Hadoop Install On Windows 10
- Running Spark On Windows Without Hadoop
- Running Hadoop On Windows 7
- Running Cloudera Hadoop On Windows
- Starting Hadoop On Windows
Document your code
Introduction This post is to help people to install and run Apache Spark in a computer with window 10 (it may also help for prior versions of Windows or even Linux and Mac OS systems), and want to try out and learn how to interact with the engine without spend too many resources. Particularly I am interested in running Hadoop 2.1.0 as its release notes mention that running on Windows is supported. I know that I can try to run 1.x versions on Windows with Cygwin or even use prepared VM by for example Cloudera, but these options are in some reasons less convenient for me. I use Hadoop natively on Windows as a virtual 2-node cluster running on one machine. It runs inside Cygwin (so no VM). Works well to try Hadoop out and I still use it to test new code in small before putting it on the cluster. You basically get every bit of functionality as with a full cluster. Installing Hadoop-2.6.x on Windows 10. Shantanu Sharma Department of Computer Science, Ben-Gurion University, Israel. [email protected] 1. Install Java 8: Download Java 8 from the link. Go to the location: “D:hadoop-2.6.0sbin.” Run the following files as. Set the path HADOOPHOME Environment variable on windows 10(see Step 1,2,3 and 4 below). Set the path JAVAHOME Environment variable on windows 10(see Step 1,2,3 and 4 below). Next we set the Hadoop bin directory path and JAVA bin directory path.
Every project on GitHub comes with a version-controlled wiki to give your documentation the high level of care it deserves. It’s easy to create well-maintained, Markdown or rich text documentation alongside your code.
Sign up for free See pricing for teams and enterprises Prepare:
These softwares should be prepared to install Hadoop 2.8.0 on window 10 64bit
- Download Hadoop 2.8.0 (Link: http://www-eu.apache.org/dist/hadoop/common/hadoop-2.8.0/hadoop-2.8.0.tar.gz OR http://archive.apache.org/dist/hadoop/core//hadoop-2.8.0/hadoop-2.8.0.tar.gz)
- Java JDK 1.8.0.zip (Link: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
Set up
- Check either Java 1.8.0 is already installed on your system or not, use 'Javac -version' to check.
- If Java is not installed on your system then first install java under 'C:JAVA'
- Extract file Hadoop 2.8.0.tar.gz or Hadoop-2.8.0.zip and place under 'C:Hadoop-2.8.0'.
- Set the path HADOOP_HOME Environment variable on windows 10(see Step 1,2,3 and 4 below).
- Set the path JAVA_HOME Environment variable on windows 10(see Step 1,2,3 and 4 below).
- Next we set the Hadoop bin directory path and JAVA bin directory path.
Configuration
- Edit file C:/Hadoop-2.8.0/etc/hadoop/core-site.xml, paste below xml paragraph and save this file.
- Rename 'mapred-site.xml.template' to 'mapred-site.xml' and edit this file C:/Hadoop-2.8.0/etc/hadoop/mapred-site.xml, paste below xml paragraph and save this file.
- Create folder 'data' under 'C:Hadoop-2.8.0'
- Create folder 'datanode' under 'C:Hadoop-2.8.0data'
- Create folder 'namenode' under 'C:Hadoop-2.8.0data'
- Edit file C:Hadoop-2.8.0/etc/hadoop/hdfs-site.xml, paste below xml paragraph and save this file.
- Edit file C:/Hadoop-2.8.0/etc/hadoop/yarn-site.xml, paste below xml paragraph and save this file.
- Edit file C:/Hadoop-2.8.0/etc/hadoop/hadoop-env.cmd by closing the command line 'JAVA_HOME=%JAVA_HOME%' instead of set 'JAVA_HOME=C:Java' (On C:java this is path to file jdk.18.0)
Hadoop Configuration
- Dowload file Hadoop Configuration.zip (Link: https://github.com/MuhammadBilalYar/HADOOP-INSTALLATION-ON-WINDOW-10/blob/master/Hadoop%20Configuration.zip)
- Delete file bin on C:Hadoop-2.8.0bin, replaced by file bin on file just download (from Hadoop Configuration.zip).
- Open cmd and typing command 'hdfs namenode –format' . You will see
Testing
- Open cmd and change directory to 'C:Hadoop-2.8.0sbin' and type 'start-all.cmd' to start apache.
- Make sure these apps are running
- Hadoop Namenode
- Hadoop datanode
- YARN Resourc Manager
- YARN Node Manager
- Open: http://localhost:8088
- Open: http://localhost:50070
Clone this wiki locally
Active1 year, 1 month ago
I am new to Hadoop and have run into problems trying to run it on my Windows 7 machine. Particularly I am interested in running Hadoop 2.1.0 as its release notes mention that running on Windows is supported. I know that I can try to run 1.x versions on Windows with Cygwin or even use prepared VM by for example Cloudera, but these options are in some reasons less convenient for me.
Having examined a tarball from http://apache-mirror.rbc.ru/pub/apache/hadoop/common/hadoop-2.1.0-beta/ I found that there really are some *.cmd scripts that can be run without Cygwin. Everything worked fine when I formated HDFS partition but when I tried to run hdfs namenode daemon I faced two errors: first, non fatal, was that winutils.exe could not be found (it really wasn't present in the tarball downloaded). I found the sources of this component in the Apache Hadoop sources tree and compiled it with Microsoft SDK and MSbuild. Thanks to detailed error message it was clear where to put the executable to satisfy Hadoop. But the second error which is fatal doesn't contain enough information for me to solve:
Looks like something else should be compiled. I'm going to try to build Hadoop from the source with Maven but isn't there a simpler way? Isn't there some option-I-know-not-of that can disable native code and make that tarball usable on Windows?
Thank you.
UPDATED. Yes, indeed. 'Homebrew' package contained some extra files, most importantly winutils.exe and hadoop.dll. With this files namenode and datanode started successfully. I think the question can be closed. I didn't delete it in case someone face the same difficulty.
UPDATED 2. To build the 'homebrew' package I did the following:
- Got sources, and unpacked them.
- Read carefully BUILDING.txt.
- Installed dependencies:
3a) Windows SDK 7.1
3b) Maven (I used 3.0.5)3c) JDK (I used 1.7.25)
3d) ProtocolBuffer (I used 2.5.0 - http://protobuf.googlecode.com/files/protoc-2.5.0-win32.zip). It is enough just to put compiler (protoc.exe) into some of the PATH folders.
3e) A set of UNIX command line tools (I installed Cygwin) - Started command line of Windows SDK. Start | All programs | Microsoft Windows SDK v7.1 | ... Command Prompt (I modified this shortcut, adding option /release in the command line to build release versions of native code). All the next steps are made from inside SDK command line window)
- Set up the environment:set JAVA_HOME={path_to_JDK_root}
It seems that JAVA_HOME MUST NOT contain space!
- Changed dir to sources root folder (BUILDING.txt warns that there are some limitations on the path length so sources root should have short name - I used D:hds)
- Ran building process:mvn package -Pdist -DskipTests
You can try without 'skipTests' but on my machine some tests failed and building was terminated. It may be connected to sybolic link issues mentioned in BUILDING .txt.8. Picked the result in hadoop-disttargethadoop-2.1.0-beta (windows executables and dlls are in 'bin' folder)
Hatter
HatterHatter51311 gold badge55 silver badges1111 bronze badges
Hadoop Install On Windows 10
12 Answers
I have followed following steps to install Hadoop 2.2.0
Steps to build Hadoop bin distribution for Windows
- Download and install Microsoft Windows SDK v7.1.
- Download and install Unix command-line tool Cygwin.
- Download and install Maven 3.1.1.
- Download Protocol Buffers 2.5.0 and extract to a folder (say c:protobuf).
- Add Environment Variables JAVA_HOME, M2_HOME and Platform if not added already. Note : Variable name Platform is case sensitive. And value will be either x64 or Win32 for building on a 64-bit or 32-bit system. Edit Path Variable to add bin directory of Cygwin (say C:cygwin64bin), bin directory of Maven (say C:mavenbin) and installation path of Protocol Buffers (say c:protobuf).
- Download hadoop-2.2.0-src.tar.gz and extract to a folder having short path (say c:hdfs) to avoid runtime problem due to maximum path length limitation in Windows.
- Select Start --> All Programs --> Microsoft Windows SDK v7.1 and open Windows SDK 7.1 Command Prompt. Change directory to Hadoop source code folder (c:hdfs). Execute mvn package with options -Pdist,native-win -DskipTests -Dtar to create Windows binary tar distribution.
- If everything goes well in the previous step, then native distribution hadoop-2.2.0.tar.gz will be created inside C:hdfshadoop-disttargethadoop-2.2.0 directory.
Install Hadoop
- Extract hadoop-2.2.0.tar.gz to a folder (say c:hadoop).
- Add Environment Variable HADOOP_HOME and edit Path Variable to add bin directory of HADOOP_HOME (say C:hadoopbin).
Configure Hadoop
Running Spark On Windows Without Hadoop
C:hadoopetchadoopcore-site.xml
C:hadoopetchadoophdfs-site.xml
C:hadoopetchadoopmapred-site.xml
C:hadoopetchadoop yarn-site.xml
Format namenode
For the first time only, namenode needs to be formatted.
Start HDFS (Namenode and Datanode)
Start MapReduce aka YARN (Resource Manager and Node Manager)
Total four separate Command Prompt windows will be opened automatically to run Namenode, Datanode, Resource Manager, Node Manager
Reference : Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS
AbhijitAbhijit
I had the same problem but with recent hadoop v. 2.2.0. Here are my steps for solving that problem:
- I've built
winutils.exe
from sources. Project directory:hadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutils
My OS: Windows 7. Tool for building: MS Visual Studio Express 2013 for Windows Desktop (it's free and can be loaded from http://www.microsoft.com/visualstudio/).Open Studio,File -> Open -> winutils.sln
. Right click on solution on the right side ->Build
.There were a couple errors in my case (you might need to fix project properties, specify output folder).Viola! You getwinutils.exe
- put it into hadoop's bin. - Next we need to build
hadoop.dll
.Some woodoo magic here goes: openhadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.sln
in MS VS; right click on solution -> build.I got a bunch of errors. I created manually several missed header files (don't ask me why they are missed in source tarball!):(and don't ask me what this project on git is for! I don't know - google pointed it out by searching header file names)I've copiedhadoop-2.2.0-srchadoop-common-projecthadoop-commontargetwinutilsDebuglibwinutils.lib
(result of step # 1) intohadoop-2.2.0-srchadoop-common-projecthadoop-commontargetbin
And finally build operation produces hadoop.dll!Put it again into hadoop's bin and happily run namenode!
Hope my steps will help somebody.
Aleksei EgorovAleksei Egorov
Han has prepared the Hadoop 2.2 Windows x64 binaries (see his blog) and uploaded them to Github.
After putting the two binaries
winutils.exe
and hadoop.dll
into the %hadoop_prefix%bin
folder, I got the same UnsatisfiedLinkError
.The problem was that some dependency of
hadoop.dll
was missing. I used Dependency Walker to check the dependencies of the binaries and the Microsoft Visual C++ 2010 Redistributables were missing.So besides building all the components yourself, the answer to the problem is
- make sure to use the same architecture for Java and the native code.
java -version
tells you if you use 32 or x64. - then use Dependency Walker to make sure all native binaries are pure and of the same architecture. Sometimes a x64 dependency is missing and Windows falls back to x86, which does not work. See answer of another question.
- also check if all dependencies of the native binaries are satisfied.
Community♦
Running Hadoop On Windows 7
Peter KoflerPeter Kofler6,83666 gold badges4242 silver badges7474 bronze badges
In addition to other solutions, here is a pre-built copy of winutil.exe. Donload it and add to $HADOOP_HOME/bin. It works for me.
(Source :Click here)
Prasad DPrasad D
Please add hadoop.dll (version sensitive) to the system32 directory under Windows Directory.
You can get the hadoop.dll at winutils
futuredaemonfuturedaemon
Instead of using the official branch I would suggest the windows optimized
You need to compile it, build winutils.exe under windows and place it in the hadoop/bin directory
Marco SeravalliMarco Seravalli
You might need to copy hadoop.dll and winutils.exe files from hadoop-common-bin to %HADOOP_HOME%binAdd %HADOOP_HOME%/bin to your %PATH% variable.
You can download hadoop-common from https://github.com/amihalik/hadoop-common-2.6.0-bin
Vikash PareekVikash Pareek
I ran into same problem with Hadoop 2.4.1 on Windows 8.1; there were a few differences with the resulting solution caused mostly by the newer OS.
I first installed Hadoop 2.4.1 binary, unpacking it into %HADOOP_HOME%.
The previous answers describe how to set up Java, protobuf, cygwin, and maven, and the needed environment variables. I had to change my Platform environment variable from HP's odd 'BCD' value.
I downloaded the source from an Apache mirror, and unpacked it in a short directory (HADOOP_SRC = C:hsrc). Maven ran fine from a standard Windows command prompt in that directory: mvn package -DskipTests.
Instead of using the Windows 7 SDK (which I could not get to load) or the Windows 8.1 SDK (which doesn't have the command line build tools), I used the free Microsoft Visual Studio Express 2013 for Windows Desktop. Hadoop's build needed the MSBuild location (C:Program Files (x86)MSBuild12.0) in the PATH, and required that the various Hadoop native source projects be upgraded to the newer (MS VS 2013) format. The maven build failures were nice enough to point out the absolute path of each project as it failed, making it easy to load the project into Visual Studio (which automatically converts, after asking).
Once built, I copied the native executables and libraries into the Hadoop bin directory. They were built in %HADOOP_SRC%hadoop-common-projecthadoop-commontargetbin, and needed to be copied into %HADOOP_HOME%bin.
leifbennettleifbennett
Adding hadoop.dll and hdfs.dll to the %HADOOP_HOME%bin folder did the trick for me.
Kunal KanojiaKunal Kanojia
Just installed Hadoop 2.2.0 in my environment win7 X64.
following BUILD.txt makes me did that.Note that:the dir in the hdfs-site.xml and mapred-site.xml is starts with / like below
E.G
May help u!
DerryDerry
Download & Install Java in
c:/java/
make sure the path is this way, if java is installed in 'program files', then hadoop-env.cmd will not recognize java path
Download Hadoop binary distribution.
I am using binary distribution Hadoop-2.8.1. Also I would recommend to keep extraction path as short as possible
Set Environment Variables:
Hadoop will work on windows if Hadoop-src is built using maven in your windows machine. Building the Hadoop-src(distribution) will create a Hadoop binary distribution, which will work as windows native version.
But if you don't want to do that, then download pre-builted
winutils of Hadoop distribution.
Here is a GitHub link, which has winutils of some versions of Hadoop. if the version you are using is not in the list, the follow the conventional method for setting up Hadoop on windows - link
If you found your version, then copy paste all content of folder into path: /bin/
Set all the .xml configuration files - Link & set JAVA_HOME path in hadoop-env.cmd file
From cmd go to:
Hope this helps.
Raxit SolankiRaxit Solanki
- Get Hadoop binaries (which include
winutils.exe
andhadoop.dll
) - Make sure hadoopbin is available via PATH (System PATH if you run it as a Service)Note that setting
java.library.path
overridesPATH
. If you setjava.library.path
, make sure it is correct and points to the hadoop library.
38.5k1212 gold badges113113 silver badges157157 bronze badges
Running Cloudera Hadoop On Windows
protected by Community♦Nov 4 '13 at 0:59
Starting Hadoop On Windows
Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?
Would you like to answer one of these unanswered questions instead?