hadoop windows下的编译与运行

# 0.参考
[1] [Build and Install Hadoop 2.x or newer on Windows](https://cwiki.apache.org/confluence/display/HADOOP2/Hadoop2OnWindows)
[2] [Build and Install Hadoop 2.7.2 on Windows](http://oldoldb.com/2016/02/17/Build-and-Install-Hadoop-2-7-2-on-Windows/) 
[3] [cannot run rc.exe x64 Visual Studio](https://stackoverflow.com/questions/35215971/lnk1158-cannot-run-rc-exe-x64-visual-studio)

# 1.环境准备

**系统:** win10
**vs:** visual studio 2015

## 1.1 JAVA安装
下载:[JDK 8.0](https://www.oracle.com/java/technologies/jdk8-downloads.html)
环境变量设置:JAVA_HOME;path;CLASSPATH(.;%JAVA_HOME%\lib\dt.jar;%JAVA_HOME%\lib\tools.jar;)

## 1.2 MAVEN
下载:[MAVEN 3.6.2](https://maven.apache.org/)
环境变量:添加PATH环境变量

## 1.3 ProtocolBuffer 2.5.0
下载:[protoc-2.5.0](https://github.com/google/protobuf/releases/download/v2.5.0/protoc-2.5.0-win32.zip)
环境变量:添加PATH中(D:\Program Files\protoc-2.5.0-win32)

## 1.4 CMake
下载:[3.15.0](https://cmake.org/)
默认添加path中

## 1.5 zlib headers
下载:[zlib128-dll](https://zlib.net/zlib128-dll.zip)
环境变量:ZLIB_HOME(D:\Program Files\zlib128-dll\include)

## 1.6 git
下载:[git-2.7.1](https://git-scm.com/download/win)
安装时:“Use Git and optional Unix tools from the Windows Command Prompt”

## 1.7 Visual studio 2015
下载:[VS2015-community](https://visualstudio.microsoft.com/zh-hans/vs/older-downloads/)

## 1.8 MSBuild 
下载:[Windows 10 SDK](https://developer.microsoft.com/en-US/windows/downloads/windows-10-sdk)
环境变量:将C:\Windows\Microsoft.NET\Framework64\v4.0.30319添加到PATH环境变量中

## 1.9 VS Project文件

- E:\code\qz\hadoop-release-2.7.2\hadoop-common-project\hadoop-common\src\main\winutils\winutils.sln
 - E:\code\qz\hadoop-release-2.7.2\hadoop-common-project\hadoop-common\src\main\native\native.sln

<center> 
![设置](https://leanote.com/api/file/getImage?fileId=5d722e35ab64412b860068fd)
</center>

## 1.10 更改编译选项
>E:\code\qz\hadoop-release-2.7.2\hadoop-hdfs-project\pom.xml

修改前:
<center>
![before](https://leanote.com/api/file/getImage?fileId=5d72399aab64412b86006b02)
</center>
修改后:
<center>
![after](https://leanote.com/api/file/getImage?fileId=5d72399aab64412b86006b03)
</center>

# 2. 编译

> 启动Developer Command Prompt for VS2015(admin)

<center>
![command promt](https://leanote.com/api/file/getImage?fileId=5d723a86ab64412b86006b33)
</center>

> 到存放代码路径下

<center>
![set](https://leanote.com/api/file/getImage?fileId=5d723b3fab64412b86006b56)
</center>

> build

---
tips:翻墙比较快

---
<center>
![mvn](https://leanote.com/api/file/getImage?fileId=5d723b40ab64412b86006b57)
</center>

<center>
![result](https://leanote.com/api/file/getImage?fileId=5d7599e8ab64411a56000ef9)
</center>

> release
编译成功后，文件存放于hadoop-dist\target目录下

# 3.单机部署模式(starting a Single Node(pseudo-distributed伪分布式) Cluster)
文件目录:
```

C:\deploy
├───bin
├───etc
│   └───hadoop
├───include
├───libexec
├───sbin
└───share

```
## 3.1 Example HDFS Configuration
> hadoop-env.cmd(add the following lines near the end of the file)

```
set HADOOP_PREFIX=c:\deploy
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR=%HADOOP_CONF_DIR%
set PATH=%PATH%;%HADOOP_PREFIX%\bin
```

>core-site.xml

```
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://0.0.0.0:19000</value>
  </property>
</configuration>
```

> hdfs-site.xml

```
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>
```

> edit or create the file `slaves`

```
localhost
```

## 3.2 Example YARN Configuration

> edit or create `mapred-site.xml` under `%HADOOP_PREFIX%\etc\hadoop`(replacing %USERNAME% with your Windows user name)

```
<configuration>

<property>
     <name>mapreduce.job.user.name</name>
     <value>%USERNAME%</value>
   </property>

<property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>
   </property>

<property>
    <name>yarn.apps.stagingDir</name>
    <value>/user/%USERNAME%/staging</value>
  </property>

<property>
    <name>mapreduce.jobtracker.address</name>
    <value>local</value>
  </property>

</configuration>
```

> edit or create `yarn-site.xml`

```
<configuration>
  <property>
    <name>yarn.server.resourcemanager.address</name>
    <value>0.0.0.0:8020</value>
  </property>

<property>
    <name>yarn.server.resourcemanager.application.expiry.interval</name>
    <value>60000</value>
  </property>

<property>
    <name>yarn.server.nodemanager.address</name>
    <value>0.0.0.0:45454</value>
  </property>

<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

<property>
    <name>yarn.server.nodemanager.remote-app-log-dir</name>
    <value>/app-logs</value>
  </property>

<property>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/dep/logs/userlogs</value>
  </property>

<property>
    <name>yarn.server.mapreduce-appmanager.attempt-listener.bindAddress</name>
    <value>0.0.0.0</value>
  </property>

<property>
    <name>yarn.server.mapreduce-appmanager.client-service.bindAddress</name>
    <value>0.0.0.0</value>
  </property>

<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>-1</value>
  </property>

<property>
    <name>yarn.application.classpath</name>
    <value>%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*</value>
  </property>
</configuration>

```

## 3.3 Initialize Environment Variables
Run ` c:\deploy\etc\hadoop\hadoop-env.cmd` to setup environment variables.

## 3.4 format the filesystem
format the filesystem withe the following command

```
%HADOOP_PREFIX%\bin\hdfs namenode -format
```

## 3.5 start HDFS Daemons

```
%HADOOP_PREFIX%\sbin\start-dfs.cmd
```

To verify that HDFS daemons are running,try copying a file to hdfs
```

C:\deploy>%HADOOP_PREFIX%\bin\hdfs dfs -put myfile.txt /

C:\deploy>%HADOOP_PREFIX%\bin\hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - username supergroup          4640 2014-01-18 08:40 /myfile.txt

```

## 3.6 start yarn daemons and run a yarn job

start the yarn daemons

```
%HADOOP_PREFIX%\sbin\start-yarn.cmd
```

To verify,run a simple wordcount job on the text file

```
%HADOOP_PREFIX%\bin\yarn jar %HADOOP_PREFIX%\share\hadoop\mapreduce\hadoop-mapreduce-examples-2.7.2.jar wordcount /myfile.txt /out
```
# 4 issues

## 4.1 cannot run rc.exe x64 Visual Studio
拷贝`C:\Program Files (x86)\Windows Kits\10\bin\10.0.18362.0\x64`下
```
rc.exe
rcdll.dll
```
到`C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin`下

信息安全从业人员^_^

导航

最近发表

友情链接