hadoop-intellij本地(windows)调试

# 0.参考
[0] [官方wiki-Developing hadoop](https://cwiki.apache.org/confluence/display/HADOOP2/DevelopingHadoop?highlight=%28%28HowToContribute%29%29)
[1] [Debug Hadoop source code using an IDE](http://www.techbite.in/2013/05/debug-hadoop-source-code-using.html)
[2] [Debugging Apache Hadoop using Intellij](https://leveragebigdata.blogspot.com/2017/01/debugging-apache-hadoop.html)
[3] [Debugging Hadoop HDFS using Intellij IDEA on Linux](https://www.codeproject.com/Articles/1067129/Debugging-Hadoop-HDFS-using-IntelliJ-IDEA-on-Linux)
[4] [hadoop windows下的编译与运行](http://blog.leanote.com/post/heming/hadoop-windows%E4%B8%8B%E7%9A%84%E7%BC%96%E8%AF%91)
[5] [Setting hadoop.tmp.dir on Windows gives error: URI has an authority component](https://stackoverflow.com/questions/31353226/setting-hadoop-tmp-dir-on-windows-gives-error-uri-has-an-authority-component)

# 1. build and debug

- 0.编译参考[hadoop windows下的编译与运行](http://blog.leanote.com/post/heming/hadoop-windows%E4%B8%8B%E7%9A%84%E7%BC%96%E8%AF%91),然后Intellij导入工程
 - 1.将hadoop-hdfs-project/hadoop-hdfs/src/main/webapps 文件目录拷贝到hadoop-hdfs-project/hadoop-hdfs/src/test/resources/目录下
 - 2.在hadoop-hdfs-project/hadoop-hdfs/src/test/resources/目录下创建文件core-site.xml
```
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>
 <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    <description>HDFS的URI，文件系统://namenode标识:端口号</description>
</property>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/E:/code/qz/hdfs</value>
    <description>namenode上本地的hadoop临时文件夹</description>
</property>
</configuration>
```

- 3.hadoop-hdfs-project/hadoop-hdfs/src/test/java/org.apache.hadoop目录下创建测试类NameNodeTest

```
package org.apache.hadoop;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hdfs.server.namenode.NameNode;

import java.io.IOException;

public class NameNodeTest {
    public static void main(String[] args) throws IOException {

System.setProperty("hadoop.home.dir", "E:\\code\\qz\\hadoop-release-2.7.2\\hadoop-dist\\target\\hadoop-2.7.2");
        Configuration conf = new Configuration();
        NameNode.format(conf);
        NameNode node = new NameNode(conf);
        if(node != null){
            node.join();
        }
    }
}
```

- 4.debug on NameNodeTest
 
<center>
 ![debug on nnTest](https://leanote.com/api/file/getImage?fileId=5db7a06eab6441133600019b)
</center>

---
tips for hadoop:
```
 mvn clean install -Pdist -Dtar -Ptest-patch -DskipTests -Denforcer.skip=true
```
---

# 2. issues
## 2.1 Package com.sun.javadoc does not exist
>解决方法:add Tools.jar to the project path
<center>
![添加Tools.jar-1](https://leanote.com/api/file/getImage?fileId=5db658a8ab6441484700053a)
</center>
<center>
![添加Tools.jar-2](https://leanote.com/api/file/getImage?fileId=5db65907ab64414a420004e7)
</center>

## 2.2 Setting hadoop.tmp.dir on Windows gives error: URI has an authority component
>解决方法(修改core-site.xml):
```
<property>
    <name>hadoop.tmp.dir</name>
    <value>/D:/tmp/hdp</value> //not D:/tmp/hdp
</property>
```

信息安全从业人员^_^

导航

最近发表

友情链接