centos7下编译hadoop2.7.2

之前有编译过hadoop2.2版本，不过一直都是迷迷糊糊，不知道怎么回事。今天周末拿hadoop2.7.2版本重新编译一下。顺便做一下记录 相关软件和环境： 系统：centos7 1），jdk1.8.0_91，官网说是使用jdk1.7版本，但是我任性地想试一试1.8，所以就没用1.7版本。 2):maven-3.3.9 3):protobuf-2.5.0 4):findbugs-1.3.9 安装就不想啰嗦了，安装protobuf的时候可能会出现错误： configure: error: in `/opt/protobuf/protobuf-2.5.0’: configure: error: C++ preprocessor “/lib/cpp” fails sanity check 只需要 yum install glibc-headers yum install gcc-c++ 之外，没什么特别的，都是解压后配置一下环境变量即可。 以下是我的 /etc/profile的内容

export JAVA_HOME=/opt/jdk/jdk1.8.0_91
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin
export MAVEN_HOME=/opt/maven/apache-maven-3.3.9
export PATH=$PATH:$MAVEN_HOME/bin
export FINDBUGS_HOME=/opt/findbugs/findbugs-1.3.9
export PATH=$PATH:$FINDBUGS_HOME/bin

配置好后就可以编译了，编译过程中可能出现如下错误： main: [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/opt/jdk/jdk1.8.0_91/jre/lib/amd64/server/libjvm.so [exec] JAVA_INCLUDE_PATH=/opt/jdk/jdk1.8.0_91/include, JAVA_INCLUDE_PATH2=/opt/jdk/jdk1.8.0_91/include/linux [exec] Located all JNI components successfully. [exec] – Configuring incomplete, errors occurred! [exec] CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:108 (message): [exec] Could NOT find ZLIB (missing: ZLIB_INCLUDE_DIR) [exec] Call Stack (most recent call first): [exec] /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:315 (_FPHSA_FAILURE_MESSAGE) [exec] /usr/share/cmake/Modules/FindZLIB.cmake:85 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) [exec] CMakeLists.txt:107 (find_package) [exec] [exec] [INFO] ———————————————————————— [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main …………………………… SUCCESS [ 1.818 s] [INFO] Apache Hadoop Project POM …………………….. SUCCESS [ 1.859 s] [INFO] Apache Hadoop Annotations …………………….. SUCCESS [ 4.010 s] [INFO] Apache Hadoop Assemblies ……………………… SUCCESS [ 0.237 s] [INFO] Apache Hadoop Project Dist POM ………………… SUCCESS [ 2.187 s] [INFO] Apache Hadoop Maven Plugins …………………… SUCCESS [ 3.686 s] [INFO] Apache Hadoop MiniKDC ………………………… SUCCESS [ 3.249 s] [INFO] Apache Hadoop Auth …………………………… SUCCESS [ 4.148 s] [INFO] Apache Hadoop Auth Examples …………………… SUCCESS [ 3.824 s] [INFO] Apache Hadoop Common …………………………. FAILURE [ 23.049 s] [INFO] Apache Hadoop NFS ……………………………. SKIPPED [INFO] Apache Hadoop KMS ……………………………. SKIPPED [INFO] Apache Hadoop Common Project ………………….. SKIPPED [INFO] Apache Hadoop HDFS …………………………… SKIPPED [INFO] Apache Hadoop HttpFS …………………………. SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal ………….. SKIPPED [INFO] Apache Hadoop HDFS-NFS ……………………….. SKIPPED [INFO] Apache Hadoop HDFS Project ……………………. SKIPPED [INFO] hadoop-yarn …………………………………. SKIPPED [INFO] hadoop-yarn-api ……………………………… SKIPPED [INFO] hadoop-yarn-common …………………………… SKIPPED [INFO] hadoop-yarn-server …………………………… SKIPPED [INFO] hadoop-yarn-server-common …………………….. SKIPPED [INFO] hadoop-yarn-server-nodemanager ………………… SKIPPED [INFO] hadoop-yarn-server-web-proxy ………………….. SKIPPED [INFO] hadoop-yarn-server-applicationhistoryservice ……. SKIPPED [INFO] hadoop-yarn-server-resourcemanager …………….. SKIPPED [INFO] hadoop-yarn-server-tests ……………………… SKIPPED [INFO] hadoop-yarn-client …………………………… SKIPPED [INFO] hadoop-yarn-server-sharedcachemanager ………….. SKIPPED [INFO] hadoop-yarn-applications ……………………… SKIPPED [INFO] hadoop-yarn-applications-distributedshell ………. SKIPPED [INFO] hadoop-yarn-applications-unmanaged-am-launcher ….. SKIPPED [INFO] hadoop-yarn-site …………………………….. SKIPPED [INFO] hadoop-yarn-registry …………………………. SKIPPED [INFO] hadoop-yarn-project ………………………….. SKIPPED [INFO] hadoop-mapreduce-client ………………………. SKIPPED [INFO] hadoop-mapreduce-client-core ………………….. SKIPPED [INFO] hadoop-mapreduce-client-common ………………… SKIPPED [INFO] hadoop-mapreduce-client-shuffle ……………….. SKIPPED [INFO] hadoop-mapreduce-client-app …………………… SKIPPED [INFO] hadoop-mapreduce-client-hs ……………………. SKIPPED [INFO] hadoop-mapreduce-client-jobclient ……………… SKIPPED [INFO] hadoop-mapreduce-client-hs-plugins …………….. SKIPPED [INFO] Apache Hadoop MapReduce Examples ………………. SKIPPED [INFO] hadoop-mapreduce …………………………….. SKIPPED [INFO] Apache Hadoop MapReduce Streaming ……………… SKIPPED [INFO] Apache Hadoop Distributed Copy ………………… SKIPPED [INFO] Apache Hadoop Archives ……………………….. SKIPPED [INFO] Apache Hadoop Rumen ………………………….. SKIPPED [INFO] Apache Hadoop Gridmix ………………………… SKIPPED [INFO] Apache Hadoop Data Join ………………………. SKIPPED [INFO] Apache Hadoop Ant Tasks ………………………. SKIPPED [INFO] Apache Hadoop Extras …………………………. SKIPPED [INFO] Apache Hadoop Pipes ………………………….. SKIPPED [INFO] Apache Hadoop OpenStack support ……………….. SKIPPED [INFO] Apache Hadoop Amazon Web Services support ………. SKIPPED [INFO] Apache Hadoop Azure support …………………… SKIPPED [INFO] Apache Hadoop Client …………………………. SKIPPED [INFO] Apache Hadoop Mini-Cluster ……………………. SKIPPED [INFO] Apache Hadoop Scheduler Load Simulator …………. SKIPPED [INFO] Apache Hadoop Tools Dist ……………………… SKIPPED [INFO] Apache Hadoop Tools ………………………….. SKIPPED [INFO] Apache Hadoop Distribution ……………………. SKIPPED [INFO] ———————————————————————— [INFO] BUILD FAILURE [INFO] ———————————————————————— [INFO] Total time: 52.054 s [INFO] Finished at: 2016-09-03T10:00:06+08:00 [INFO] Final Memory: 75M/244M [INFO] ———————————————————————— [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 [ERROR] around Ant part …… @ 4:125 in /opt/hadoop-2.7.2-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml [ERROR] -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1 around Ant part …… @ 4:125 in /opt/hadoop-2.7.2-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 1 around Ant part …… @ 4:125 in /opt/hadoop-2.7.2-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) … 20 more Caused by: /opt/hadoop-2.7.2-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml:4: exec returned: 1 at org.apache.tools.ant.taskdefs.ExecTask.runExecute(ExecTask.java:646) at org.apache.tools.ant.taskdefs.ExecTask.runExec(ExecTask.java:672) at org.apache.tools.ant.taskdefs.ExecTask.execute(ExecTask.java:498) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:291) at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) at org.apache.tools.ant.Task.perform(Task.java:348) at org.apache.tools.ant.Target.execute(Target.java:390) at org.apache.tools.ant.Target.performTasks(Target.java:411) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399) at org.apache.tools.ant.Project.executeTarget(Project.java:1368) at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:327) … 22 more [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command

解决方法： yum install cmake yum install openssl-devel zlib-devel

之后再次执行： mvn package -Pdist,native -DskipTests -Dtar

等待。。。。。。。。。。。。。。 在我的电脑上40多分钟后出现如下结果 ：顺利编译完成。

main: [exec]

gzip -f hadoop-2.7.2.tar [exec] [exec] Hadoop dist tar available at: /opt/hadoop-2.7.2-src/hadoop-dist/target/hadoop-2.7.2.tar.gz [exec] [INFO] Executed tasks [INFO] [INFO] — maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist — [INFO] Building jar: /opt/hadoop-2.7.2-src/hadoop-dist/target/hadoop-dist-2.7.2-javadoc.jar [INFO] ———————————————————————— [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main …………………………… SUCCESS [ 1.812 s] [INFO] Apache Hadoop Project POM …………………….. SUCCESS [ 1.904 s] [INFO] Apache Hadoop Annotations …………………….. SUCCESS [ 3.932 s] [INFO] Apache Hadoop Assemblies ……………………… SUCCESS [ 0.249 s] [INFO] Apache Hadoop Project Dist POM ………………… SUCCESS [ 2.343 s] [INFO] Apache Hadoop Maven Plugins …………………… SUCCESS [ 3.942 s] [INFO] Apache Hadoop MiniKDC ………………………… SUCCESS [ 2.845 s] [INFO] Apache Hadoop Auth …………………………… SUCCESS [ 4.504 s] [INFO] Apache Hadoop Auth Examples …………………… SUCCESS [ 3.794 s] [INFO] Apache Hadoop Common …………………………. SUCCESS [03:13 min] [INFO] Apache Hadoop NFS ……………………………. SUCCESS [ 7.804 s] [INFO] Apache Hadoop KMS ……………………………. SUCCESS [09:51 min] [INFO] Apache Hadoop Common Project ………………….. SUCCESS [ 0.050 s] [INFO] Apache Hadoop HDFS …………………………… SUCCESS [03:35 min] [INFO] Apache Hadoop HttpFS …………………………. SUCCESS [06:29 min] [INFO] Apache Hadoop HDFS BookKeeper Journal ………….. SUCCESS [02:22 min] [INFO] Apache Hadoop HDFS-NFS ……………………….. SUCCESS [ 5.897 s] [INFO] Apache Hadoop HDFS Project ……………………. SUCCESS [ 0.041 s] [INFO] hadoop-yarn …………………………………. SUCCESS [ 0.113 s] [INFO] hadoop-yarn-api ……………………………… SUCCESS [ 45.280 s] [INFO] hadoop-yarn-common …………………………… SUCCESS [02:25 min] [INFO] hadoop-yarn-server …………………………… SUCCESS [ 0.031 s] [INFO] hadoop-yarn-server-common …………………….. SUCCESS [ 11.953 s] [INFO] hadoop-yarn-server-nodemanager ………………… SUCCESS [ 18.431 s] [INFO] hadoop-yarn-server-web-proxy ………………….. SUCCESS [ 3.784 s] [INFO] hadoop-yarn-server-applicationhistoryservice ……. SUCCESS [ 8.515 s] [INFO] hadoop-yarn-server-resourcemanager …………….. SUCCESS [ 25.016 s] [INFO] hadoop-yarn-server-tests ……………………… SUCCESS [ 6.034 s] [INFO] hadoop-yarn-client …………………………… SUCCESS [ 12.240 s] [INFO] hadoop-yarn-server-sharedcachemanager ………….. SUCCESS [ 6.293 s] [INFO] hadoop-yarn-applications ……………………… SUCCESS [ 0.022 s] [INFO] hadoop-yarn-applications-distributedshell ………. SUCCESS [ 3.841 s] [INFO] hadoop-yarn-applications-unmanaged-am-launcher ….. SUCCESS [ 3.042 s] [INFO] hadoop-yarn-site …………………………….. SUCCESS [ 0.045 s] [INFO] hadoop-yarn-registry …………………………. SUCCESS [ 13.733 s] [INFO] hadoop-yarn-project ………………………….. SUCCESS [ 8.739 s] [INFO] hadoop-mapreduce-client ………………………. SUCCESS [ 0.048 s] [INFO] hadoop-mapreduce-client-core ………………….. SUCCESS [ 27.219 s] [INFO] hadoop-mapreduce-client-common ………………… SUCCESS [ 29.534 s] [INFO] hadoop-mapreduce-client-shuffle ……………….. SUCCESS [ 6.170 s] [INFO] hadoop-mapreduce-client-app …………………… SUCCESS [ 14.114 s] [INFO] hadoop-mapreduce-client-hs ……………………. SUCCESS [ 9.860 s] [INFO] hadoop-mapreduce-client-jobclient ……………… SUCCESS [ 40.504 s] [INFO] hadoop-mapreduce-client-hs-plugins …………….. SUCCESS [ 2.173 s] [INFO] Apache Hadoop MapReduce Examples ………………. SUCCESS [ 6.295 s] [INFO] hadoop-mapreduce …………………………….. SUCCESS [ 2.545 s] [INFO] Apache Hadoop MapReduce Streaming ……………… SUCCESS [ 17.347 s] [INFO] Apache Hadoop Distributed Copy ………………… SUCCESS [ 34.758 s] [INFO] Apache Hadoop Archives ……………………….. SUCCESS [ 2.437 s] [INFO] Apache Hadoop Rumen ………………………….. SUCCESS [ 6.878 s] [INFO] Apache Hadoop Gridmix ………………………… SUCCESS [ 5.061 s] [INFO] Apache Hadoop Data Join ………………………. SUCCESS [ 2.651 s] [INFO] Apache Hadoop Ant Tasks ………………………. SUCCESS [ 2.600 s] [INFO] Apache Hadoop Extras …………………………. SUCCESS [ 3.498 s] [INFO] Apache Hadoop Pipes ………………………….. SUCCESS [ 6.683 s] [INFO] Apache Hadoop OpenStack support ……………….. SUCCESS [ 5.276 s] [INFO] Apache Hadoop Amazon Web Services support ………. SUCCESS [04:44 min] [INFO] Apache Hadoop Azure support …………………… SUCCESS [ 33.704 s] [INFO] Apache Hadoop Client …………………………. SUCCESS [ 8.468 s] [INFO] Apache Hadoop Mini-Cluster ……………………. SUCCESS [ 0.118 s] [INFO] Apache Hadoop Scheduler Load Simulator …………. SUCCESS [ 4.912 s] [INFO] Apache Hadoop Tools Dist ……………………… SUCCESS [ 10.041 s] [INFO] Apache Hadoop Tools ………………………….. SUCCESS [ 0.077 s] [INFO] Apache Hadoop Distribution ……………………. SUCCESS [ 30.884 s] [INFO] ———————————————————————— [INFO] BUILD SUCCESS [INFO] ———————————————————————— [INFO] Total time: 41:25 min [INFO] Finished at: 2016-09-03T10:54:41+08:00 [INFO] Final Memory: 113M/444M