文档编写目的
在前面的文档中,介绍了在有CM和无CM两种情况下使用rpm方式安装CDH5.10.0,本文档将介绍如何在无CM的情况下使用rpm方式安装CDH6.2.0,与之前安装C5进行对比。
环境介绍:
安装前置准备
2.1 服务器相关设置
安装CDH集群时需要做一些前置的准备,本次安装使用的环境已经做好前置准备,需要做的准备如下:
1.hosts以及hostname配置正确
2.服务器没有启用IPv6且配置了静态IP
3.禁用SELinux
4.关闭防火墙
5.设置swappiness为1
6.关闭透明大页面
7.配置NTP时钟同步
2.2 配置本地yum源
1.在官网下载好需要的rpm包,地址如下:
https://archive.cloudera.com/cdh6/6.2.0/redhat7/yum/RPMS/
将上面所有的rpm包下载到服务器,如下:
在浏览器进行验证
2.执行createrepo命令
createrepo .
3.创建repo文件
[cdhrepo]
name = cdh_repo
baseurl = http://192.168.0.178/cdh6_rpm/
enable = true
gpgcheck = false
4.执行yum命令,查看本地yum源是否配置成功
yum clean all
yum repolist
上图可以看到,下载的rpm包制作的本地yum源成功
CDH组件安装
3.1 Zookeeper
1.在所有节点安装Zookeeper
yum -y install zookeeper
2.创建数据目录并修改属主
mkdir -p /var/lib/zookeeper
chown -R zookeeper /var/lib/zookeeper
3.修改配置文件/etc/zookeeper/conf/zoo.cfg
maxClientCnxns=60
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/lib/zookeeper
clientPort=2181
dataLogDir=/var/lib/zookeeper
minSessionTimeout=4000
maxSessionTimeout=40000
server.1=cdh178.macro.com:3181:4181
server.2=cdh177.macro.com:3181:4181
server.3=cdh176.macro.com:3181:4181
保存修改并同步到所有节点
4.所有节点创建myid文件并修改属主
5.所有节点启动Zookeeper
/usr/lib/zookeeper/bin/zkServer.sh start
查看所有节点启动状态,三个节点均启动成功
/usr/lib/zookeeper/bin/zkServer.sh status
至此Zookeeper安装完成
3.2 HDFS
1.在所有节点安装HDFS必需的包,由于只有三个节点,所以三个节点都安装DataNode
yum -y install hadoop hadoop-hdfs hadoop-client hadoop-doc hadoop-debuginfo hadoop-hdfs-datanode
2.在一个节点安装NameNode以及SecondaryNameNode
yum -y install hadoop-hdfs-namenode hadoop-hdfs-secondarynamenode
3.创建数据目录并修改属主和权限
所有节点创建DataNode的目录
mkdir -p /data0/dfs/dn
chown -R hdfs:hadoop /data0/dfs/dn
chmod 700 /data0/dfs/dn
NameNode和SecondaryNameNode节点创建数据目录
mkdir -p /data0/dfs/nn
chown -R hdfs:hadoop /data0/dfs/nn
chmod 700 /data0/dfs/nn
mkdir -p /data0/dfs/snn
chown -R hdfs:hadoop /data0/dfs/snn
chmod 700 /data0/dfs/snn
4.修改配置文件
/etc/hadoop/conf/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://cdh178.macro.com:8020</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
</property>
</configuration>
/etc/hadoop/conf/hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data0/dfs/nn</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data0/dfs/dn</value>
</property>
<property>
<name>dfs.namenode.servicerpc-address</name>
<value>cdh178.macro.com:8022</value>
</property>
<property>
<name>dfs.https.address</name>
<value>cdh178.macro.com:9871</value>
</property>
<property>
<name>dfs.secondary.http.address</name>
<value>cdh178.macro.com:50090</value>
</property>
<property>
<name>dfs.https.port</name>
<value>9871</value>
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>cdh178.macro.com:9870</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:///data0/dfs/snn</value>
</property>
</configuration>
5.将修改的配置文件保存并同步到所有节点
6.格式化NameNode
sudo -u hdfs hdfs namenode -format
7.在所有节点运行命令启动HDFS
systemctl start hadoop-hdfs-namenode
systemctl start hadoop-hdfs-secondarynamenode
systemctl start hadoop-hdfs-datanode
systemctl status hadoop-hdfs-namenode
systemctl status hadoop-hdfs-secondarynamenode
systemctl status hadoop-hdfs-datanode
8.创建/tmp临时目录,并设置目录权限,然后使用hadoop命令查看创建的目录成功
sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
9.访问NameNode的Web UI
至此HDFS安装完成
3.3 Yarn
1.安装Yarn的包,在一个节点安装ResourceManager和JobHistory Server,所有节点安装NodeManager
yum -y install hadoop-yarn hadoop-yarn-resourcemanager hadoop-mapreduce-historyserver hadoop-yarn-proxyserver hadoop-mapreduce
yum -y install hadoop-yarn hadoop-yarn-nodemanager hadoop-mapreduce
2.创建目录并修改属主和权限
在所有节点创建本地目录
mkdir -p /data0/yarn/nm
chown yarn:hadoop /data0/yarn/nm
mkdir -p /data0/yarn/container-logs
chown yarn:hadoop /data0/yarn/container-logs
在HDFS上创建logs目录
sudo -u hdfs hdfs dfs -mkdir /tmp/logs
sudo -u hdfs hdfs dfs -chown mapred:hadoop /tmp/logs
sudo -u hdfs hdfs dfs -chmod 1777 /tmp/logs
在HDFS上创建/user/history目录
sudo -u hdfs hdfs dfs -mkdir -p /user
sudo -u hdfs hdfs dfs -chmod 777 /user
sudo -u hdfs hdfs dfs -mkdir -p /user/history
sudo -u hdfs hdfs dfs -chown mapred:hadoop /user/history
sudo -u hdfs hdfs dfs -chmod 1777 /user/history
sudo -u hdfs hdfs dfs -mkdir -p /user/history/done
sudo -u hdfs hdfs dfs -mkdir -p /user/history/done_intermediate
sudo -u hdfs hdfs dfs -chown -R mapred:hadoop /user/history
sudo -u hdfs hdfs dfs -chmod 771 /user/history/done
sudo -u hdfs hdfs dfs -chmod 1777 /user/history/done_intermediate
3.修改配置文件
/etc/hadoop/conf/yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///data0/yarn/nm</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>file:///data0/yarn/container-logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/tmp/logs</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>cdh178.macro.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>cdh178.macro.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>cdh178.macro.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>cdh178.macro.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>cdh178.macro.com:8088</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>cdh178.macro.com:8090</value>
</property>
</configuration>
/etc/hadoop/conf/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>cdh178.macro.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>cdh178.macro.com:19888</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.https.address</name>
<value>cdh178.macro.com:19890</value>
</property>
<property>
<name>mapreduce.jobhistory.admin.address</name>
<value>cdh178.macro.com:10033</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/user</value>
</property>
</configuration>
/etc/hadoop/conf/core-site.xml,下面只贴出修改的部分配置
<property>
<name>hadoop.proxyuser.mapred.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.mapred.hosts</name>
<value>*</value>
</property>
4.将配置文件保存后同步到所有节点
5.启动Yarn服务
在JobHistoryServer节点上启动mapred-historyserver
/etc/init.d/hadoop-mapreduce-historyserver start
在RM节点启动ResourceManager
systemctl start hadoop-yarn-resourcemanager
systemctl status hadoop-yarn-resourcemanager
在NM节点启动NodeManager
systemctl start hadoop-yarn-nodemanager
systemctl status hadoop-yarn-nodemanager
6.访问Yarn服务的Web UI
Yarn的管理页面
JobHistory的管理页面
查看在线的节点
7.运行MR示例程序
使用root用户运行示例程序,所以要先创建root用户的目录
sudo -u hdfs hdfs dfs -mkdir /user/root
sudo -u hdfs hdfs dfs -chown root:root /user/root
运行MR示例程序,运行成功
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 5 5
至此Yarn服务安装完成
3.4 Spark
1.安装Spark所需的包
yum -y install spark-core spark-master spark-worker spark-history-server spark-python
2.创建目录并修改属主和权限
sudo -u hdfs hadoop fs -mkdir /user/spark
sudo -u hdfs hadoop fs -mkdir /user/spark/applicationHistory
sudo -u hdfs hadoop fs -chown -R spark:spark /user/spark
sudo -u hdfs hadoop fs -chmod 1777 /user/spark/applicationHistory
3.修改配置文件/etc/spark/conf/spark-defaults.conf
spark.eventLog.enabled=true
spark.eventLog.dir=hdfs://cdh178.macro.com:8020/user/spark/applicationHistory
spark.yarn.historyServer.address=http://cdh178.macro.com:18088
4.启动spark-history-server
systemctl start spark-history-server
systemctl status spark-history-server
访问Web UI
5.修改配置文件并同步到所有节点
6.测试Spark使用
至此Spark安装完成
3.5 Hive
1.安装Hive服务之前,先安装元数据库MySQL并创建好服务需要的库和用户如下:
create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON metastore.* TO 'hive'@'%';
FLUSH PRIVILEGES;
2.安装Hive服务的包
在NameNode节点hive-metastore
yum -y install hive-metastore
在所有节点安装其他所需的包
yum -y install hive hive-server2 hive-jdbc hive-hbase
3.创建目录
在HDFS上创建目录并设置权限以及修改属主
sudo -u hdfs hadoop fs -mkdir /user/hive
sudo -u hdfs hadoop fs -chown hive:hive /user/hive
sudo -u hdfs hadoop fs -mkdir /user/hive/warehouse
sudo -u hdfs hadoop fs -chmod 1777 /user/hive/warehouse
sudo -u hdfs hadoop fs -chown hive:hive /user/hive/warehouse
4.修改配置文件
/etc/hive/conf/hive-site.xml
<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://cdh178.macro.com:3306/metastore?useUnicode=true&characterEncoding=UTF-8</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>password</value>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>false</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>cdh178.macro.com:8031</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>hive.exec.reducers.max</name>
<value>1099</value>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<property>
<name>hive.warehouse.subdir.inherit.perms</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.server.min.threads</name>
<value>200</value>
</property>
<property>
<name>hive.metastore.server.max.threads</name>
<value>100000</value>
</property>
<property>
<name>hive.metastore.client.socket.timeout</name>
<value>3600</value>
</property>
<property>
<name>hive.support.concurrency</name>
<value>true</value>
</property>
<property>
<name>hive.zookeeper.quorum</name>
<value>cdh178.macro.com,cdh177.macro.com,cdh176.macro.com</value>
</property>
<property>
<name>hive.zookeeper.client.port</name>
<value>2181</value>
</property>
</configuration>
/etc/hadoop/conf/core-site.xml,只贴出修改的部分
<property>
<name>hadoop.proxyuser.hive.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hive.groups</name>
<value>*</value>
</property>
5.将配置文件同步到所有节点
6.将MySQL驱动包在Hive服务的lib目录下设置软链
7.启动Hive服务
启动hive-metastore
systemctl start hive-metastore
systemctl status hive-metastore
启动hive-server2
systemctl start hive-server2
systemctl status hive-server2
8.测试Hive服务是否正常
连接Hive,建表正常
至此Hive安装完成
3.6 Oozie
1.在MySQL中创建Oozie服务所需要的库和用户
create database oozie default character set utf8;
CREATE USER 'oozie'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON oozie.* TO 'oozie'@'%';
FLUSH PRIVILEGES;
2.安装Oozie的包
yum -y install oozie oozie-client
3.配置Oozie
配置Oozie使用Yarn
alternatives --set oozie-tomcat-deployment /etc/oozie/tomcat-conf.http
修改/etc/oozie/conf/oozie-site.xml配置文件
<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://cdh178.macro.com:3306/oozie</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>oozie</value>
</property>
<property>
<name>oozie.service.JPAService.jdbc.password</name>
<value>password</value>
</property>
将MySQL驱动包在Oozie目录下生成软链
4.运行Oozie数据库工具
sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -run
5.配置Oozie的Web控制台
下载ExtJS library到服务器,地址如下:
https://archive.cloudera.com/gplextras/misc/ext-2.2.zip
将下载的包解压到/var/lib/oozie
unzip ext-2.2.zip -d /var/lib/oozie/
6.在HDFS中安装Oozie共享库
sudo -u hdfs hadoop fs -mkdir /user/oozie
sudo -u hdfs hadoop fs -chown oozie:oozie /user/oozie
sudo oozie-setup sharelib create -fs hdfs://cdh178.macro.com:8020 -locallib /usr/lib/oozie/oozie-sharelib-yarn
7.启动Oozie Server
systemctl start oozie
systemctl status oozie
8.访问Oozie服务的Web UI
至此Oozie服务安装完成
3.7 Impala
1.安装Impala的包
在一个节点上安装Impala Catalog Server和Impala StateStore
yum -y install impala-state-store impala-catalog
在所有节点安装其他的包
yum -y install impala impala-server
2.将Impala需要的配置文件拷贝到Impala的配置文件目录下
3.安装impala-shell
yum -y install impala-shell
4.安装完Impala后需要的配置
修改/etc/hadoop/conf/hdfs-site.xml配置文件,启用块位置追踪和短路读取
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hdfs-sockets/dn</value>
</property>
<property>
<name>dfs.client.file-block-storage-locations.timeout.millis</name>
<value>10000</value>
</property>
将配置同步到所有节点
重启所有DataNode
将修改后的hdfs-site.xml复制到Impala的配置文件目录
5.启动Impala服务
启动Impala Catalog Server和Impala StateStore
systemctl start impala-state-store
systemctl status impala-state-store
systemctl start impala-catalog
systemctl status impala-catalog
所有节点启动impala-server
systemctl start impala-server
systemctl status impala-server
6.测试Impala使用
使用impala-shell连接Impala,进行插入、查询操作成功
至此Impala安装完成
3.8 Hue
1.安装Hue的包
yum -y install hue
2.为Hue配置CDH组件
1)修改配置文件
/etc/hadoop/conf/hdfs-site.xml
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
/etc/hadoop/conf/core-site.xml
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.groups</name>
<value>*</value>
</property>
/etc/hue/conf/hue.ini
将修改的HDFS的配置文件同步到所有节点
2)重启HDFS服务
systemctl restart hadoop-hdfs-namenode
systemctl restart hadoop-hdfs-secondarynamenode
systemctl restart hadoop-hdfs-datanode
修改配置文件/etc/hue/conf/hue.ini
3.创建Hue服务所需的数据库和用户
create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON hue.* TO 'hue'@'%';
FLUSH PRIVILEGES;
修改配置文件/etc/hue/conf/hue.ini数据库的部分
4.初始化数据库
/usr/lib/hue/build/env/bin/hue syncdb
/usr/lib/hue/build/env/bin/hue migrate
5.启动Hue服务
systemctl start hue
systemctl status hue
6.访问Hue服务的Web UI
至此Hue服务安装完成
总结
1.无CM使用rpm的方式安装CDH6.2.0与之前安装CDH5.10.0基本没有太大的区别。
2.此安装方式需要下载相关的所有rpm包到服务器,然后制作本地的yum源进行安装,下载的包的总大小在4.3G左右。
3.同样的在安装过程中需要最先安装Zookeeper。
Fayson的github: https://github.com/fayson/cdhproject