本篇的目的是:梳理一份CentOS/RHEL7+11gR2 RAC部署的较标准文档。 结合了公司内部文档及mos等多方材料,尽可能梳理完整。
以下环境在VMware Workstation 15 Pro 下验证实验
1. 主机环境规划:
设置项 | 版本 |
---|---|
操作系统 | CentOS Linux release 7.7.1908 (Core) |
GRID 版本 | GI 11.2.0.4 |
数据库版本 | Oracle EE 11.2.0.4 |
db_name | xkdb |
节点数 | 2 Nodes RAC |
内存 | 4G |
SWAP | 4G |
2. 共享存储规划:
磁盘组 | 冗余方式 | 用途 | 大小 |
---|---|---|---|
OCR | Normal | OCR,Votedisk表决磁盘等 | 1Gx3 |
DATA | External | 数据文件、控制文件等 | 20G |
FRA | External | 存放归档、闪回文件等 | 5G |
3. IP地址规划:
节点 hostname | xk1 | xk2 |
---|---|---|
public ip | 192.168.10.60 | 192.168.10.61 |
private ip | 10.10.10.10 | 10.10.10.11 |
vip | 192.168.10.62 | 192.168.10.63 |
scan ip | 192.168.10.65 | 192.168.10.65 |
4. 虚拟机设置:
注意:
这部分是软件安装前的操作系统参数配置,分别在两个节点操作
yum -y install binutils compat-libstdc++-33 gcc gcc-c++ glibc glibc-common glibc-devel ksh libaio libaio-devel libgcc libstdc++ libstdc++-devel make sysstat openssh-clients compat-libcap1 xorg-x11-utils xorg-x11-xauth elfutils unixODBC unixODBC-devel libXp elfutils-libelf elfutils-libelf-devel smartmontools
--关闭防火墙:
systemctl stop firewalld
--禁止开机启动:
systemctl disable firewalld.service
参考文档:How to Disable or set SELinux to Permissive mode (Doc ID 457458.1)
setenforce 0
vi /etc/selinux/config
--更改以下内容为disabled,重启操作系统才生效
SELINUX=disabled
rac环境下会导致节点重启及性能问题 查看设置前情况
cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
修改设置,加入如下配置
vi /etc/rc.d/rc.local
if test -f /sys/kernel/mm/transparent_hugepage/enabled;then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag;then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
chmod +x /etc/rc.d/rc.local
设置后需要重启系统
groupadd oinstall -g 1001
groupadd dba -g 1031
groupadd oper -g 1032
groupadd asmadmin -g 1020
groupadd asmdba -g 1021
groupadd asmoper -g 1022
useradd -g oinstall -G asmadmin,asmdba,asmoper,dba,oper -u 1100 grid
useradd -g oinstall -G dba,oper,asmadmin,asmdba -u 1101 oracle
mkdir -p /oracle/app/11.2.0/grid
mkdir -p /oracle/app/grid
mkdir -p /oracle/app
mkdir -p /oracle/app/oracle/product/11.2.0/dbhome_1
chown -R oracle:oinstall /oracle
chown -R grid:oinstall /oracle/app/11.2.0/grid
chown -R grid:oinstall /oracle/app/grid
chown -R oracle:oinstall /oracle/app/oracle
chmod 771 /oracle/
chmod 771 /oracle/app
--修改grid、oracle用户密码
passwd grid
passwd oracle
内核参数的说明参照【安装】CentOS7.7下图形化安装Oracle11gR2 2.9小节说明
--以下参数按照4G物理内存情况下计算
--vi /etc/sysctl.conf
# Oracle install config
kernel.shmall = 1048576
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
##Huge page
vm.nr_hugepages = 1300
--生效命令
sysctl -p
注意:
--vi /etc/profile,加到unset i上方
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -u 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
--vi /etc/security/limits.conf
#SETTING for ORACLE
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 4096
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 4096
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
oracle hard memlock unlimited
oracle soft memlock unlimited
hosts文件是Linux系统上一个负责ip地址与域名快速解析的文件,hosts文件包含了ip地址与主机名之间的映射
--vi /etc/hosts
# Public
192.168.10.60 xk1
192.168.10.61 xk2
# Virtual
192.168.10.62 xk1-vip
192.168.10.63 xk2-vip
# Private
10.10.10.10 xk1-priv
10.10.10.11 xk2-priv
# Scan-ip
192.168.10.65 rac-scan
grid用户增加:
--vi ~/.bash_profile
export ORACLE_BASE=/oracle/app/grid
export ORACLE_HOME=/oracle/app/11.2.0/grid
export ORACLE_SID=+ASM1 --节点2是 +ASM2
export NLS_LANG=american_america.ZHS16GBK
export NLS_DATE_FORMAT="yyyy-mm-dd hh24:mi:ss"
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
umask 022
export TMOUT=0
oracle用户增加:
export ORACLE_BASE=/oracle/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1
export ORACLE_SID=xkdb1 --节点2是 xkdb2
export LANG=en_US.UTF-8
export NLS_LANG=american_america.ZHS16GBK
export NLS_DATE_FORMAT="yyyy-mm-dd hh24:mi:ss"
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin:/home/oracle/run
export ORACLE_TERM=xterm
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
export GI_HOME=/oracle/app/11.2.0/grid
export PATH=${PATH}:$GI_HOME/bin
export ORA_NLS10=$GI_HOME/nls/data
umask 022
export TMOUT=0
vi /etc/fstab
--没有的话加入下一行,size=物理内存大小
none /dev/shm tmpfs defaults,size=4G 0 0
--生效
mount -o remount /dev/shm
vi /etc/default/grub
在GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet numa=off" 这一行加上numa=off
--重新编译
grub2-mkconfig -o /etc/grub2.cfg
参考:
--/etc/pam.d/login
echo "session required pam_limits.so" >> /etc/pam.d/login
cat /etc/pam.d/login
为防止私有网路原因引起network HB丢失,导致root.sh无法成功执行,需要禁用linux上虚拟网卡,此处操作后在grid安装中就看不到virbr0了
参考: How to Remove virbr0 and lxcbr0 Interfaces on Oracle Linux (Doc ID 2197674.1)
/bin/systemctl stop libvirtd.service
/bin/systemctl disable libvirtd.service
/bin/systemctl status libvirtd.service
vi /etc/sysconfig/network
--增加如下条目
NOZEROCONF=yes
参考: How To Prevent Bogus Entry 169.254.0.0/255.255.0.0 Automatically Added To Routing Table (Doc ID 1161144.1)
nsswitch.conf保存了域名的检测顺序。某些情况下,NIS(Network Informaion System)可能会导致SCAN域名解析异常,所以需要编辑如下文件设置顺序
vi /etc/nsswitch.conf
--将hosts行修改如下
hosts: files dns myhostname nis
avahi-daemon该守护进程配合缓存用户程序的答复,以帮助减少因答复而产生的网络流量。 根据官方一些文档已经列出一些BUG,建议关闭。 只要上面NOZEROCONF配置为打开,则avahi-daemon就为其服务
systemctl stop avahi-daemon.socket avahi-daemon.service
systemctl disable avahi-daemon.socket avahi-daemon.service
p13390677_112040_Linux-x86-64_1of7.zip p13390677_112040_Linux-x86-64_2of7.zip p13390677_112040_Linux-x86-64_3of7.zip 前两个是DB安装包,第三个是grid安装包
cd /opt mkdir soft/ chmod 777 /opt/soft/ chmod 777 p13390677_112040_Linux-x86-64_3of7.zip unzip p13390677_112040_Linux-x86-64_3of7.zip
在VM虚拟机实现共享存储有两种方式
为模拟存储多路径功能,本篇采用Openfiler方式,请参考前序文章 【多路径】CentOS7.7下使用Openfiler+Multipath+UDEV
这种方式比较方便,平常自己搭建测试环境就用这种就好,UDEV同上
scsi1:0.deviceType = "disk"(多个磁盘的就加多条)
disk.locking="false"
diskLib.dataCacheMaxSize="0"
diskLib.dataCacheMaxReadAheadSize="0"
diskLib.DataCacheMinReadAheadSize="0"
diskLib.dataCachePageSize="4096"
diskLib.maxUnsyncedWrites="0"
scsi1.sharedBus="virtual"
scsi1:0.mode = "independent-persistent"(设置为独立模式,如果永久选项是灰色的话添加)
disk.EnableUUID = "TRUE"(启用UUID)
为解决grid安装BUG需要打两个补丁 参考:Installation walk-through - Oracle Grid/RAC 11.2.0.4 on Oracle Linux 7 (文档 ID 1951613.1) 将两个补丁上传到/opt/soft路径下
unzip /opt/soft/p19404309_112040_Linux-x86-64.zip
cp b19404309/grid/cvu_prereq.xml /soft/grid/stage/cvu/
--装一个包
cd /opt/soft/grid/rpm
rpm -ivh cvuqdisk-1.0.9-1.rpm
cd /opt/soft/grid
--执行命令(根据自己主机IP设置 export DISPLAY=192.168.10.1:0.0)
./runInstaller -jreLoc /etc/alternatives/jre_1.8.0
说明: ASM磁盘组使用的是默认的1M AU大小,对于大型数据库,这会造成较多的内存占用,同时对性能略微有些影响,建议对于新增的用于放置数据文件的ASM磁盘组,适当调大AU大小,比如4M或8M(2的幂值)。根据实际经验,建议设置AU 为4m
第一个脚本先在两个节点分别执行 第二个脚本执行前需要先打补丁,因为在7下执行脚本会报错,需要先打p18370031_112040_Linux-x86-64.zip这个补丁
cd /opt/soft
unzip p18370031_112040_Linux-x86-64.zip
--节点1:
cd $ORACLE_HOME/OPatch
./opatch napply -local /opt/soft/18370031
--节点2:
scp /opt/soft/p18370031_112040_Linux-x86-64.zip xk2:/opt/
解压安装同上,安装完成用opatch lsinventory命令检查两节点是否打上
下面继续执行第二个脚本,这里放一下执行第二个脚本的内容
[root@xk1 OPatch]# /oracle/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /oracle/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /oracle/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to oracle-ohasd.service
CRS-2672: Attempting to start 'ora.mdnsd' on 'xk1'
CRS-2676: Start of 'ora.mdnsd' on 'xk1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'xk1'
CRS-2676: Start of 'ora.gpnpd' on 'xk1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'xk1'
CRS-2672: Attempting to start 'ora.gipcd' on 'xk1'
CRS-2676: Start of 'ora.cssdmonitor' on 'xk1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'xk1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'xk1'
CRS-2672: Attempting to start 'ora.diskmon' on 'xk1'
CRS-2676: Start of 'ora.diskmon' on 'xk1' succeeded
CRS-2676: Start of 'ora.cssd' on 'xk1' succeeded
ASM created and started successfully.
Disk Group OCR created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 8d4334c64c694f49bfc095b9ca9a79df.
Successful addition of voting disk 1e4141dd3eab4febbf98bc9a277e4b8f.
Successful addition of voting disk 3d6746c493b54f79bf437ae9308aed30.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 8d4334c64c694f49bfc095b9ca9a79df (/dev/asmdisk1) [OCR]
2. ONLINE 1e4141dd3eab4febbf98bc9a277e4b8f (/dev/asmdisk2) [OCR]
3. ONLINE 3d6746c493b54f79bf437ae9308aed30 (/dev/asmdisk3) [OCR]
Located 3 voting disk(s).
CRS-2672: Attempting to start 'ora.OCR.dg' on 'xk1'
CRS-2676: Start of 'ora.OCR.dg' on 'xk1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
点击OK继续
到此grid软件安装完成
最后按照规划创建完成所有磁盘组并退出 将grid用户的ORACLE_HOME写入两个节点root用户的.bash_profile文件中,方便操作
su - root
vi .bash_profile
--添加到PATH行,修改后如下
PATH=$PATH:/oracle/app/11.2.0/grid/bin:$HOME/bin
--使生效
source .bash_profile
关于ASM的知识单独写一篇,到这里磁盘组创建完毕
之前已经将db软件上传到/opt/soft目录下,分别解压缩
--使用oracle用户解压
su - oracle
unzip p13390677_112040_Linux-x86-64_1of7.zip
unzip p13390677_112040_Linux-x86-64_2of7.zip
执行安装:
cd /opt/soft/database
./runInstaller -jreLoc /etc/alternatives/jre_1.8.0
安装到56%报错,这里为CentOS7下安装11gR2的bug,处理方式: /oracle/app/oracle/product/11.2.0/db_1/sysman/lib/ins_emagent.mk 先备份,再修改,只改节点1 搜索NMECTL这一行 在括号后面加上 -lnnz11 修改保存
继续执行,点击retry
完成
su - oracle
dbca
SGA+PGA要小于物理内存的80%,推荐范围在40-60% 本环境物理内存4G,留给grid集群1G,余3G用于分配SGA+PGA 3Gx80%=2.4G SGA=2.4x80%=1920M PGA=2.4.20%=480M 最后2.4/4=60%,在推荐范围内
完工
共享内存段、内核参数还要加深理解,最好从linux内存结构学起
本篇里没写时间同步服务,这块实验后单独写