大家好！通过Virtual Box搭建Oracle 11g R2 RAC学习测试环境，如需要Oracle 10g RAC搭建教程请查看上一篇教程——《Oracle 10g R2 RAC学习环境搭建教程》，文章中如有不足之处或者有任何疑问，请指出。

由于时间仓促，今天只写了“安装前的环境准备”，软件安装过程将在明天更新上去，期待各位的指点……

分享知识，只为共同进步^_^

1. 相关软件

OS:Oracle Linux 6.6 64bit

SoftWare(oracle 11g R2): Grid(Grid_linux_x86-64 11G R2 11.2.0.4)

Database(database_linux_x86-64 11G R2 11.2.0.4)

相关工具：CRT、Xmanager、rlwrap

注：安装环境为VirtualBox虚拟机

节点物理内存：4G

共享存储（ASM）：DATAS（50G）、BACKUP（50G）、OCR（1G*3）

AD&DNS&NTP：Win Server 2012 R2（192.168.31.31/24）

2. 安装OS注意事项【RAC1&RAC2】

分区：简单分区两个/和swap 注：swap分区为物理内存的两倍

软件包安装：

Base System > BaseBase System > Client management toolsBase System > Compatibility librariesBase System > Hardware monitoring utilitiesBase System > Large Systems PerformanceBase System > Network file system clientBase System > Performance ToolsBase System > Perl SupportServers > Server PlatformServers > System administration toolsDesktops > DesktopDesktops > Desktop PlatformDesktops > FontsDesktops > General Purpose DesktopDesktops > Graphical Administration ToolsDesktops > Input MethodsDesktops > X Window SystemApplications > Internet BrowserDevelopment > Additional DevelopmentDevelopment > Development Tools

字符界面设置【RAC1$RAC2】

[root@RAC1 ~]# vi /etc/inittab

[root@RAC2 ~]# vi /etc/inittab

id:3:initdefault:

关闭防火墙

service iptables stopchkconfig iptables offchkconfig --list iptablessetenforce 0getenforcevi /etc/sysconfig/selinuxSELINUX=disabled

关闭sendmail服务

service sendmail off chkconfig sendmail off chkconfig –list sendmail

3. 安装rlwrap软件【RAC1&RAC2】

[root@rac1 ~]# tar -zxvf rlwrap-0.42.tar.gz[root@rac1 ~]# ll[root@rac1 ~]# cd rlwrap-0.42[root@rac1 rlwrap-0.42]# ./configure[root@rac1 rlwrap-0.42]# make[root@rac1 rlwrap-0.42]# make install

注：rlwrap软件安装后，就可以在sqlplus或者rman中使用上下箭头翻页查找执行过的语句会在oracle用户下的.bash_profile后面加alias sqlplus='rlwrap sqlplus’、alias rman='rlwrap rman'等等

4. 安装oracle软件需要的RPM包【RAC1&RAC2】

[root@RAC1 Packages]# mount /dev/cdrom /tmp/[root@RAC1 Packages]# cd /tmp/Server/Packagesrpm -Uvh binutils-2.*rpm -Uvh compat-libstdc++-33*rpm -Uvh elfutils-libelf-0.*rpm -Uvh libaio-0.*rpm -Uvh libaio-devel-0.*rpm -Uvh sysstat-9.*rpm -Uvh glibc-2.*rpm -Uvh glibc-common-2.*rpm -Uvh glibc-devel-2.* glibc-headers-2.*rpm -Uvh ksh-2*rpm -Uvh make-3.*rpm -Uvh libgcc-4.*rpm -Uvh libstdc++-4.*rpm -Uvh libstdc++-4.*.i686*rpm -Uvh libstdc++-devel-4.*rpm -Uvh gcc-4.*x86_64*rpm -Uvh gcc-c++-4.*x86_64*rpm -Uvh --allfiles elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*rpm -Uvh elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*rpm -Uvh libtool-ltdl*i686*rpm -Uvh ncurses*i686*rpm -Uvh readline*i686*rpm -Uvh unixODBC*

5. 修改hosts文件【RAC1&RAC2】

vi /etc/hosts

192.168.31.11 rac1 rac1.cwucao.com192.168.31.12 rac2 rac2.cwucao.com 192.168.31.13 rac1-vip rac1-vip.cwucao.com192.168.31.14 rac2-vip rac2-vip.cwucao.com 10.0.0.1 rac1-priv rac1-priv.cwucao.com10.0.0.2 rac2-priv rac2-priv.cwucao.com

测试【RAC1&RAC2】：

ping rac1 ping rac1.cwucao.com ping rac2 ping rac2.cwucao.comping rac1-priv ping rac1-priv.cwucao.comping rac2-priv ping rac2-priv.cwucao.com

在DNS服务器中添加三个主机记录

192.168.31.15/24 scan-cluster.cwucao.com192.168.31.16/24 scan-cluster.cwucao.com192.168.31.17/24 scan-cluster.cwucao.com

测试【RAC1&RAC2】：

nslookup scan-cluster

6. 设置内核参数和其它参数【RAC1&RAC2】

[root@RAC1 Packages]# vi /etc/sysctl.conf

# Disable netfilter on bridges.#net.bridge.bridge-nf-call-ip6tables = 0#net.bridge.bridge-nf-call-iptables = 0#net.bridge.bridge-nf-call-arptables = 0 # Controls the maximum number of shared memory segments, in pages#kernel.shmall = 4294967296 # oracle-rdbms-server-11gR2-preinstall setting for fs.file-max is 6815744fs.file-max = 6815744 # oracle-rdbms-server-11gR2-preinstall setting for kernel.sem is '250 32000 100 128'kernel.sem = 250 32000 100 128 # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmni is 4096kernel.shmmni = 4096 # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmall is 1073741824 on x86_64# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmall is 2097152 on i386kernel.shmall = 1073741824 # oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4398046511104 on x86_64# oracle-rdbms-server-11gR2-preinstall setting for kernel.shmmax is 4294967295 on i386kernel.shmmax = 4398046511104 # oracle-rdbms-server-11gR2-preinstall setting for net.core.rmem_default is 262144net.core.rmem_default = 262144 # oracle-rdbms-server-11gR2-preinstall setting for net.core.rmem_max is 4194304net.core.rmem_max = 4194304 # oracle-rdbms-server-11gR2-preinstall setting for net.core.wmem_default is 262144net.core.wmem_default = 262144 # oracle-rdbms-server-11gR2-preinstall setting for net.core.wmem_max is 1048576net.core.wmem_max = 1048576 # oracle-rdbms-server-11gR2-preinstall setting for fs.aio-max-nr is 1048576fs.aio-max-nr = 1048576 # oracle-rdbms-server-11gR2-preinstall setting for net.ipv4.ip_local_port_range is 9000 65500net.ipv4.ip_local_port_range = 9000 65500

[root@RAC1 Packages]# sysctl -p

使上面修改的配置生效

[root@RAC1 Packages]# vi /etc/security/limits.conf

# oracle-rdbms-server-11gR2-preinstall setting for nofile soft limit is 1024#oracle soft nofile 1024 # oracle-rdbms-server-11gR2-preinstall setting for nofile hard limit is 65536#oracle hard nofile 65536 # oracle-rdbms-server-11gR2-preinstall setting for nproc soft limit is 16384# refer orabug15971421 for more info.#oracle soft nproc 16384 # oracle-rdbms-server-11gR2-preinstall setting for nproc hard limit is 16384#oracle hard nproc 16384 # oracle-rdbms-server-11gR2-preinstall setting for stack soft limit is 10240KBoracle soft stack 10240grid soft stack 10240 # oracle-rdbms-server-11gR2-preinstall setting for stack hard limit is 32768KBoracle hard stack 32768grid hard stack 32768 grid soft nproc 2047grid hard nproc 16384grid soft nofile 1024grid hard nofile 65536oracle soft nproc 2047oracle hard nproc 16384oracle soft nofile 1024oracle hard nofile 65536

[root@RAC1 Packages]# vi /etc/pam.d/login

session required /lib64/security/pam_limits.sosession required pam_limits.so

注意：因为是64位的系统，所以一定要写/lib64/security/pam_limits.so

如果写成/lib/security/pam_limits.so（这是32位系统的设置）

[root@RAC1 Packages]# vi /etc/profile

if [ $USER = "oracle" ] || [ $USER = "grid" ]; thenif [ $SHELL = "/bin/ksh" ]; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fiumask 022fi

执行source /etc/profile命令使配置生效

[root@RAC1 Packages]# vi /etc/csh.login

if ( $USER = "oracle" || $USER = "grid" ) thenlimit maxproc 16384limit descriptors 65536endifEOFCSH

[root@RAC1 Packages]# vi /etc/security/limits.d/90-nproc.conf

#* soft nproc 1024* - nproc 16384

7. 配置NTP服务，实现网络时间同步

1）配置Win server 2012 R2为NTP服务器

1.修改以下选项的键值HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\TimeProviders\NtpServer内的[Enabled]设定为1。

打开NTP服务器功能（默认是不开启NTP Server服务，除非电脑升级成为域控制站）。

2.修改以下键值HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config\AnnounceFlags设定为5．

该设定强制主机将它自身宣布为可靠的时间源，从而使用内置的互补金属氧化物半导体 (CMOS) 时钟。如果要采用外面的时间服务器就用默认的a值即可．

3.修改以下键值HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Parameters\Type内的[Type]设定为NTP。

4.重启Win32Time服务：先关闭 windows time 服务，再开启该服务。可以在“管理工具”的“服务”界面下完成，也可以以 DOS 方式输入“net stop w32time”、“net start w32time”。

至此，已完成服务器端设定．

2）配置节点【RAC1&RAC2】

[root@rac1 ~]# service ntpd statusntpd is stoppedchkconfig ntpd offmv /etc/ntp.conf /etc/ntp.conf.original[root@rac1 ~]# ntpdate 192.168.31.3130 Aug 19:40:31 ntpdate[12102]: step time server 192.168.75.1 offset -28799.377040 sec[root@rac1 ~]# crontab –e* * * * * ntpdate 192.168.31.31

这个脚本表示每分钟同步一次，crontab脚本间隔最短时间只能设置为1分钟

[root@rac1 ~]# crontab -l* * * * * ntpdate 192.168.31.31[root@rac1 ~]# service crond restartStopping crond: [ OK ]Starting crond: [ OK ]

查看crontab的运行日志

[root@rac1 ~]# tail -10f /var/log/cron

8. 确认物理内存和交换内存信息

[root@RAC1 ~]# grep MemTotal /proc/meminfoMemTotal: 3785084 kB[root@RAC1 ~]# grep SwapTotal /proc/meminfo SwapTotal: 8388604 kB

9. 创建用户和组【RAC1&RAC2】

userdel -r oraclegroupdel dbagroupdel oinstall groupadd -g 501 oinstallgroupadd -g 502 dbagroupadd -g 503 opergroupadd -g 504 asmadmingroupadd -g 506 asmdbagroupadd -g 507 asmoperuseradd -u 501 -g oinstall -G dba,asmadmin,asmdba,asmoper griduseradd -u 502 -g oinstall -G dba,asmdba,oper oracle passwd gridpasswd oracle

10. 创建软件目录【RAC1&RAC2】

创建Oracle RAC 所使用的目录，并修改相应的权限，在所有节点执行；

mkdir -p /u01/app/grid/mkdir -p /u01/app/11.2.0/grid/product/db_1mkdir -p /u01/app/oraInventorychown -R grid:oinstall /u01/appmkdir -p /u01/app/oracle/product/11.2.0/db_1chown -R oracle:oinstall /u01/app/oraclechmod -R 775 /u01

注：grid用户的ORACLE_BASE不能包含ORACLE_HOME,比如 /u01/grid,/u01/grid/11.2.0就不行

11. 修改grid用户环境变量【RAC1&RAC2】

[root@RAC1 ~]# su - grid

[grid@RAC1 ~]$ cd /home/grid/

[grid@RAC1 ~]$ vi .bash_profile

TMP=/tmp; export TMPTMPDIR=$TMP; export TMPDIR ORACLE_SID=+ASM1; export ORACLE_SIDORACLE_BASE=/u01/app/grid; export ORACLE_BASEORACLE_HOME=/u01/app/11.2.0/grid/product/db_1; export ORACLE_HOMENLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMATTHREADS_FLAG=native; export THREADS_FLAGPATH=$ORACLE_HOME/bin:$PATH; export PATH if [ $USER = "oracle" ] || [ $USER = "grid" ]; thenif [ $SHELL = "/bin/ksh" ]; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fiumask 022fi export DISPLAY=192.168.31.104:0.0

[grid@RAC1 ~]$ . .bash_profile --使上面修改的配置生效

12. 修改oracle用户环境变量【RAC1&RAC2】

[root@RAC1 ~]# su - oracle

[oracle@RAC1 ~]$ cd /home/oracle/

[oracle@RAC1 ~]$ vi .bash_profile

TMP=/tmp; export TMPTMPDIR=$TMP; export TMPDIR ORACLE_HOSTNAME=RAC1.cwucao.com; export ORACLE_HOSTNAMEORACLE_UNQNAME=racdb; export ORACLE_UNQNAMEORACLE_BASE=/u01/app/oracle; export ORACLE_BASEORACLE_HOME=$ORACLE_BASE/product/11.2.0/db_1; export ORACLE_HOMEORACLE_SID=racdb1; export ORACLE_SIDORACLE_TERM=xterm; export ORACLE_TERM PATH=/usr/sbin:$PATH; export PATHPATH=$ORACLE_HOME/bin:$PATH; export PATH LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATHCLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATHNLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMATNLS_LANG=AMERICAN_AMERICA.ZHS16GBK; export NLS_LANG if [ $USER = "oracle" ] || [ $USER = "grid" ]; thenif [ $SHELL = "/bin/ksh" ]; thenulimit -p 16384ulimit -n 65536elseulimit -u 16384 -n 65536fiumask 022fi export DISPLAY=192.168.31.104:0.0alias sqlplus='rlwrap sqlplus'alias rman='rlwrap rman'

[oracle@RAC1 ~]$ . .bash_profile --使上面修改的配置生效

查看环境变量已经生效

env | grep ORA

13. 配置用户等效性

在rac1上

[root@rac1 ~]# su - grid[grid@rac1 ~]$ mkdir ~/.ssh[grid@rac1 ~]$ chmod 755 ~/.ssh[grid@rac1 ~]$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/grid/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_rsa.Your public key has been saved in /home/grid/.ssh/id_rsa.pub.The key fingerprint is:e4:a4:16:17:ce:66:9e:ac:53:0f:92:02:9e:a3:23:89 grid@rac1.cwucao.com[grid@rac1 ~]$ ssh-keygen -t dsaGenerating public/private dsa key pair.Enter file in which to save the key (/home/grid/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_dsa.Your public key has been saved in /home/grid/.ssh/id_dsa.pub.The key fingerprint is:5a:8a:b5:7b:6e:f9:bd:17:2c:a9:f2:76:16:d5:03:54 grid@rac1.cwucao.com

在rac2上

[root@rac2 ~]# su - grid[grid@rac2 ~]$ mkdir ~/.ssh[grid@rac2 ~]$ chmod 755 ~/.ssh[grid@rac2 ~]$ ssh-keygen -t rsaGenerating public/private rsa key pair.Enter file in which to save the key (/home/grid/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_rsa.Your public key has been saved in /home/grid/.ssh/id_rsa.pub.The key fingerprint is:ec:73:d6:01:68:7e:be:92:dc:08:ec:c0:26:dd:b9:4e grid@rac2.cwucao.com[grid@rac2 ~]$ ssh-keygen -t dsaGenerating public/private dsa key pair.Enter file in which to save the key (/home/grid/.ssh/id_dsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/grid/.ssh/id_dsa.Your public key has been saved in /home/grid/.ssh/id_dsa.pub.The key fingerprint is:09:5f:88:8b:0f:cf:df:39:93:dc:c7:e3:fe:2e:92:d7 grid@rac2.cwucao.com

在rac1上

[grid@rac1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys[grid@rac1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys[grid@rac1 ~]$ ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keysThe authenticity of host 'rac2 (192.168.20.32)' can't be established.RSA key fingerprint is 6e:5b:7d:2c:d7:d2:e3:ca:59:79:59:73:22:da:8b:77.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added 'rac2,192.168.20.32' (RSA) to the list of known hosts.grid@rac2's password: [grid@rac1 ~]$ ssh rac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keysgrid@rac2's password: [grid@rac1 ~]$ scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keysgrid@rac2's password: authorized_keys 100% 2032 2.0KB/s 00:00 

在两个节点中，验证ssh

注：第一次執行，需輸入yes.再次執行後，即可

[grid@rac1 ~]$ ssh rac1 date[grid@rac1 ~]$ ssh rac2 date[grid@rac1 ~]$ ssh rac1-priv date[grid@rac1 ~]$ ssh rac2-priv date[grid@rac2 ~]$ ssh rac1 date[grid@rac2 ~]$ ssh rac2 date[grid@rac2 ~]$ ssh rac1-priv date[grid@rac2 ~]$ ssh rac2-priv date

注：oracle用户也需要进行用户等效性，执行上面的操作即可

14. 安装并配置ASM 驱动

查询系统信息

uname -alsb_release -a

确保电脑安装以下软件包

[root@RAC1 ~]# rpm -qa *oracleasm*oracleasmlib-2.0.12-1.el6.x86_64kmod-oracleasm-2.0.8-4.el6_6.x86_64oracleasm-support-2.1.8-1.el6.x86_64

初始化asmlib【RAC1&RAC2】

[root@rac1 oracle]# /etc/init.d/oracleasm configure -iConfiguring the Oracle ASM library driver. This will configure the on-boot properties of the Oracle ASM librarydriver. The following questions will determine whether the driver isloaded on boot and what permissions it will have. The current valueswill be shown in brackets ('[]'). Hitting <ENTER> without typing ananswer will keep that current value. Ctrl-C will abort. Default user to own the driver interface []: gridDefault group to own the driver interface []: asmadminStart Oracle ASM library driver on boot (y/n) [n]: yFix permissions of Oracle ASM disks on boot (y/n) [y]: Writing Oracle ASM library driver configuration: [ OK ]Loading module "oracleasm": [ OK ]Mounting ASMlib driver filesystem: [ OK ]Scanning system for ASM disks: [ OK ]

创建asm磁盘【RAC1】

[root@rac1 oracle]# /etc/init.d/oracleasm createdisk Vol1_OCR /dev/sdb1Marking disk "/dev/sdb1" as an ASM disk: [ OK ][root@rac1 oracle]# /etc/init.d/oracleasm createdisk Vol2_OCR /dev/sdc1Marking disk "/dev/sdc1" as an ASM disk: [ OK ][root@rac1 oracle]# /etc/init.d/oracleasm createdisk Vol3_OCR /dev/sdd1Marking disk "/dev/sdd1" as an ASM disk: [ OK ][root@rac1 oracle]# /etc/init.d/oracleasm createdisk DATA_Vol1 /dev/sde1Marking disk "/dev/sde1" as an ASM disk: [ OK ][root@rac1 oracle]# /etc/init.d/oracleasm createdisk BACKUP_Vol1 /dev/sdf1Marking disk "/dev/sdf1" as an ASM disk: [ OK ] [root@rac1 oracle]# /etc/init.d/oracleasm listdisksBACKUP_VOL1DATA_VOL1VOL1_OCRVOL2_OCRVOL3_OCR

在【RAC2】上执行扫描，识别asm磁盘

[root@rac2 ~]# /etc/init.d/oracleasm scandisksScanning system for ASM disks: [ OK ][root@rac2 ~]# /etc/init.d/oracleasm listdisksBACKUP_VOL1DATA_VOL1VOL1_OCRVOL2_OCRVOL3_OCR

15. 安装grid前检查

安装cvuqdisk软件包

[root@rac1 software]# export CVUQDISK_GRP=oinstall[root@rac1 software]# cd grid/rpm/[root@rac1 rpm]# rpm -ivh cvuqdisk-1.0.7-1.rpm Preparing... ########################################### [100%] 1:cvuqdisk ########################################### [100%]

注：在rac2上也需要执行安装

进行检查grid安装环境

[grid@RAC1 ~]$ /soft/grid/runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose

至此，安装环境准备好。。。。。。。。。。