下载软件hadoop3.1版本

linux 操作系统 centos镜像下载

linux jdk

hadoop下载

准备3台机器

IP

机器名

root密码

用户名

密码

进程

192.168.20.44

master

hd

hd

hd

namenode,ResourceManager，SecondaryNameNode

192.168.20.45

slave01

hd

hd

hd

datanode，NodeManager

192.168.20.46

slave02

hd

hd

hd

datanode，NodeManager

用户权限

brroot 超级管理员（系统文件修改）brbrhd 普通角色用户 （在/home/hd/*   进行所有增删改查）brbr密码统一设置成：hd 或者 123456

修改静态IP地址第一台机器

设置网卡为静态的地址

br切换root用户br[hd@bogon Desktop]$ su rootbrPassword: br修改网卡br[root@bogon Desktop]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 brDEVICE="eth0"brBOOTPROTO="static"     #brONBOOT="yes"brTYPE="Ethernet"brIPADDR=192.168.245.20  #[2-254]br重启网卡br[root@bogon Desktop]# service network restart br查看IP信息br[root@bogon Desktop]# ifconfig 

第二台机器

br切换root用户br[hd@bogon Desktop]$ su rootbrPassword: br修改网卡br[root@bogon Desktop]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 brDEVICE="eth0"brBOOTPROTO="static"     #brONBOOT="yes"brTYPE="Ethernet"brIPADDR=192.168.245.30br删除电脑网卡与系统绑定关系br[root@localhost Desktop]# rm -rf /etc/udev/rules.d/70-persistent-net.rules br重启系统br[root@localhost Desktop]# rebootbr查看IP信息br[root@bogon Desktop]# ifconfig 

第三台机器

br切换root用户br[hd@bogon Desktop]$ su rootbrPassword: br修改网卡br[root@bogon Desktop]# vim /etc/sysconfig/network-scripts/ifcfg-eth0 brDEVICE="eth0"brBOOTPROTO="static"     #brONBOOT="yes"brTYPE="Ethernet"brIPADDR=192.168.245.40br删除电脑网卡与系统绑定关系br[root@localhost Desktop]# rm -rf /etc/udev/rules.d/70-persistent-net.rules br重启系统br[root@localhost Desktop]# rebootbr查看IP信息br[root@bogon Desktop]# ifconfigbrbr

获取机器的IP地址

br[root@localhost ~]# ifconfig breno16777736: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500br        inet 192.168.20.46  netmask 255.255.255.0  broadcast 192.168.20.46brbr#如果没有查看IP的命令就安装步骤br[hd@localhost root]$ su rootbrPassword: br[root@localhost ~]# yum install -y net-toolsbrbr#也可查看ip addr 命令查看IP地址。 br[hd@slave02 ~]$ ip addrbr1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000br    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00br    inet 127.0.0.1/8 scope host lobr       valid_lft forever preferred_lft foreverbr    inet6 ::1/128 scope host br       valid_lft forever preferred_lft foreverbr2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000br    link/ether 00:0c:29:4f:71:df brd ff:ff:ff:ff:ff:ffbr    inet 192.168.20.46/24 brd 192.168.20.255 scope global noprefixroute dynamic ens160br       valid_lft 32521sec preferred_lft 32521secbr    inet6 fe80::f52e:651e:721b:a15a/64 scope link tentative noprefixroute dadfailed br       valid_lft forever preferred_lft foreverbr    inet6 fe80::5ebb:a198:b90f:fe68/64 scope link tentative noprefixroute dadfailed br       valid_lft forever preferred_lft foreverbr    inet6 fe80::c180:4b3e:a100:224c/64 scope link tentative noprefixroute dadfailed br       valid_lft forever preferred_lft foreverbr

Linux安装Java环境

Linux常用的命令

brsu 切换用户brbrpwd 显示当前目录brbrtar 解压，压宿brbrcd 进入目录brbrmv 移动文件，也可以改文件名brbrsource /etc/profile 加载环境变量brbrvi 文本编辑器，相当于我们的记事本一样，但是使用的都是命令的操作。

把Linux自带Java环境删除

br[hd@localhost ~]$ su root brPassword: br[root@localhost hd]# yum remove  -y  java*  

上传Java包

br[root@localhost hd]# su hdbr[hd@localhost ~]$ br[hd@localhost ~]$ br[hd@localhost ~]$ pwdbr/home/hdbr[hd@localhost ~]$ mkdir apps #上传到此目录 br[hd@localhost ~]$ cd apps/br[hd@localhost apps]$ br#上传过程br[hd@localhost apps]$ llbrtotal 178952br-rw-rw-r--. 1 hd hd 183246769 Apr 26  2018 jdk-8u121-linux-x64.tar.gz

解压java包

br#解压br[hd@localhost apps]$ tar -zxvf jdk-8u121-linux-x64.tar.gz br[hd@localhost apps]$ llbrtotal 178956brdrwxr-xr-x. 8 hd hd      4096 Dec 12  2016 jdk1.8.0_121br-rw-rw-r--. 1 hd hd 183246769 Apr 26  2018 jdk-8u121-linux-x64.tar.gzbr[hd@localhost apps]$ br#目录改名br[hd@localhost apps]$ mv jdk1.8.0_121/ java  br[hd@localhost apps]$ llbrtotal 178956brdrwxr-xr-x. 8 hd hd      4096 Dec 12  2016 javabr-rw-rw-r--. 1 hd hd 183246769 Apr 26  2018 jdk-8u121-linux-x64.tar.gz

配置java环境

br[hd@localhost apps]$ su rootbrPassword: br[root@localhost apps]# cd java/br[root@localhost java]# pwdbr/home/hd/apps/javabr[root@localhost java]# br[root@localhost java]# vi /etc/profile

使用vi编辑器，在/etc/profile增加java环境变量

brexport JAVA_HOME=/home/hd/apps/javabrexport PATH=$PATH:$JAVA_HOME/bin

重加载一下系统环境

br[root@localhost java]# source /etc/profilebr[root@localhost java]# java -versionbrjava version "1.8.0_121"brJava(TM) SE Runtime Environment (build 1.8.0_121-b13)brJava HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

配置第二台，第三台机器的Java环境

使用scp远程拷贝命令

brscp  file2  [[user@]host2:]file2

1.把每一台机的java目录，拷贝到第二台机器

br[root@localhost apps]# su hdbr[hd@localhost apps]$ br[hd@localhost apps]$  scp -r java  hd@192.168.126.129:/home/hd/apps/

2.把每一台机的profile文件，拷贝到第二台机器

br[hd@localhost apps]$ su rootbrPassword: br[root@localhost apps]# scp /etc/profile root@192.168.126.129:/etc/brThe authenticity of host '192.168.126.129 (192.168.126.129)' can't be established.brECDSA key fingerprint is fb:0a:7a:9f:9a:bc:4f:ff:66:29:1d:1d:b9:a0:35:d1.brAre you sure you want to continue connecting (yes/no)? yesbrWarning: Permanently added '192.168.126.129' (ECDSA) to the list of known hosts.brroot@192.168.126.129's password: brprofile                                                                                                        100% 1820     1.8KB/s   00:00    br[root@localhost apps]# 

3.第二台机器加载profile

br[hd@localhost apps]$ source /etc/profilebr[hd@localhost apps]$ br[hd@localhost apps]$ java -versionbrjava version "1.8.0_121"brJava(TM) SE Runtime Environment (build 1.8.0_121-b13)brJava HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)br[hd@localhost apps]$ 

第三台机，执行以上的步骤

安装hadoop之前准备1.修改主机名第一台机器master第二台机器slave01第三台机器slave02

br[hd@localhost ~]$ hostnamectl set-hostname masterbr==== AUTHENTICATING FOR org.freedesktop.hostname1.set-static-hostname ===brAuthentication is required to set the statically configured local host name, as well as the pretty host name.brAuthenticating as: rootbrPassword: br==== AUTHENTICATION COMPLETE ===

br[hd@localhost ~]$ hostnamectl set-hostname slave01

br[hd@localhost ~]$ hostnamectl set-hostname slave02

2.修改/etc/hosts 文件

br[hd@master ~]$ su rootbrPassword: br[root@master hd]# vi /etc/hostsbr127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4br::1         localhost localhost.localdomain localhost6 localhost6.localdomain6brbr192.168.126.128 masterbr192.168.126.129 slave01br192.168.126.130 slave02

同步到第二，三台机器

br#第二台机器br[root@master hd]# scp /etc/hosts root@slave01:/etc/br#第三台机器br[root@master hd]# scp /etc/hosts root@slave02:/etc/

3.关闭防火墙

br启动：systemctl start firewalld.servicebrbr查看状态： systemctl status firewalld.servicebrbr停止： systemctl stop firewalld.servicebrbr禁用： systemctl disable firewalld.service brbr重启: systemctl  restart firewalld.service

4.免密登录

需要做的免密的机器

机器---->机器（免密登录）

master ----> slave01

master ----> slave02

master ----> master

生成密钥

br[hd@master ~]$ ssh-keygen brGenerating public/private rsa key pair.brEnter file in which to save the key (/home/hd/.ssh/id_rsa): brEnter passphrase (empty for no passphrase): brEnter same passphrase again: brYour identification has been saved in /home/hd/.ssh/id_rsa.brYour public key has been saved in /home/hd/.ssh/id_rsa.pub.brThe key fingerprint is:bref:ff:98:6c:a4:66:ca:66:a0:cd:a4:da:75:9c:c0:9f hd@slave02brThe key's randomart image is:br+--[ RSA 2048]----+br|                 |br|                 |br|                 |br|     .           |br|      o S        |br|      o+ +  .    |br|     *..E .o     |br|   .o.ooo.+..o   |br|  ...  oo+.o=..  |br+-----------------+

拷贝密钥到你需要免密登录的机器

br[hd@master ~]$ ssh-copy-id slave02brThe authenticity of host 'slave02 (192.168.126.130)' can't be established.brECDSA key fingerprint is 09:57:a3:56:3b:5f:f0:01:55:0e:42:f3:4c:43:3d:d5.brAre you sure you want to continue connecting (yes/no)? yesbr/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installedbr/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keysbrhd@slave02's password: brbrNumber of key(s) added: 1brbrNow try logging into the machine, with:   "ssh 'slave02'"brand check to make sure that only the key(s) you wanted were added.

测试免密登录

br[hd@master ~]$ ssh slave01brLast login: Sun Jul 12 23:54:32 2020 from 192.168.22.130br[hd@slave01 ~]$ br[hd@slave01 ~]$ 

Hadoop安装上传hadoop安装包

br[hd@master hadoop]$ rzbrrz waiting to receive.br zmodem trl+C ȡ

解压安装包

br[hd@master apps]$ su hdbrPassword: br[hd@master apps]$ pwdbr/home/hd/appsbr[hd@master apps]$ tar -zxvf hadoop-3.0.0.tar.gz 

改目录名称

brhd@master apps]$ mv hadoop-3.0.0 hadoopbr[hd@master apps]$ llbrtotal 324644brdrwxr-xr-x. 12 hd hd       192 Jul 11 00:09 hadoop

修改hadoop配置文件修改hadoop-env.sh

br[hd@master ~]$ cd /home/hd/apps/hadoop/etc/hadoop/br[hd@master hadoop]$ br[hd@master hadoop]$ pwdbr/home/hd/apps/hadoop/etc/hadoopbr[hd@master hadoop]$ vi hadoop-env.shbrbr#在文件的尾部（按“G”可以跳到文档的尾部），增加brexport JAVA_HOME=/home/hd/apps/java

修改core-site.xml

br<configuration>br    <!-- 指定HADOOP所使用的文件系统schema（URI），HDFS的老大（NameNode）的地址 -->br    <property>br        <name>fs.defaultFS</name>br        <value>hdfs://master:9000</value>br    </property>br    <!-- 指定hadoop运行时产生文件的存储目录 -->br    <property>br        <name>hadoop.tmp.dir</name>br        <value>/home/hd/apps/hadoop/tmpdata</value>br    </property>br</configuration>

修改hdfs-site.xml

br<configuration>br    <!-- 指定HDFS副本的数量 -->br    <property>br        <name>dfs.replication</name>br        <value>2</value>br    </property>br    <!-- 设置namenode的http通讯地址 -->br    <property>br        <name>dfs.namenode.http-address</name>br        <value>master:50070</value>br    </property>brbr    <!-- 设置secondarynamenode的http通讯地址 -->br    <property>br        <name>dfs.namenode.secondary.http-address</name>br        <value>master:50090</value>br    </property>brbr    <!-- 设置namenode存放的路径 -->br    <property>br        <name>dfs.namenode.name.dir</name>br        <value>/home/hd/apps/hadoop/namenode</value>br    </property>brbr    <!-- 设置datanode存放的路径 -->br    <property>br        <name>dfs.datanode.data.dir</name>br        <value>/home/hd/apps/hadoop/datanode</value>br    </property>br</configuration>

修改mapred-site.xml

br<configuration> br    <!-- 指定mr运行在yarn上 -->br    <property>br        <name>mapreduce.framework.name</name>br        <value>yarn</value>br    </property>br    <property>br        <name>yarn.app.mapreduce.am.env</name>br        <value>HADOOP_MAPRED_HOME=/home/hd/apps/hadoop</value>br    </property>br    <property>br          <name>mapreduce.map.env</name>br          <value>HADOOP_MAPRED_HOME=/home/hd/apps/hadoop</value>br    </property>br    <property>br          <name>mapreduce.reduce.env</name>br          <value>HADOOP_MAPRED_HOME=/home/hd/apps/hadoop</value>br    </property>br</configuration>

修改yarn-site.xml

br<configuration>br    <!-- 指定YARN的老大（ResourceManager）的地址 -->br    <property>br        <name>yarn.resourcemanager.hostname</name>br        <value>master</value>br    </property>br    <!-- reducer获取数据的方式 -->br    <property>br        <name>yarn.nodemanager.aux-services</name>br        <value>mapreduce_shuffle</value>br    </property>br</configuration>

修改workers

br[hd@master hadoop]$ vi workers brslave01brslave02

修改环境变量

br[hd@master hadoop]$ su root brPassword: br[root@master hadoop]# vi /etc/profilebrbr#增加brexport HADOOP_HOME=/home/hd/apps/hadoopbr#增加brexport PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

拷贝到第二，三台机

br[root@master hadoop]# su hdbr[hd@master hadoop]$ br[hd@master hadoop]$ br[hd@master hadoop]$ scp  -r /home/hd/apps/hadoop hd@slave01:/home/hd/apps/br[hd@master hadoop]$ scp  -r /home/hd/apps/hadoop hd@slave02:/home/hd/apps/

br[hd@master hadoop]$ su rootbrPassword: br[root@master hadoop]# scp /etc/profile root@slave01:/etc/brroot@slave01's password: brprofile                                                                                                        100% 1896     1.9KB/s   00:00    br[root@master hadoop]# scp /etc/profile root@slave02:/etc/brprofile                                                                                                        100% 1896     1.9KB/s   00:00    br[root@master hadoop]# 

在第三台机器里加载环境

br[root@master hadoop]#  source /etc/profilebrbr[hd@master hadoop]$ hadoop versionbrHadoop 3.0.0

格式化

br[hd@master hadoop]$ ll /home/hd/apps/hadoop/namenodebrls: cannot access /home/hd/apps/hadoop/namenode: No such file or directorybr[hd@master hadoop]$ hadoop namenode -format 

启动hadoop

start-dfs.sh 启动HDFS分布式文件系统，停止stop-dfs.sh

start-yarn.sh 启动Yarn资源管理器，停止stop-yarn.sh

start-all.sh HDFS分布式文件系统与Yarn启动，停止stop-all.sh

jps查看进程

br[hd@master ~]$ jpsbr23668 SecondaryNameNodebr23467 NameNodebr23903 ResourceManagerbr24207 Jps

br[hd@slave01 ~]$ jpsbr22341 DataNodebr22649 Jpsbr22458 NodeManager

br[hd@slave02 ~]$ jpsbr23367 Jpsbr23176 NodeManagerbr23051 DataNode

测试

hdfs 文件系统访问地址：

Yarn资源管理器访问地址：