大数据虚拟机集群搭建(Big data virtual machine cluster construction)

大数据集群搭建

@

  • 大数据集群搭建一、虚拟机环境准备(1)系统以Centos-7.5-x86-1804为例最小化安装后,使用yum安装基本的工具,安装epel-release(2)安装net-tools:工具包集合包含很多命令(3)关闭防火墙以及防火墙开机自启(4)创建用户(5)配置flink用户具有root权限,方便后期加sudo执行root权限命令(6)在/opt目录下创建module(后面放软件)和software(后面放安装包)文件夹(7)虚拟机克隆(8)修改克隆机IP,以flink102为例(9)修改虚拟机主机名(10)修改windows系统中的host文件,路径:C:\Windows\System32\drivers\etc二、在flink102上安装JDK(这边开始用自建的用户flink开始操作)(1)这边安装jdk1.8,具体安装包:jdk-8u212-linux-x64.tar.gz(2)新建/etc/profile.d/my_env.sh文件三、Hadoop安装,先在flink02上(1)Hadoop安装,这里用的是Hadoop-3.1.3.tar.gz,将文件传输到/opt/software再进行解压(2)搭建集群,需要在flink103、flink104上安装hadoop-3,1.3,因为flink102上已经搭建完成了,所以只要分发给flink103、flink104即可(3)分发jdk和hadoop到flink103、flink104(4)ssh无密登录四、集群部署(1)集群部署规划(2)配置文件修改(3)在集群上分发配置好的Hadoop配置文件(4)群起集群(5)配置历史服务器(6)配置日志的聚集(7)集群群起/群停(8)集群查看后台进程运行情况脚本(9)集群时间同步
  • 一、虚拟机环境准备(1)系统以Centos-7.5-x86-1804为例最小化安装后,使用yum安装基本的工具,安装epel-release(2)安装net-tools:工具包集合包含很多命令(3)关闭防火墙以及防火墙开机自启(4)创建用户(5)配置flink用户具有root权限,方便后期加sudo执行root权限命令(6)在/opt目录下创建module(后面放软件)和software(后面放安装包)文件夹(7)虚拟机克隆(8)修改克隆机IP,以flink102为例(9)修改虚拟机主机名(10)修改windows系统中的host文件,路径:C:\Windows\System32\drivers\etc
  • (1)系统以Centos-7.5-x86-1804为例最小化安装后,使用yum安装基本的工具,安装epel-release
  • (2)安装net-tools:工具包集合包含很多命令
  • (3)关闭防火墙以及防火墙开机自启
  • (4)创建用户
  • (5)配置flink用户具有root权限,方便后期加sudo执行root权限命令
  • (6)在/opt目录下创建module(后面放软件)和software(后面放安装包)文件夹
  • (7)虚拟机克隆
  • (8)修改克隆机IP,以flink102为例
  • (9)修改虚拟机主机名
  • (10)修改windows系统中的host文件,路径:C:\Windows\System32\drivers\etc
  • 二、在flink102上安装JDK(这边开始用自建的用户flink开始操作)(1)这边安装jdk1.8,具体安装包:jdk-8u212-linux-x64.tar.gz(2)新建/etc/profile.d/my_env.sh文件
  • (1)这边安装jdk1.8,具体安装包:jdk-8u212-linux-x64.tar.gz
  • (2)新建/etc/profile.d/my_env.sh文件
  • 三、Hadoop安装,先在flink02上(1)Hadoop安装,这里用的是Hadoop-3.1.3.tar.gz,将文件传输到/opt/software再进行解压(2)搭建集群,需要在flink103、flink104上安装hadoop-3,1.3,因为flink102上已经搭建完成了,所以只要分发给flink103、flink104即可(3)分发jdk和hadoop到flink103、flink104(4)ssh无密登录
  • (1)Hadoop安装,这里用的是Hadoop-3.1.3.tar.gz,将文件传输到/opt/software再进行解压
  • (2)搭建集群,需要在flink103、flink104上安装hadoop-3,1.3,因为flink102上已经搭建完成了,所以只要分发给flink103、flink104即可
  • (3)分发jdk和hadoop到flink103、flink104
  • (4)ssh无密登录
  • 四、集群部署(1)集群部署规划(2)配置文件修改(3)在集群上分发配置好的Hadoop配置文件(4)群起集群(5)配置历史服务器(6)配置日志的聚集(7)集群群起/群停(8)集群查看后台进程运行情况脚本(9)集群时间同步
  • (1)集群部署规划
  • (2)配置文件修改
  • (3)在集群上分发配置好的Hadoop配置文件
  • (4)群起集群
  • (5)配置历史服务器
  • (6)配置日志的聚集
  • (7)集群群起/群停
  • (8)集群查看后台进程运行情况脚本
  • (9)集群时间同步

一、虚拟机环境准备

(1)系统以Centos-7.5-x86-1804为例最小化安装后,使用yum安装基本的工具,安装epel-release

yum install -y epel-release

注:该工具相当于是一个软件仓库

(2)安装net-tools:工具包集合包含很多命令

yum install -y net-tools

安装vim:编辑器

yum install -y vim

安装一些其他工具

yum install -y psmisc nc rsync lrzsz ntp libzstd openssl-static tree iotop git

(3)关闭防火墙以及防火墙开机自启

    systemctl stop firewalld
    systemctl disable firewalld.service

(4)创建用户

      useradd flink
      password flink

(5)配置flink用户具有root权限,方便后期加sudo执行root权限命令

vim /etc/sudoers
## Allow root to run any commands anywhere
root	  ALL=(ALL)	ALL
flink	  ALL=(ALL)	NOPASSWD:ALL

(6)在/opt目录下创建module(后面放软件)和software(后面放安装包)文件夹

mkdir /opt/module
mkdir /opt/software

将两个文件夹的权限赋予给flink用户

chown flink:flink /opt/module
chown flink:flink /opt/software

(7)虚拟机克隆

克隆出虚拟机flink102、flink103、flink104三台虚拟机

(8)修改克隆机IP,以flink102为例

vim /etc/sysconfig/network-scripts/ifcfg-ens33 # 这边会有不同,随机生成的

修改为

DEVICE=ens33
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
NAME="ens33"
IPADDR=192.168.20.202
PREFIX=24
GATEWAY=192.168.20.1
DNS1=192.168.20.1

查看windows系统中网络适配器VMware Network Adapter VMnet8的IP地址(virtual box是VirtualBox Host-Only Ethernet Adapter)
默认网关设置为192.168.20.1

(9)修改虚拟机主机名

vim /etc/hostname
flink102

配置Linux克隆机主机映射hosts文件

vim /etc/hosts
192.168.20.202 flink102
192.168.20.203 flink103
192.168.20.204 flink104
192.168.20.205 flink105 # 备用
192.168.20.206 flink106 # 备用

重启虚拟机

reboot

(10)修改windows系统中的host文件,路径:C:\Windows\System32\drivers\etc

添加如下内容:

192.168.20.202 flink102
192.168.20.203 flink103
192.168.20.204 flink104
192.168.20.205 flink105 # 备用
192.168.20.206 flink106 # 备用

二、在flink102上安装JDK(这边开始用自建的用户flink开始操作)

(1)这边安装jdk1.8,具体安装包:jdk-8u212-linux-x64.tar.gz

将安装包传输到/opt/software,然后解压

tar -zxvf  jdk-8u212-linux-x64.tar.gz -C /opt/module

(2)新建/etc/profile.d/my_env.sh文件

sudo vim /etc/profile.d/my_env.sh

添加如下内容

#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_212
export PATH=$PATH:$JAVA_HOME/bin
source /etc/proflie # source一下,让新环境变量Path生效

测试jdk1.8是否生效

java -version

三、Hadoop安装,先在flink02上

(1)Hadoop安装,这里用的是Hadoop-3.1.3.tar.gz,将文件传输到/opt/software再进行解压

 tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module

将Hadoop添加到环境变量中

vim /etc/profile.d/my_env.sh

添加如下设置:

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

记得source一下,使环境变量生效
测试Hadoop是否安装成功

hadoop version

如果没生效可以重启虚拟机试试reboot

(2)搭建集群,需要在flink103、flink104上安装hadoop-3,1.3,因为flink102上已经搭建完成了,所以只要分发给flink103、flink104即可

xsync脚本编写

cd /home/flink
mkdir bin
cd bin
vim xsync

具体脚本如下:

#!/bin/bash

#1.判断参数个数
if [ $# -lt 1 ]
then
	echo Not Enough Arguement!
	exit;
fi

#2.遍历集群所有机器
for host in flink102 flink103 flink104
do
	echo ===============$host===============
	#3.遍历所有目录,发送文件
	for file in $@
	do
		#4.判断文件是否存在
		if [ -e $file ]
			then
				#5.获取父目录
				pdir=$(cd -P $(dirname $file); pwd)
				#6.获取当前文件的名称
				fname=$(basename $file)
				ssh $host "mkdir -p $pdir"
				rsync -av $pdir/$fname $host:$pdir
			else
				echo $file dose not exists!
		fi
	done
done

将该脚本赋予执行权限

chmod 777 xsync

(3)分发jdk和hadoop到flink103、flink104

cd /opt/module
xsync jdk1.8.0_212
xsync hadoop-3.1.3
sudo xsync /etc/profile.d/my_env.sh

记得在flink103、flink104上source一下
注意:这边分发的时候比较麻烦是需要输好多遍密码的,下面设置一下公共钥匙就可以无密码登录了

(4)ssh无密登录

先在102上生成公钥和私钥

cd /home/flink/.ssh
ssh-keygen -t rsa

然后敲三个回车,就会生成公钥和私钥
将公钥拷贝到要免密登录的机器

ssh-copy-id flink102
ssh-copy-id flink103
ssh-copy-id flink104

同样的操作需要在103和104上实施一遍

四、集群部署

(1)集群部署规划

102上部署HDFS的NameNode和DataNode以及YARN的NodeManager
103上部署HDFS的DataNode以及YARN的ResourceManager和NodeManager
104上部署SecondaryNamNode和DataNode以及YARN的NodeManager

(2)配置文件修改

配置core-site.xml

cd $HADOOP_HOME/etc/hadoop
vim core-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
   <!-- 指定NameNode的地址 -->
   <property>
       <name>fs.defaultFS</name>
       <value>hdfs://flink102:8020</value>
   </property>
 
   <!-- 指定hadoop数据的存储目录 -->
   <property>
       <name>hadoop.tmp.dir</name>
       <value>/opt/module/hadoop-3.1.3/data</value>
   </property>
 
   <!-- 配置HDFS网页登录使用的静态用户为flink -->
   <property>
       <name>hadoop.http.staticuser.user</name>
       <value>flink</value>
   </property>
</configuration>

HDFS配置文件
配置hdfs-site.xml

vim hdfs-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
<!-- nn web端访问地址-->
<property>
       <name>dfs.namenode.http-address</name>
       <value>flink102:9870</value>
   </property>
<!-- 2nn web端访问地址-->
   <property>
       <name>dfs.namenode.secondary.http-address</name>
       <value>flink104:9868</value>
   </property>
</configuration>

YARN配置文件
配置yarn-site.xml

vim yarn-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
   <!-- 指定MR走shuffle -->
   <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
   </property>
 
   <!-- 指定ResourceManager的地址-->
   <property>
       <name>yarn.resourcemanager.hostname</name>
       <value>flink103</value>
   </property>
 
   <!-- 环境变量的继承 -->
   <property>
       <name>yarn.nodemanager.env-whitelist</name>
       <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
   </property>
</configuration>

MapReduce配置文件
配置mapred-site.xml

vim mapred-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
<!-- 指定MapReduce程序运行在Yarn上 -->
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

(3)在集群上分发配置好的Hadoop配置文件

xsync /opt/module/hadoop-3.1.3/etc/hadoop/

去103和104上查看文件分发情况

cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml
cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml

(4)群起集群

配置workers

vim /opt/module/hadoop-3.1.3/etc/hadoop/workers
在该文件中增加如下内容:
flink102
flink103
flink104

启动集群
1,如果集群是第一次启动,需要在102节点格式化NameNode(注意:格式化NameNode,会产生新的集群id,导致NameNode和DataNode的集群id不一致,集群找不到已往数据。如果集群在运行过程中报错,需要重新格式化NameNode的话,一定要先停止namenode和datanode进程,并且要删除所有机器的data和logs目录,然后再进行格式化。)

hdfs namenode -format

2,启动HDFS

sbin/start-dfs.sh

3,在配置了ResourceManager的节点(103)启动YARN

sbin/start-yarn.sh

4,Web端查看HDFS的NameNode
浏览器中输入:http://flink102:9870
查看HDFS上存储的数据信息
5,Web端查看YARN的ResourceManager
浏览器中输入:http://flink103:8088
查看YARN上运行的Job信息

(5)配置历史服务器

为了查看程序的历史运行情况,需要配置一下历史服务器。具体配置步骤如下:
1)配置mapred-site.xml

vim mapred-site.xml
在该文件里面增加如下配置。
<!-- 历史服务器端地址 -->
<property>
   <name>mapreduce.jobhistory.address</name>
   <value>flink102:10020</value>
</property>
 
<!-- 历史服务器web端地址 -->
<property>
   <name>mapreduce.jobhistory.webapp.address</name>
   <value>flink102:19888</value>
</property>

2)分发配置

xsync $HADOOP_HOME/etc/hadoop/mapred-site.xml

3)在flink102启动历史服务器

mapred --daemon start historyserver

4)查看历史服务器是否启动

jps

5)查看JobHistory
http://flink102:19888/jobhistory

(6)配置日志的聚集

日志聚集概念:应用运行完成以后,将程序运行日志信息上传到HDFS系统上。

日志聚集功能好处:可以方便的查看到程序运行详情,方便开发调试。
注意:开启日志聚集功能,需要重新启动NodeManager 、ResourceManager和HistoryServer。
开启日志聚集功能具体步骤如下:
1)配置yarn-site.xml

vim yarn-site.xml
在该文件里面增加如下配置。
<!-- 开启日志聚集功能 -->
<property>
   <name>yarn.log-aggregation-enable</name>
   <value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>  
   <name>yarn.log.server.url</name>  
   <value>http://flink102:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
   <name>yarn.log-aggregation.retain-seconds</name>
   <value>604800</value>
</property>

2)分发配置

xsync $HADOOP_HOME/etc/hadoop/yarn-site.xml

3)关闭NodeManager 、ResourceManager和HistoryServer(103节点)

sbin/stop-yarn.sh
mapred --daemon stop historyserver

4)启动NodeManager 、ResourceManage和HistoryServer

start-yarn.sh
mapred --daemon start historyserver

(7)集群群起/群停

需要在写一个脚本来群起集群

cd /home/flink/bin
vim myhadoop.sh

脚本如下:

#!/bin/bash

if [ $# -lt 1 ]
then
   echo "No Args Input..."
   exit ;
fi
 
case $1 in
"start")
       echo " =================== 启动 hadoop集群 ==================="
 
       echo " --------------- 启动 hdfs ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"
       echo " --------------- 启动 yarn ---------------"
       ssh flink103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"
       echo " --------------- 启动 historyserver ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver"
;;
"stop")
       echo " =================== 关闭 hadoop集群 ==================="
 
       echo " --------------- 关闭 historyserver ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
       echo " --------------- 关闭 yarn ---------------"
       ssh flink103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"
       echo " --------------- 关闭 hdfs ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;;
*)
   echo "Input Args Error..."
;;
esac

脚本完成后别忘了给脚本加上执行权限

chmod 777 myhadoop.sh

(8)集群查看后台进程运行情况脚本

上文有提到jps查看后台进程运行情况,但是不足的是我们的集群查看后台进程需要一台一台去看服务是否起来了,如果集群数量增加会很麻烦,所以需要编写一个集群后台进程查看脚本

cd /home/flink/bin
vim jpsall

脚本如下:

#!/bin/bash

for host in flink102 flink103 flink104
do
	echo " ============ $host ============"
	ssh $host jps
done

赋予其执行权限

chmod 777 jpsall

(9)集群时间同步

需要一台集群中的机器作为时间同步的标准,避免集群中各台机器时间不同步发生数据上的问题
修改102节点上的ntp配置文件

vim /etc/ntp.conf

修改内容如下
修改1(授权192.168.10.0-192.168.10.255网段上的所有机器可以从这台机器上查询和同步时间)

#restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap
为restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap

修改2(集群在局域网中,不使用其他互联网上的时间)

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
为
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

添加3(当该节点丢失网络连接,依然可以采用本地时间作为时间服务器为集群中的其他节点提供时间同步)

server 127.127.1.0
fudge 127.127.1.0 stratum 10

修改flink102的/etc/sysconfig/ntpd 文件

sudo vim /etc/sysconfig/ntpd

增加内容如下(让硬件时间与系统时间一起同步)

SYNC_HWCLOCK=yes

重新启动ntpd服务

sudo systemctl start ntpd

设置ntpd服务开机启动

sudo systemctl enable ntpd

其他机器配置(必须root用户)
(1)关闭所有节点上ntp服务和自启动

[flink@flink103 ~]$ sudo systemctl stop ntpd
[flink@flink103 ~]$ sudo systemctl disable ntpd
[flink@flink104 ~]$ sudo systemctl stop ntpd
[flink@flink104 ~]$ sudo systemctl disable ntpd

(2)在其他机器配置1分钟与时间服务器同步一次

[flink@flink103 ~]$ sudo crontab -e

编写定时任务如下:

*/1 * * * * /usr/sbin/ntpdate flink102

(3)修改任意机器时间

[flink@flink103 ~]$ sudo date -s "2021-9-11 11:11:11"

(4)1分钟后查看机器是否与时间服务器同步

[flink@flink103 ~]$ sudo date
————————

Big data cluster construction

@

  • Big data cluster setup 1. Virtual machine environment preparation (1) system take centos-7.5-x86-1804 as an example to minimize the installation. After installation, use Yum to install the basic tools and install EPEL release (2) install net tools: the toolkit collection contains many commands (3) turn off the firewall and start the firewall automatically (4) create a user (5) configure the Flink user to have root authority, It is convenient to add sudo later and execute the root permission command (6) create the module (software) and software (installation package) folders in the / opt directory (7) clone the virtual machine (8) modify the IP of the clone machine, take flink102 as an example (9) modify the host name of the virtual machine (10) modify the host file in the windows system, path: C: \ windows \ system32 \ drivers \ etc Install JDK on flink102 (start to operate with self built user Flink) (1) install jdk1 here 8. Specific installation package: jdk-8u212-linux-x64 tar. GZ (2) create a new / etc / profile d/my_ env. SH file 3. For Hadoop installation, first (1) install Hadoop on flink02. Here, hadoop-3.1.3 is used tar. GZ, transfer the files to / opt / software and decompress them. (2) to build the cluster, you need to install hadoop-3,1.3 on flink103 and flink104. Because the construction on flink102 has been completed, you can only distribute them to flink103 and flink104. (3) distribute JDK and Hadoop to flink103 and flink104. (4) SSH non secret login IV Cluster deployment (1) cluster deployment planning (2) modification of configuration file (3) distribution of configured Hadoop configuration file on the cluster (4) cluster rise (5) configuration history server (6) aggregation of configuration logs (7) cluster rise / stop (8) cluster viewing background process running script (9) cluster time synchronization
  • 1、 Virtual machine environment preparation (1) the system takes centos-7.5-x86-1804 as an example to minimize the installation. After the installation, use Yum to install the basic tools and install EPEL release (2) install net tools: the toolkit collection contains many commands (3) turn off the firewall and start the firewall automatically (4) create a user (5) configure the Flink user to have root permission, It is convenient to add sudo later and execute the root permission command (6) create the module (software) and software (installation package) folders in the / opt directory (7) clone the virtual machine (8) modify the IP of the clone machine, take flink102 as an example (9) modify the host name of the virtual machine (10) modify the host file in the windows system, path: C: \ windows \ system32 \ drivers \ etc
  • (1) Taking centos-7.5-x86-1804 as an example, after minimizing the installation, use Yum to install the basic tools and install EPEL release
  • (2) Installing net tools: the toolkit collection contains many commands
  • (3) Turn off the firewall and start the firewall automatically
  • (4) Create user
  • (5) Configure the Flink user to have root permission, which is convenient for sudo to execute the root permission command later
  • (6) Create the module (software) and software (installation package) folders in the / opt directory
  • (7) Virtual machine cloning
  • (8) Modify the clone machine IP, taking flink102 as an example
  • (9) Modify virtual machine hostname
  • (10)修改windows系统中的host文件,路径:C:\Windows\System32\drivers\etc
  • 2、 Install JDK on flink102 (start to operate with self built user Flink) (1) install jdk1 here 8. Specific installation package: jdk-8u212-linux-x64 tar. GZ (2) create a new / etc / profile d/my_ env. SH file
  • (1) Install jdk1 here 8. Specific installation package: jdk-8u212-linux-x64 tar. gz
  • (2)新建/etc/profile.d/my_env.sh文件
  • 三、Hadoop安装,先在flink02上(1)Hadoop安装,这里用的是Hadoop-3.1.3.tar.gz,将文件传输到/opt/software再进行解压(2)搭建集群,需要在flink103、flink104上安装hadoop-3,1.3,因为flink102上已经搭建完成了,所以只要分发给flink103、flink104即可(3)分发jdk和hadoop到flink103、flink104(4)ssh无密登录
  • (1) Hadoop installation, here is hadoop-3.1.3 tar. GZ, transfer the file to / opt / software and unzip it
  • (2) To build a cluster, you need to install hadoop-3,1.3 on flink103 and flink104. Because the cluster has been built on flink102, you can only distribute it to flink103 and flink104
  • (3)分发jdk和hadoop到flink103、flink104
  • (4) SSH non secret login
  • 4、 Cluster deployment (1) cluster deployment planning (2) modification of configuration file (3) distribution of configured Hadoop configuration file on the cluster (4) cluster rise (5) configuration history server (6) aggregation of configuration logs (7) cluster rise / stop (8) cluster viewing background process running script (9) cluster time synchronization
  • (1) Cluster deployment planning
  • (2) Profile modification
  • (3) Distribute the configured Hadoop configuration file on the cluster
  • (4) Group together
  • (5) Configure history server
  • (6) Configure log aggregation
  • (7) Cluster start / stop
  • (8) Script for viewing background process running status of cluster
  • (9) Cluster time synchronization

1、 Virtual machine environment preparation

(1) Taking centos-7.5-x86-1804 as an example, after minimizing the installation, use Yum to install the basic tools and install EPEL release

yum install -y epel-release

Note: This tool is equivalent to a software warehouse

(2) Installing net tools: the toolkit collection contains many commands

yum install -y net-tools

Installing VIM: Editor

yum install -y vim

Install some other tools

yum install -y psmisc nc rsync lrzsz ntp libzstd openssl-static tree iotop git

(3) Turn off the firewall and start the firewall automatically

    systemctl stop firewalld
    systemctl disable firewalld.service

(4) Create user

      useradd flink
      password flink

(5) Configure the Flink user to have root permission, which is convenient for sudo to execute the root permission command later

vim /etc/sudoers
## Allow root to run any commands anywhere
root	  ALL=(ALL)	ALL
flink	  ALL=(ALL)	NOPASSWD:ALL

(6) Create the module (software) and software (installation package) folders in the / opt directory

mkdir /opt/module
mkdir /opt/software

Give the permissions of the two folders to the flick user

chown flink:flink /opt/module
chown flink:flink /opt/software

(7) Virtual machine cloning

Three virtual machines, flink102, flink103 and flink104, were cloned

(8) Modify the clone machine IP, taking flink102 as an example

vim /etc/sysconfig/network-scripts/ifcfg-ens33 # 这边会有不同,随机生成的

Change to

DEVICE=ens33
TYPE=Ethernet
ONBOOT=yes
BOOTPROTO=static
NAME="ens33"
IPADDR=192.168.20.202
PREFIX=24
GATEWAY=192.168.20.1
DNS1=192.168.20.1

查看windows系统中网络适配器VMware Network Adapter VMnet8的IP地址(virtual box是VirtualBox Host-Only Ethernet Adapter)
默认网关设置为192.168.20.1

(9) Modify virtual machine hostname

vim /etc/hostname
flink102

Configuring the Linux clone host mapping hosts file

vim /etc/hosts
192.168.20.202 flink102
192.168.20.203 flink103
192.168.20.204 flink104
192.168.20.205 flink105 # 备用
192.168.20.206 flink106 # 备用

Restart the virtual machine

reboot

(10)修改windows系统中的host文件,路径:C:\Windows\System32\drivers\etc

Add the following:

192.168.20.202 flink102
192.168.20.203 flink103
192.168.20.204 flink104
192.168.20.205 flink105 # 备用
192.168.20.206 flink106 # 备用

2、 Install JDK on flink102 (start operation here with self built user Flink)

(1) Install jdk1 here 8. Specific installation package: jdk-8u212-linux-x64 tar. gz

Transfer the installation package to / opt / software and unzip it

tar -zxvf  jdk-8u212-linux-x64.tar.gz -C /opt/module

(2)新建/etc/profile.d/my_env.sh文件

sudo vim /etc/profile.d/my_env.sh

Add the following

#JAVA_HOME
export JAVA_HOME=/opt/module/jdk1.8.0_212
export PATH=$PATH:$JAVA_HOME/bin
source /etc/proflie # source一下,让新环境变量Path生效

Test jdk1 8 effective or not

java -version

三、Hadoop安装,先在flink02上

(1) Hadoop installation, here is hadoop-3.1.3 tar. GZ, transfer the file to / opt / software and unzip it

 tar -zxvf hadoop-3.1.3.tar.gz -C /opt/module

Add Hadoop to the environment variable

vim /etc/profile.d/my_env.sh

Add the following settings:

#HADOOP_HOME
export HADOOP_HOME=/opt/module/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

Remember to use source to make the environment variable effective
Test whether Hadoop is installed successfully

hadoop version

If it doesn’t work, you can restart the virtual machine and try reboot

(2) To build a cluster, you need to install hadoop-3,1.3 on flink103 and flink104. Because the cluster has been built on flink102, you can only distribute it to flink103 and flink104

XSync scripting

cd /home/flink
mkdir bin
cd bin
vim xsync

The specific script is as follows:

#!/bin/bash

#1.判断参数个数
if [ $# -lt 1 ]
then
	echo Not Enough Arguement!
	exit;
fi

#2.遍历集群所有机器
for host in flink102 flink103 flink104
do
	echo ===============$host===============
	#3.遍历所有目录,发送文件
	for file in $@
	do
		#4.判断文件是否存在
		if [ -e $file ]
			then
				#5.获取父目录
				pdir=$(cd -P $(dirname $file); pwd)
				#6.获取当前文件的名称
				fname=$(basename $file)
				ssh $host "mkdir -p $pdir"
				rsync -av $pdir/$fname $host:$pdir
			else
				echo $file dose not exists!
		fi
	done
done

Give the script execution permission

chmod 777 xsync

(3)分发jdk和hadoop到flink103、flink104

cd /opt/module
xsync jdk1.8.0_212
xsync hadoop-3.1.3
sudo xsync /etc/profile.d/my_env.sh

Remember to source on flink103 and flink104
Note: it’s troublesome to distribute here. You need to enter the password many times. Set the public key below to log in without password

(4) SSH non secret login

First, the public key and private key are generated on 102

cd /home/flink/.ssh
ssh-keygen -t rsa

Then press three returns to generate the public key and private key
Copy the public key to the machine where you want to log in without secret

ssh-copy-id flink102
ssh-copy-id flink103
ssh-copy-id flink104

The same operation needs to be performed on 103 and 104

4、 Cluster deployment

(1) Cluster deployment planning

102上部署HDFS的NameNode和DataNode以及YARN的NodeManager
103上部署HDFS的DataNode以及YARN的ResourceManager和NodeManager
104上部署SecondaryNamNode和DataNode以及YARN的NodeManager

(2) Profile modification

配置core-site.xml

cd $HADOOP_HOME/etc/hadoop
vim core-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
   <!-- 指定NameNode的地址 -->
   <property>
       <name>fs.defaultFS</name>
       <value>hdfs://flink102:8020</value>
   </property>
 
   <!-- 指定hadoop数据的存储目录 -->
   <property>
       <name>hadoop.tmp.dir</name>
       <value>/opt/module/hadoop-3.1.3/data</value>
   </property>
 
   <!-- 配置HDFS网页登录使用的静态用户为flink -->
   <property>
       <name>hadoop.http.staticuser.user</name>
       <value>flink</value>
   </property>
</configuration>

HDFS profile
Configure HDFS site xml

vim hdfs-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
<!-- nn web端访问地址-->
<property>
       <name>dfs.namenode.http-address</name>
       <value>flink102:9870</value>
   </property>
<!-- 2nn web端访问地址-->
   <property>
       <name>dfs.namenode.secondary.http-address</name>
       <value>flink104:9868</value>
   </property>
</configuration>

Yarn profile
Configure yarn site xml

vim yarn-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
   <!-- 指定MR走shuffle -->
   <property>
       <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce_shuffle</value>
   </property>
 
   <!-- 指定ResourceManager的地址-->
   <property>
       <name>yarn.resourcemanager.hostname</name>
       <value>flink103</value>
   </property>
 
   <!-- 环境变量的继承 -->
   <property>
       <name>yarn.nodemanager.env-whitelist</name>
       <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
   </property>
</configuration>

MapReduce配置文件
配置mapred-site.xml

vim mapred-site.xml
文件内容如下:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
 
<configuration>
<!-- 指定MapReduce程序运行在Yarn上 -->
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>
</configuration>

(3) Distribute the configured Hadoop configuration file on the cluster

xsync /opt/module/hadoop-3.1.3/etc/hadoop/

Go to 103 and 104 to check the distribution of documents

cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml
cat /opt/module/hadoop-3.1.3/etc/hadoop/core-site.xml

(4) Group together

配置workers

vim /opt/module/hadoop-3.1.3/etc/hadoop/workers
在该文件中增加如下内容:
flink102
flink103
flink104

Start cluster
1. If the cluster is started for the first time, The namenode needs to be formatted at node 102 (Note: formatting namenode will generate a new cluster ID, resulting in inconsistent cluster IDS between namenode and datanode, and the cluster cannot find past data. If the cluster reports an error during operation and needs to reformat namenode, be sure to stop the namenode and datanode processes, delete the data and logs directories of all machines, and then format.)

hdfs namenode -format

2. Start HDFS

sbin/start-dfs.sh

3. Start yarn on the node (103) where the ResourceManager is configured

sbin/start-yarn.sh

4. View the namenode of HDFS on the web side
Enter in the browser: http://flink102:9870
View data information stored on HDFS
5. View the resource manager of yarn on the web side
Enter in the browser: http://flink103:8088
View job information running on yarn

(5) Configure history server

In order to view the historical operation of the program, you need to configure the history server. The specific configuration steps are as follows:
1) Configure mapred site xml

vim mapred-site.xml
在该文件里面增加如下配置。
<!-- 历史服务器端地址 -->
<property>
   <name>mapreduce.jobhistory.address</name>
   <value>flink102:10020</value>
</property>
 
<!-- 历史服务器web端地址 -->
<property>
   <name>mapreduce.jobhistory.webapp.address</name>
   <value>flink102:19888</value>
</property>

2) Distribution configuration

xsync $HADOOP_HOME/etc/hadoop/mapred-site.xml

3) Start the history server in flink102

mapred --daemon start historyserver

4) Check whether the history server is started

jps

5)查看JobHistory
http://flink102:19888/jobhistory

(6) Configure log aggregation

Log aggregation concept: after the application runs, upload the program running log information to the HDFS system.

Benefits of log aggregation function: you can easily view the details of program operation, which is convenient for development and debugging.
Note: to enable the log aggregation function, you need to restart nodemanager, ResourceManager and historyserver.
The specific steps to enable log aggregation are as follows:
1) Configure yarn site xml

vim yarn-site.xml
在该文件里面增加如下配置。
<!-- 开启日志聚集功能 -->
<property>
   <name>yarn.log-aggregation-enable</name>
   <value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>  
   <name>yarn.log.server.url</name>  
   <value>http://flink102:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
   <name>yarn.log-aggregation.retain-seconds</name>
   <value>604800</value>
</property>

2) Distribution configuration

xsync $HADOOP_HOME/etc/hadoop/yarn-site.xml

3)关闭NodeManager 、ResourceManager和HistoryServer(103节点)

sbin/stop-yarn.sh
mapred --daemon stop historyserver

4)启动NodeManager 、ResourceManage和HistoryServer

start-yarn.sh
mapred --daemon start historyserver

(7) Cluster start / stop

You need to write a script to cluster

cd /home/flink/bin
vim myhadoop.sh

The script is as follows:

#!/bin/bash

if [ $# -lt 1 ]
then
   echo "No Args Input..."
   exit ;
fi
 
case $1 in
"start")
       echo " =================== 启动 hadoop集群 ==================="
 
       echo " --------------- 启动 hdfs ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/sbin/start-dfs.sh"
       echo " --------------- 启动 yarn ---------------"
       ssh flink103 "/opt/module/hadoop-3.1.3/sbin/start-yarn.sh"
       echo " --------------- 启动 historyserver ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon start historyserver"
;;
"stop")
       echo " =================== 关闭 hadoop集群 ==================="
 
       echo " --------------- 关闭 historyserver ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/bin/mapred --daemon stop historyserver"
       echo " --------------- 关闭 yarn ---------------"
       ssh flink103 "/opt/module/hadoop-3.1.3/sbin/stop-yarn.sh"
       echo " --------------- 关闭 hdfs ---------------"
       ssh flink102 "/opt/module/hadoop-3.1.3/sbin/stop-dfs.sh"
;;
*)
   echo "Input Args Error..."
;;
esac

After the script is completed, don’t forget to add execution permission to the script

chmod 777 myhadoop.sh

(8) Script for viewing background process running status of cluster

As mentioned above, JPS is used to check the operation of background processes, but the disadvantage is that our cluster needs to check whether the service is up one by one. If the number of clusters increases, it will be very troublesome, so we need to write a cluster background process check script

cd /home/flink/bin
vim jpsall

The script is as follows:

#!/bin/bash

for host in flink102 flink103 flink104
do
	echo " ============ $host ============"
	ssh $host jps
done

Give it execution permission

chmod 777 jpsall

(9) Cluster time synchronization

A machine in the cluster is required as the time synchronization standard to avoid data problems caused by time synchronization of each machine in the cluster
Modify NTP profile on node 102

vim /etc/ntp.conf

The amendments are as follows:
Modification 1 (all machines in the 192.168.10.0-192.168.10.255 network segment are authorized to query and synchronize time from this machine)

#restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap
为restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap

Modification 2 (time when the cluster is in the LAN and does not use other Internet)

server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
为
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

Add 3 (when the node loses network connection, the local time can still be used as the time server to provide time synchronization for other nodes in the cluster)

server 127.127.1.0
fudge 127.127.1.0 stratum 10

修改flink102的/etc/sysconfig/ntpd 文件

sudo vim /etc/sysconfig/ntpd

Add the following contents (synchronize the hardware time with the system time)

SYNC_HWCLOCK=yes

Restart the ntpd service

sudo systemctl start ntpd

Set ntpd service startup

sudo systemctl enable ntpd

Other machine configurations (must be root)
(1) Turn off NTP service and self startup on all nodes

[flink@flink103 ~]$ sudo systemctl stop ntpd
[flink@flink103 ~]$ sudo systemctl disable ntpd
[flink@flink104 ~]$ sudo systemctl stop ntpd
[flink@flink104 ~]$ sudo systemctl disable ntpd

(2) Configure other machines to synchronize with the time server once a minute

[flink@flink103 ~]$ sudo crontab -e

The scheduled tasks are as follows:

*/1 * * * * /usr/sbin/ntpdate flink102

(3) Modify any machine time

[flink@flink103 ~]$ sudo date -s "2021-9-11 11:11:11"

(4) Check whether the machine is synchronized with the time server after 1 minute

[flink@flink103 ~]$ sudo date