大数据调度组件之一DolphinScheduler的安装

极目馆主 09-05 631

前言：

如今各位老铁们对“centosredmine”可能比较讲究，同学们都需要分析一些“centosredmine”的相关内容。那么小编也在网络上网罗了一些有关“centosredmine””的相关内容，希望看官们能喜欢，我们一起来了解一下吧！

1.1安装环境

1. hadoop2.9.2java

2. CentOS7.5node

3. JDK1.8python

4. Mysql5.7mysql

5. Zookeeper-3.4.14linux

1.2安装说明

1. 这里直接使用root用户进行安装,若是是使用普通用户安装,须要开启sudo免密和集群各节点之间实现ssh免密.web

2. 配置/etc/hosts文件,作好主机名与ip之间的映射.面试

3. 安装ds的各台机器之间须要作ssh免密登录(安装ds的用户之间的免密)spring

4. ds的安装节点与hdfs和yarn所在节点没有关系,只须要保证ds的worker所在节点有hadoop的安装目录并配置环境变量便可,由于要做为客户端提交命令.sql

5. 使用阿里云与本地虚拟机安装时配置有一点点区别,后面安装时配置有备注.shell

1.3节点规划

主机名称

master-server

worker-server

api-server

logger-server

alertserver

linux121

√

linux122

√

linux123

√

1.4具体安装步骤

1. 下载最新版的二进制安装包

wget

2. 解压,重命名

[root@linux122 software]# tar -zvxf apache-dolphinscheduler-incubating-1.3.2-dolphinscheduler-bin.tar.gz

[root@linux122 software]# mv apache-dolphinscheduler-incubating-1.3.2-dolphinscheduler-bin dolphinscheduler-bin

3. 将MySQL的驱动包放到dolphinscheduler-bin/lib目录下(使用超连接的方式)

[root@linux122 software]# ln -s /opt/servers/hive-2.3.7/lib/mysql-connector-java-5.1.46.jar /opt/software/dolphinscheduler-bin/lib/mysql-connector-java.jar

4. 在MySQL中建立dolphinscheduler数据库

mysql> create database dolphinscheduler;

5. 修改dolphinscheduler-bin/conf/datasource.properties配置文件,将数据库改成mysql(默认是postgresql)

## Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##      Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.#        # postgresqlspring.datasource.driver-class-name=com.mysql.jdbc.Driverspring.datasource.url=jdbc:mysql://bigdata:3306/dolphinscheduler?characterEncoding=UTF-8&allowMultiQueries=truespring.datasource.username=rootspring.datasource.password=hry110@qq.com# connection configuration#spring.datasource.initialSize=5# min connection number#spring.datasource.minIdle=5# max connection number#spring.datasource.maxActive=50# max wait time for get a connection in milliseconds. if configuring maxWait, fair locks are enabled by default and concurrency efficiency decreases.# If necessary, unfair locks can be used by configuring the useUnfairLock attribute to true.#spring.datasource.maxWait=60000# milliseconds for check to close free connections#spring.datasource.timeBetweenEvictionRunsMillis=60000# the Destroy thread detects the connection interval and closes the physical connection in milliseconds if the connection idle time is greater than or equal to minEvictableIdleTimeMillis.#spring.datasource.timeBetweenConnectErrorMillis=60000# the longest time a connection remains idle without being evicted, in milliseconds#spring.datasource.minEvictableIdleTimeMillis=300000#the SQL used to check whether the connection is valid requires a query statement. If validation Query is null, testOnBorrow, testOnReturn, and testWhileIdle will not work.#spring.datasource.validationQuery=SELECT 1#check whether the connection is valid for timeout, in seconds#spring.datasource.validationQueryTimeout=3# when applying for a connection, if it is detected that the connection is idle longer than time Between Eviction Runs Millis,# validation Query is performed to check whether the connection is valid#spring.datasource.testWhileIdle=true#execute validation to check if the connection is valid when applying for a connection#spring.datasource.testOnBorrow=true#execute validation to check if the connection is valid when the connection is returned#spring.datasource.testOnReturn=false#spring.datasource.defaultAutoCommit=true#spring.datasource.keepAlive=true# open PSCache, specify count PSCache for every connection#spring.datasource.poolPreparedStatements=true#spring.datasource.maxPoolPreparedStatementPerConnectionSize=20

6. 执行ds自带的脚本

sh script/create-dolphinscheduler.sh

7. 修改ds运行时的参数(即conf/env/dolphinscheduler_env.sh文件)

[root@bigdata env]# cat dolphinscheduler_env.sh ## Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##      Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.#export HADOOP_HOME=/opt/hadoop-3.1.3/export HADOOP_CONF_DIR=/opt/hadoop-3.1.3/etc/hadoop#export SPARK_HOME1=/opt/spark-yarn#export SPARK_HOME2=/opt/soft/spark2#export PYTHON_HOME=/opt/soft/pythonexport JAVA_HOME=/opt/jdk1.8.0_212export HIVE_HOME=/opt/hive-3.1.2export FLINK_HOME=/opt/flink-1.13.2#export DATAX_HOME=/opt/soft/datax/bin/datax.pyexport PATH=$HADOOP_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$PATH

8. 在准备安装ds的全部节点上执行下面的命令(将java命令超连接到本身安装的jdk下)

[root@linux122 dolphinscheduler-bin]# rm -rf /usr/bin/java[root@linux121 dolphinscheduler-1.3.2]# ln -s /opt/servers/jdk1.8.0_231/bin/java /usr/bin/java[root@linux122 dolphinscheduler-1.3.2]# ln -s /opt/servers/jdk1.8.0_231/bin/java /usr/bin/java[root@linux123 dolphinscheduler-1.3.2]# ln -s /opt/servers/jdk1.8.0_231/bin/java /usr/bin/java

说明:若是/usr/bin/java已经存在,先删除掉,从新创建超连接.

9. 修改一键部署的相关配置文件(具体内容根据本身的实际规划进行修改)

[root@bigdata conf]# cat datasource.properties ## Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##      Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.#        # postgresqlspring.datasource.driver-class-name=com.mysql.jdbc.Driverspring.datasource.url=jdbc:mysql://bigdata:3306/dolphinscheduler?characterEncoding=UTF-8&allowMultiQueries=truespring.datasource.username=rootspring.datasource.password=hry110@qq.com# connection configuration#spring.datasource.initialSize=5# min connection number#spring.datasource.minIdle=5[root@bigdata env]# vi /opt/dolphinscheduler/conf/config/install_config.conf [1]+  已停止               vi /opt/dolphinscheduler/conf/config/install_config.conf[root@bigdata env]# cat /opt/dolphinscheduler/conf/config/install_config.conf ## Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements.  See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License.  You may obtain a copy of the License at##      Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.## NOTICE :  If the following config has special characters in the variable `.*[]^${}\+?|()@#&`, Please escape, for example, `[` escape to `\[`# postgresql or mysqldbtype="mysql"# db config# db address and portdbhost="bigdata:3306"# db usernameusername="root"# database namedbname="dolphinscheduler"# db passwprd# NOTICE: if there are special characters, please use the \ to escape, for example, `[` escape to `\[`password="hry110@qq.com"# zk clusterzkQuorum="bigdata:2181"# Note: the target installation path for dolphinscheduler, please not config as the same as the current path (pwd)installPath="/opt/dolphinscheduler-bin"# deployment user# Note: the deployment user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled, the root directory needs to be created by itselfdeployUser="root"# alert config# mail server hostmailServerHost="smtp.163.com"# mail server port# note: Different protocols and encryption methods correspond to different ports, when SSL/TLS is enabled, make sure the port is correct.mailServerPort="465"# sendermailSender="hry6668888@163.com"# usermailUser="hry6668888@163.com"# sender password# note: The mail.passwd is email service authorization code, not the email login password.mailPassword="hry10086,"# TLS mail protocol supportstarttlsEnable="false"# SSL mail protocol support# only one of TLS and SSL can be in the true state.sslEnable="false"#note: sslTrust is the same as mailServerHostsslTrust="smtp.163.com"# resource storage type：HDFS,S3,NONEresourceStorageType="HDFS"# if resourceStorageType is HDFS，defaultFS write namenode address，HA you need to put core-site.xml and hdfs-site.xml in the conf directory.# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，# Note，s3 be sure to create the root directory /dolphinschedulerdefaultFS="hdfs://bigdata:9820"# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore#s3Endpoint=";#s3AccessKey="xxxxxxxxxx"#s3SecretKey="xxxxxxxxxx"# if resourcemanager HA enable, please type the HA ips ; if resourcemanager is single, make this value empty#yarnHaIps="192.168.xx.xx,192.168.xx.xx"# if resourcemanager HA enable or not use resourcemanager, please skip this value setting; If resourcemanager is single, you only need to replace yarnIp1 to actual resourcemanager hostname.singleYarnIp="bigdata"# resource store on HDFS/S3 path, resource file will store to this hadoop hdfs path, self configuration, please make sure the directory exists on hdfs and have read write permissions。/dolphinscheduler is recommendedresourceUploadPath="/dolphinscheduler"# who have permissions to create directory under HDFS/S3 root path# Note: if kerberos is enabled, please config hdfsRootUser=hdfsRootUser="root"# kerberos config# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignorekerberosStartUp="false"# kdc krb5 config file pathkrb5ConfPath="$installPath/conf/krb5.conf"# keytab usernamekeytabUserName="hdfs-mycluster@ESZ.COM"# username keytab pathkeytabPath="$installPath/conf/hdfs.headless.keytab"# api server portapiServerPort="12345"# install hosts# Note: install the scheduled hostname list. If it is pseudo-distributed, just write a pseudo-distributed hostnameips="bigdata"# ssh port, default 22# Note: if ssh port is not default, modify heresshPort="22"# run master machine# Note: list of hosts hostname for deploying mastermasters="bigdata"# run worker machine# note: need to write the worker group name of each worker, the default value is "default"workers="bigdata:default"# run alert machine# note: list of machine hostnames for deploying alert serveralertServer="bigdata"# run api machine# note: list of machine hostnames for deploying api server

说明: 机器打标签能够按照机器不一样的性能打不一样的标签,好比机器性能高的打high的标签,将重要任务就能够放在性能高的机器执行.性能低的low标签,通常任务就指定在性能低的机器执行.

10. 将core-site.xml和hdfs-site.xml文件拷贝到conf目录下(以超连接的方式)

[root@linux122 dolphinscheduler-bin]# ln -s /opt/servers/hadoop-2.9.2/etc/hadoop/core-site.xml ./conf/core-site.xml[root@linux122 dolphinscheduler-bin]# ln -s /opt/servers/hadoop-2.9.2/etc/hadoop/hdfs-site.xml ./conf/hdfs-site.xml

11. 修改各个服务日志的存放目录(这里所有修改成存放在/logs/ds目录下)

[root@linux122 conf]# vim logback-master.xml[root@linux122 conf]# vim logback-worker.xml[root@linux122 conf]# vim logback-api.xml[root@linux122 conf]# vim logback-alert.xml<property name="log.base" value="/logs/ds"/>(第22行)

12. 切换到部署ds的用户(这里是root),执行一键部署脚本

[root@linux122 dolphinscheduler-bin]# sh install.sh

说明: 须要部署过程当中没有任何报错信息.

13. 部署完成后,服务会自动启动.查看ds各节点的进程状况

12535 ApiApplicationServer12282 MasterServer12474 AlertServer12413 LoggerServer12351 WorkerServer

14. 登录WebUI界面(api-server所在的ip地址)

登录名: admin

登录密码: dolphinscheduler123

1.5 ds的启动和中止命令(首先须要给脚本文件可执行权限)

#根据配置文件中指定的角色群起/群停进程[root@linux122 dolphinscheduler-1.3.2]# bin/start-all.sh[root@linux122 dolphinscheduler-1.3.2]# bin/stop-all.sh#单独起/停某个进程[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh start master-server[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh stop master-server[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh start worker-server[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh stop worker-server[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh start api-server[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh stop api-server

2.DolphinScheduler的基本使用(有中文版操做界面)2.1安全中心(安全中心的设置须要使用admin用户来操做)2.1.1指定yarn调度器中存在的队列

2.1.2 worker分组

若是ds的worker节点数量不少,能够将不一样的机器划分为不一样的组,以便后期不一样优先级的任务调度在不一样的组,能够配置多个.(这里在安装时只配了一个default组)

2.1.3告警组管理

增长告警组:

为不一样的告警组添加用户(须要先在用户管理界面使用admin帐户添加该用户):

说明:这样操做以后该告警组就与这个用户的邮箱进行绑定了,只要后面像该告警组发邮件,就会发给告警组里面的全部用户.

2.1.4租户管理

这里租户至关于就是linux上的用户,当关联这个租户时就是指定了以linux上的哪一个用户来提交任务,若是这个用户在linux上存在则直接切换到该用户,若是不存在则建立该用户.

2.1.5用户管理

2.2ds建立项目

这里以xiaobai这个用户在作演示,通常实际生产中admin用户也只负责管理,也不会操做具体项目.且这里xiaobai关联的是root这个租户.

2.2.1建立一个项目

2.2.2为项目建立具体工做和task并运行

工做流,任务相关说明:

建立一个工做流程:

2.2.3查看运行的结果及相关信息

2.2.4查看工做流中具体的task实例

说明:若是上面有选择告警的话查看邮箱就可看到邮件了

2.2.5针对失败的工做流的处理

2.3链接外部数据源2.3.1链接hive的数据源

2.3.2测试使用hive的数据源进行sql查询

建立一个项目用来测试Hive的sql查询:

说明: 最下面贴执行的sql语句时最后不要加分号.且只能执行一整段sql语句.

2.4执行包含hql的shell脚本2.4.1在资源中心建立文件夹

首先建立一个文件夹,能够用来管理同一模块下的许多脚本,jar包以及用到的其它文件.

2.4.2在文件夹下建立或上传脚本

2.4.3建立文件

1. 将shell脚本的内容所有粘贴到文件内容中(文件名称不用带.sh的后缀)

说明: 执行hql命令有2种方式,一种是cli的方式,另外一种是beeline的方式.生产上基本都是使用beeline的方式.

能够看到在该目录下获得了一个新的文件

2.4.4建立并执行工做流

1. 根据步骤2.2中的说明建立项目(并定义工做流)

2. 调用建立好的资源文件来建立task,且是使用shell的方式

3. 点击左下角确认添加

4. 点击右上角的保存

5. 设置工做流DAG图的相关信息

6. 将建立好的工做流上线

7. 选择立刻运行或定时调度

8. 每一次运行就会获得一个工做流实例

9. 每一个工做流实例中有1个或多个task实例

2.5子节点(sub_process)

子节点只能选择同一个项目下的工做流,即实现同一个项目下工做流之间的依赖.能够在一个工做流中引入其它工做流做为子节点.当运行当前工做流时,会先运行子节点,子节点运行成功后才继续执行后面的工做流.

2.6依赖节点(denpency)

依赖节点能够实现项目之间的依赖.B项目依赖A项目,那么只有A项目执行成功后才会继续执行B项目.

2.7重试机制

调度判断一个shell脚本执行成功仍是失败的标识: exit 0表示成功(执行完成后默认为exit 0),exit 1表示失败(有时能够手动将正常的脚本最后显性执行exit 1,则会被标记为失败).能够在定义task时开启重试.

2.8补数机制

在补数的时候会根据调度日期日后推迟一天,因此若是须要补10月1号到10月8号的数据,须要选择的调度日期是9月30号到10月7号进行补数.固然这个能够经过在传递全局参数时经过给全局变量的值为$[yyyy-MM-dd-1],选择的调度日期就是补数的日期了.

2.8.1实际场景中的应用(针对的是一个工做流中全部task的补数)

在执行shell脚本时经过${date}来获取外界传入的日期.

点击右上角保存,将该变量设置为全局变量,value值设置为$[yyyy-MM-dd-1]或不给值.这样在调度该脚本时那么脚本里面获取的日期就是当前时间的前一天.可是若是须要使用后的补数机制时必须给value的值,这里建议给$[yyyy-MM-dd-1],那么在选择调度日期时就是补数的日期.

经过设置value的值为$[yyyy-MM-dd-1],那么脚本传入的日期就是选择的调度日期(不存在加1天的状况).

补数的工做流:

2.9中止,暂停,重跑的机制

l 重跑：从新执行已经终止的流程。

l 恢复失败：针对失败的流程，能够执行恢复失败操做，从失败的节点开始执行。

l 中止：对正在运行的流程进行中止操做，后台会先killworker进程,再执行kill -9操做

l 暂停：对正在运行的流程进行暂停操做，系统状态变为等待执行，会等待正在执行的任务结束，暂停下一个要执行的任务。

l 恢复暂停：对暂停的流程恢复，直接从暂停的节点开始运行

l 删除：删除工做流实例及工做流实例下的任务实例

2.10定时任务调度(针对工做流)

1. 进入定时配置界面

2. 配置调度相关参数

3. 上线任务调度(须要修改调度任务时要先下线,再修改,再上线)

4. 从工做流中进入已经设定好的调度任务

注意:

当咱们下线工做流时,定时任务调度会跟着下线.而再次上线工做流时,里面定时任务调度不会跟着上线,须要咱们手动上线.

2.11失败策略机制

在运行以前设置失败策略:

当一个工做流中,若是有一个或几个task执行失败,能够有继续和结束两种处理策略.

继续策略:当其中一个或几个节点失败时,只有依赖它的节点会所有中止执行.而与它没有依赖关系,并行的节点仍将继续执行.

结束策略:当工做流中有1个节点失败时,整个工做流的全部后续的其它节点都会中止执行.

3.ds的企业微信告警3.1修改配置文件

在alert-server所在的机器上修改conf/alert.properties配置文件

3.2配置文件参数说明以及获取方式

1. 开启企业微信报警

# 设置企业微信告警功能是否开启：开启为 true，不然为 false。

enterprise.wechat.enable=true

2. 设置corp.id配置参数

# 设置 corpid，每一个企业都拥有惟一的corpid,获取此信息可在管理后台“个人企业”－“企业信息”下查看“企业 ID”（须要有管理员权限）

enterprise.wechat.corp.id=xxxxx

3. 将图片中所示的secret和agentid填入配置文件中

# 设置secret，secret是企业应用里面用于保障数据安全的“钥匙”，每个应用都有一个独立的访问密钥。

enterprise.wechat.secret=xxxx

# 设置 agentid，每一个应用都有惟一的 agentid。在管理后台->“应用与小程序”->“应用”，点进某个应用，便可看到 agentid。

enterprise.wechat.agent.id="xxxx"

4. 在配置文件中增长userid,表示哪些用户能够看到这个应用的推送消息(这些用户必须在上面图中所示该应用的可见范围中)

# 设置 userid，多个用逗号分隔。每一个成员都有惟一的 userid，即所谓“账号”。在管理后台->“通信录”->点进某个成员的详情页，能够看到。

enterprise.wechat.users=zhangsan,lisi,wangwu

5. 其它配置项无需改动

# 获取 access_token 的地址，无需修改enterprise.wechat.token.url= 发送应用消息地址，无需修改enterprise.wechat.push.url= 无需修改enterprise.wechat.team.send.msg={\"toparty\":\"$toParty\",\"agentid\":\"$agentId\",\"msgtype\":\"text\",\"text\":{\"con    tent\":\"$msg\"},\"safe\":\"0\"}# 无需修改enterprise.wechat.user.send.msg={\"touser\":\"$toUser\",\"agentid\":\"$agentId\",\"msgtype\":\"markdown\",\"markdown\":    {\"content\":\"$msg\"}}

6. 重启配置文件所在机器的alert-server服务,而后进行测试

[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh stop alert-server[root@linux122 dolphinscheduler-1.3.2]# bin/dolphinscheduler-daemon.sh start alert-server

4.常见问题说明

1. 默认状况下,admin用户是没有关联任何租户的,若是是使用admin来上传文件,建立工做流这些,须要先关联已有的租户.但通常生产中admin不会关联租户,由于不须要建立具体的项目.

2. 使用admin帐户对普通用户进行项目,资源文件等进行受权

3. 修改工做流或工做流里面的task须要先将工做流下线

说明:每次修改完以后记得保存.

4. 对于失败的任务进行重跑的方法

说明: 选择恢复失败时已成功的节点再也不执行.

5. 工做流下线后再上线时,设置的定时任务并无跟着上线,须要手动上线定时任务.

6. 任务优先级,在线程数不足的时候会根据任务优先级来进行处理.

7. 超时告警,若是在指定的时间内task没有执行完成就会告警,且将该task置为失败

8. 在脚本里面引用参数: ${name}

9. 针对整个工做流的超时告警: