龙空技术网

动态线程池dynamic-tp快速接入

程序那点事 507

前言:

现时朋友们对“tp多数据库配置”大致比较珍视,各位老铁们都想要分析一些“tp多数据库配置”的相关知识。那么小编在网络上汇集了一些有关“tp多数据库配置””的相关知识,希望兄弟们能喜欢,兄弟们一起来了解一下吧!

前言

dynamic-tp是一个轻量级的动态线程池插件,它是一个基于配置中心的动态线程池,线程池的参数可以通过配置中心配置进行动态的修改,目前支持的配置中心有Apollo,Nacos和Zookeeper,同时dynamic-tp支持线程池的监控和报警,具体特性如下:

基于Spring框架,现只支持SpringBoot项目使用,轻量级,引入 starter 即可使用基于配置中心实现线程池参数动态调整,实时生效;集成主流配置中心,已支持 Nacos、Apollo,Zookeeper, 同时也提供 SPI 接口可自定义扩展实现内置通知报警功能,提供多种报警维度(配置变更通知、活性报警、容量阈值报警、拒绝策略触发报警), 默认支持企业微信、钉钉报警,同时提供 SPI 接口可自定义扩展实现内置线程池指标采集功能,支持通过 MicroMeter、JsonLog 日志输出、Endpoint 三种方式,可通过 SPI 接口自定义扩展实现集成管理常用第三方组件的线程池,已集成 SpringBoot 内置 WebServer(tomcat、undertow、jetty)的线程池管理Builder 提供TTL包装功能,生成的线程池支持上下文信息传递

具体介绍及使用方式可以参考官网

为了方便用户快速接入,笔者这里对Zookeeper配置中心接入做一个详细的介绍,其他配置中心的demo示例可以参考项目的example,该模块也是笔者贡献的。这里我会详细的介绍Zookeeper配置中心的接入和对接监控平台Prometheus和Grafana。

快速开始pom依赖

dynamic-tp-spring-boot-starter-zookeeper是集成dynamic-tp的starter,这里引入了micrometer-registry-prometheus和spring-boot-starter-actuator用于对接Prometheus。

    <dependencies>        <dependency>            <groupId>org.springframework.boot</groupId>            <artifactId>spring-boot-starter-web</artifactId>        </dependency>        <dependency>            <groupId>org.springframework.boot</groupId>            <artifactId>spring-boot-starter-logging</artifactId>        </dependency>        <dependency>            <groupId>io.github.lyh200</groupId>            <artifactId>dynamic-tp-spring-boot-starter-zookeeper</artifactId>            <version>1.0.2</version>        </dependency>        <dependency>            <groupId>io.micrometer</groupId>            <artifactId>micrometer-registry-prometheus</artifactId>            <version>1.6.4</version>        </dependency>        <dependency>            <groupId>org.springframework.boot</groupId>            <artifactId>spring-boot-starter-actuator</artifactId>        </dependency>    </dependencies>复制代码
YML配置
server:  port: 8888spring:  application:    name: dynamic-tp-zookeeper-demo  # 下面是接入zk配置中心的配置    dynamic:    tp:      config-type: properties # zookeeper只支持properties配置      zookeeper:        config-version: 1.0.0 # 配置版本号        zk-connect-str: 127.0.0.1:2181 # zk配置中心,如果是集群用逗号分开        root-node: /configserver/userproject # 项目节点        node: dynamic-tp-zookeeper-demo # 配置文件节点# 对接prometheusmanagement:  metrics:    tags:      application: ${spring.application.name}  endpoints:    web:      exposure:        include: '*'复制代码
配置中心dynamic-tp-zookeeper-demo配置文件配置

注:Zookeeper配置中心只支持properties类型配置,配置示例如下:

spring.dynamic.tp.executors部分是对线程池的配置,该配置是数组类型,可以定义多个线程池 spring.dynamic.tp.executors[1].notifyItems部分配置是对线程池报警平台的配置,可以配置多个报警平台

# 开启动态线程池spring.dynamic.tp.enabled=true# 打印动态线程池bannerspring.dynamic.tp.enabledBanner=true# 开启线程池监控指标收集spring.dynamic.tp.enabledCollect=true# 线程池监控指标收集类型 logging-日志文件 micrometer-采集到第三方平台 spring.dynamic.tp.collectorType=logging# 采集监控数据时间间隔 5sspring.dynamic.tp.monitorInterval=5spring.dynamic.tp.executors[0].threadPoolName=dynamic-tp-test-1spring.dynamic.tp.executors[0].corePoolSize=50spring.dynamic.tp.executors[0].maximumPoolSize=50spring.dynamic.tp.executors[0].queueCapacity=3000spring.dynamic.tp.executors[0].queueType=VariableLinkedBlockingQueuespring.dynamic.tp.executors[0].rejectedHandlerType=CallerRunsPolicyspring.dynamic.tp.executors[0].keepAliveTime=50spring.dynamic.tp.executors[0].allowCoreThreadTimeOut=falsespring.dynamic.tp.executors[0].threadNamePrefix=test1spring.dynamic.tp.executors[0].notifyItems[0].type=capacityspring.dynamic.tp.executors[0].notifyItems[0].enabled=falsespring.dynamic.tp.executors[0].notifyItems[0].threshold=80spring.dynamic.tp.executors[0].notifyItems[0].platforms[0]=dingspring.dynamic.tp.executors[0].notifyItems[0].platforms[1]=wechatspring.dynamic.tp.executors[0].notifyItems[0].interval=120spring.dynamic.tp.executors[0].notifyItems[1].type=changespring.dynamic.tp.executors[0].notifyItems[1].enabled=falsespring.dynamic.tp.executors[0].notifyItems[2].type=livenessspring.dynamic.tp.executors[0].notifyItems[2].enabled=falsespring.dynamic.tp.executors[0].notifyItems[2].threshold=80spring.dynamic.tp.executors[0].notifyItems[3].type=rejectspring.dynamic.tp.executors[0].notifyItems[3].enabled=falsespring.dynamic.tp.executors[0].notifyItems[3].threshold=1spring.dynamic.tp.executors[1].threadPoolName=dynamic-tp-test-2spring.dynamic.tp.executors[1].corePoolSize=20spring.dynamic.tp.executors[1].maximumPoolSize=30spring.dynamic.tp.executors[1].queueCapacity=1000spring.dynamic.tp.executors[1].queueType=VariableLinkedBlockingQueuespring.dynamic.tp.executors[1].rejectedHandlerType=CallerRunsPolicyspring.dynamic.tp.executors[1].keepAliveTime=50spring.dynamic.tp.executors[1].allowCoreThreadTimeOut=falsespring.dynamic.tp.executors[1].threadNamePrefix=test2spring.dynamic.tp.executors[1].notifyItems[0].type=capacityspring.dynamic.tp.executors[1].notifyItems[0].enabled=falsespring.dynamic.tp.executors[1].notifyItems[0].threshold=80spring.dynamic.tp.executors[1].notifyItems[0].platforms[0]=dingspring.dynamic.tp.executors[1].notifyItems[0].platforms[1]=wechatspring.dynamic.tp.executors[1].notifyItems[0].interval=120spring.dynamic.tp.executors[1].notifyItems[1].type=changespring.dynamic.tp.executors[1].notifyItems[1].enabled=falsespring.dynamic.tp.executors[1].notifyItems[2].type=livenessspring.dynamic.tp.executors[1].notifyItems[2].enabled=falsespring.dynamic.tp.executors[1].notifyItems[2].threshold=80spring.dynamic.tp.executors[1].notifyItems[3].type=rejectspring.dynamic.tp.executors[1].notifyItems[3].enabled=falsespring.dynamic.tp.executors[1].notifyItems[3].threshold=1复制代码
创建线程池

创建线程池,会注册到dynamic-tp

@Configurationpublic class ThreadPoolConfiguration {    @Bean    public DtpExecutor dtpExecutor() {        return ThreadPoolCreator.createDynamicFast("dynamic-tp-test-1");    }    @Bean    public ThreadPoolExecutor threadPoolExecutor() {        return ThreadPoolBuilder.newBuilder()                .threadPoolName("dynamic-tp-test-2")                .corePoolSize(10)                .maximumPoolSize(15)                .keepAliveTime(15000)                .timeUnit(TimeUnit.MILLISECONDS)                .workQueue(QueueTypeEnum.SYNCHRONOUS_QUEUE.getName(), null, false)                .buildDynamic();    }}复制代码
使用

初次使用不注意官网文档的话可能会通过@Autowired的方式使用,目前版本这么使用是不正确的,因为当读取配置文件后,配置文件生成的线程池对象就会把@Bean创建的线程池覆盖了,所以目前版本只能根据名字从DtpRegistry获取线程池对象了,笔者也和项目作者沟通过,项目作者也觉得通过@Autowired方式更好一些,后面版本也会优化。

@Slf4j@RestController@SuppressWarnings("all")public class TestController {    @GetMapping("/dtp-zookeeper-example/test")    public String test() {        new Thread(() -> {            try {                task();            } catch (InterruptedException e) {                e.printStackTrace();            }        }).start();        return "success";    }    public void task() throws InterruptedException {        DtpExecutor dtpExecutor1 = DtpRegistry.getExecutor("dynamic-tp-test-1");        DtpExecutor dtpExecutor2 = DtpRegistry.getExecutor("dynamic-tp-test-2");        for (int i = 0; i < 100; i++) {            Thread.sleep(100);            dtpExecutor1.execute(() -> {                log.info("i am dynamic-tp-test-1 task");            });            dtpExecutor2.execute(() -> {                log.info("i am dynamic-tp-test-2 task");            });        }    }}复制代码
监控数据和第三方平台对接logging方式

spring.dynamic.tp.collectorType=logging,logging方式的监控数据,采集日志文件中,文件的位置可以通过spring.dynamic.tp.logPath配置,默认 ${user.home}/logs,日志内容如下:

2022-03-07 13:48:31.585 INFO [dtp-monitor1:D.M.LOG] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"CallerRunsPolicy","queueCapacity":0,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":0,"corePoolSize":40,"queueType":"SynchronousQueue","completedTaskCount":0,"dtpName":"dynamic-tp-test-2","maximumPoolSize":55}2022-03-07 13:48:31.585 INFO [dtp-monitor1:D.M.LOG] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"queueCapacity":2147483647,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":2147483647,"corePoolSize":10,"queueType":"TaskQueue","completedTaskCount":0,"dtpName":"tomcatWebServerTp","maximumPoolSize":200}2022-03-07 13:48:36.590 INFO [dtp-monitor1:D.M.LOG] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"CallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":50,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"dynamic-tp-test-1","maximumPoolSize":50}2022-03-07 13:48:36.590 INFO [dtp-monitor1:D.M.LOG] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"CallerRunsPolicy","queueCapacity":0,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":0,"corePoolSize":40,"queueType":"SynchronousQueue","completedTaskCount":0,"dtpName":"dynamic-tp-test-2","maximumPoolSize":55}2022-03-07 13:48:36.590 INFO [dtp-monitor1:D.M.LOG] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"queueCapacity":2147483647,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":2147483647,"corePoolSize":10,"queueType":"TaskQueue","completedTaskCount":0,"dtpName":"tomcatWebServerTp","maximumPoolSize":200}复制代码
MicroMeter方式

spring.dynamic.tp.collectorType=micrometer 采集数据到第三方平台,这里使用的是Prometheus。可视化展示需要在Grafana平台配置,数据源选择Prometheus,然后创建Dashboard,先要在Dashboard配置动态的参数Variables,如下图:

配置完参数后,我们创建Panel,可以创建多个Panel,然后配置Panel的数据查询,报警显示等等,如下图:

最终的Dashboard如下图:

总结

促使我使用dynamic-tp的一个最主要的点就是它轻量级,提供了线程池监控、报警、基于配置中心修改线程池参数功能,完全满足了我在业务中的需要,同时监控数据能够对接公司现有的监控平台(Prometheus+Grafana),并且在使用上也十分方便,我在刚接触这个插件的时候它还不支持Zookeeper,恰好我司使用的配置中心就是Zookeeper,因为我在此基础上把Zookeeper配置中心集成到了dynamic-tp并贡献给社区,希望可以帮到更多的人,下一篇简单介绍下dynamic-tp支持Zookeeper的实现细节。

作者:刘p辉

链接:

标签: #tp多数据库配置