本文最后更新于 67 天前，其中的信息可能已经过时，如有错误请发送邮件到wuxianglongblog@163.com

kafka集群的监控软件Kafka_Eagle

一.kafka启动JMX端口

1.修改kafka启动命令

[root@elk101.oldboyedu.com ~]# egrep export /oldboy/softwares/kafka/bin/kafka-server-start.sh 
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
    # export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    export KAFKA_HEAP_OPTS="-Xmx256M -Xms256M"
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# vim /oldboy/softwares/kafka/bin/kafka-server-start.sh  # 注意前后修改的变化哟~
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# egrep export /oldboy/softwares/kafka/bin/kafka-server-start.sh 
    export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/../config/log4j.properties"
    # export KAFKA_HEAP_OPTS="-Xmx1G -Xms1G"
    # export KAFKA_HEAP_OPTS="-Xmx256M -Xms256M"
    export KAFKA_HEAP_OPTS="-server -Xmx256M -Xms256M -XX:PermSize=128m -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:ParallelGCThreads=8 -XX:ConcGCThreads=5 -XX:InitiatingHeapOccupancyPercent=70"    
    export JMX_PORT="8888"
[root@elk101.oldboyedu.com ~]# 

相关参数说明：
    KAFKA_HEAP_OPTS:
        设置kafka的堆内存大小。以下是本案例中涉及到有关堆内存调优的相关参数:
            "-Xms256M":
                表示设置JVM启动内存的最小值为256M，必须以M为单位。
                kafka项目推荐设置为5-6G即可。因为kafka并不是特别吃内存，它的数据是存储在磁盘上的。

            "-Xmx256M":
                表示设置JVM启动内存的最大值为256M，必须以M为单位。将-Xmx和-Xms设置为一样可以避免JVM内存自动扩展。
                kafka项目推荐设置为5-6G即可。因为kafka并不是特别吃内存，它的数据是存储在磁盘上的。

            "-XX:PermSize=128m":
                表示JVM初始分配的永久代(方法区)的容量，必须以M为单位。

            "-XX:+UseG1GC":
                表示让JVM使用G1垃圾收集器

            "-XX:MaxGCPauseMillis=200":
                设置每次年轻代垃圾回收的最长时间为200ms，如果无法满足此时间，JVM会自动调整年轻代大小，以满足此值。

            "-XX:ParallelGCThreads=8":
                设置并行垃圾回收的线程数，此值可以设置与机器处理器数量相等。

            "-XX:ConcGCThreads=5":
                设置Concurrent Mark Sweep(简称"CMS"，CMS处理器关注的是停顿时间。由于CMS处理器较为复杂，因此该收集器参数较多，这里只是冰山一角，感兴趣的小伙伴可自行查阅相关文档)并发线程数。

            "-XX:InitiatingHeapOccupancyPercent=70":
                该参数可以指定当整个堆使用率达到多少时，触发并发标记周期的执行。默认值是45，即当堆的使用率达到45%，执行并发标记周期，该值一旦设置，始终都不会被G1修改。
                也就是说，G1就算为了满足MaxGCPauseMillis也不会修改此值。如果该值设置的很大，导致并发周期迟迟得不到启动，那么引起FGC的几率将会变大。如果过小，则会频繁标记，GC线程抢占应用程序CPU资源，性能将会下降。 

    JMX_PORT:
        设置JMX监控的端口。

温馨提示:
    "-Xms"和"-Xmx"在实际生产环境中我们通常会设置成相同的值，这是为了避免在生产环境由于heap内存扩大或缩小导致应用停顿，降低延迟，同时避免每次垃圾回收完成后JVM重新分配内存。

2.修改脚本后，记得分发到集群的其它节点哟~

[root@elk101.oldboyedu.com ~]# cat `which data_rsync.sh`
#!/bin/bash
# author:
#    oldboyedu linux77

# 判断用户是否传参
if [ $# -lt 1 ];then
    echo "Please enter parameters, using: $0 <file path>";
    exit
fi

# 获取文件路径
file=$1

# 获取子路径
filename=`basename $file`

# 获取父路径
dirpath=`dirname $file`

# 进入到父目录路径
cd $dirpath

# 值得注意的是，`pwd -P`获取的是真实的路径，如果当前路径为符号链接，则返回的是符号链接指向的目标路径。
# 若`pwd`指令不指定选项，则默认为`pwd -L`
fullpath=`pwd -L`

# 将kafka106实例的数据文件同步到Kafka集群的其他节点
for (( i=107;i<=108;i++ ))
  do
    # 使终端变绿色
    tput setaf 2
    echo =========== 10.0.0.${i} : $file ===========
    # 使终端变回原来的颜色，即白灰色
    tput setaf 7
    # 远程执行命令
    rsync -lr $filename `whoami`@10.0.0.${i}:$fullpath
    # 判断命令是否执行成功
    if [ $? == 0 ];then
        echo "命令执行成功"
    else 
        echo "命令执行失败................"
        exit 0
    fi
done
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# data_rsync.sh /oldboy/softwares/kafka/bin/kafka-server-start.sh 
=========== kafka102.oldboyedu.com : /oldboy/softwares/kafka/bin/kafka-server-start.sh ===========
命令执行成功
=========== kafka103.oldboyedu.com : /oldboy/softwares/kafka/bin/kafka-server-start.sh ===========
命令执行成功
[root@elk101.oldboyedu.com ~]#

3.重启kafka集群目的是让上面的修改生效

[root@elk101.oldboyedu.com ~]# manager-kafka.sh start
****** elk101.oldboyedu.com  ---> [manager-kafka.sh: start ] ******
elk101.oldboyedu.com 服务已启动
****** kafka102.oldboyedu.com  ---> [manager-kafka.sh: start ] ******
kafka102.oldboyedu.com 服务已启动
****** kafka103.oldboyedu.com  ---> [manager-kafka.sh: start ] ******
kafka103.oldboyedu.com 服务已启动
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# manager-kafka.sh status
****** elk101.oldboyedu.com  ---> [manager-kafka.sh: status ] ******
elk101.oldboyedu.com | CHANGED | rc=0 >>
8039 Kafka
8248 Jps
1390 QuorumPeerMain
kafka102.oldboyedu.com | CHANGED | rc=0 >>
2080 jar
6096 Kafka
6711 Jps
1531 QuorumPeerMain
kafka103.oldboyedu.com | CHANGED | rc=0 >>
1363 QuorumPeerMain
14950 Kafka
15562 Jps
[root@elk101.oldboyedu.com ~]#

二.安装MySQL数据存储kafka-eagle-web组件的内容

1.安装MariaDB系列数据库并设置开启自启动

[root@elk101.oldboyedu.com ~]# yum -y install mariadb-server
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# systemctl start mariadb
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# systemctl enable mariadb

2.创建授权用户

[root@elk101.oldboyedu.com ~]# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 2
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>
MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| test               |
+--------------------+
4 rows in set (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> CREATE DATABASE oldboyedu_kafka DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| oldboyedu_kafka    |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> SELECT user,host FROM mysql.user;
+------+--------------------------+
| user | host                     |
+------+--------------------------+
| root | 127.0.0.1                |
| root | ::1                      |
|      | elk101.oldboyedu.com |
| root | elk101.oldboyedu.com |
|      | localhost                |
| root | localhost                |
+------+--------------------------+
6 rows in set (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> CREATE USER admin@'elk101.oldboyedu.com' IDENTIFIED BY 'oldboyedu';  # 创建用户
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> SELECT user,host FROM mysql.user;
+-------+--------------------------+
| user  | host                     |
+-------+--------------------------+
| root  | 127.0.0.1                |
| root  | ::1                      |
|       | elk101.oldboyedu.com |
| admin | elk101.oldboyedu.com |
| root  | elk101.oldboyedu.com |
|       | localhost                |
| root  | localhost                |
+-------+--------------------------+
7 rows in set (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> GRANT ALL ON oldboyedu_kafka.* TO admin@'elk101.oldboyedu.com';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> SHOW GRANTS FOR admin@'elk101.oldboyedu.com';
+-----------------------------------------------------------------------------------------------------------------------------+
| Grants for admin@elk101.oldboyedu.com                                                                                   |
+-----------------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'admin'@'elk101.oldboyedu.com' IDENTIFIED BY PASSWORD '*BD0B1F48FDC55BD27555FC2F22FF29A68A25A1D7' |
| GRANT ALL PRIVILEGES ON `oldboyedu_kafka`.* TO 'admin'@'elk101.oldboyedu.com'                                                     |
+-----------------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> QUIT
Bye
[root@elk101.oldboyedu.com ~]#

3.测试授权用户

[root@elk101.oldboyedu.com ~]# mysql -u admin -poldboyedu -h elk101.oldboyedu.com
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 12
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW DATABSES;
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'DATABSES' at line 1
MariaDB [(none)]> 
MariaDB [(none)]> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| oldboyedu_kafka              |
| test               |
+--------------------+
3 rows in set (0.00 sec)

MariaDB [(none)]> 
MariaDB [(none)]> QUIT
Bye
[root@elk101.oldboyedu.com ~]#

三.监控软件Kafka Eagle的部署

1.下载Kafka Eagle软件包

    下载地址:
        http://download.kafka-eagle.org/

    官方文档:
        http://www.kafka-eagle.org/articles/docs/documentation.html

2.解压安装包

[root@elk101.oldboyedu.com ~]# ll
总用量 160896
-rw-r--r--. 1 root root 12387614 4月  24 18:31 apache-zookeeper-3.7.0-bin.tar.gz
drwxr-xr-x  2 root root       59 5月   7 17:04 kafka
-rw-r--r--  1 root root 71403603 5月   5 16:48 kafka_2.13-2.8.0.tgz
-rw-r--r--  1 root root 80959303 5月  11 10:11 kafka-eagle-bin-2.0.5.tar.gz
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# tar zxf kafka-eagle-bin-2.0.5.tar.gz 
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# ll
总用量 160896
-rw-r--r--. 1 root root 12387614 4月  24 18:31 apache-zookeeper-3.7.0-bin.tar.gz
drwxr-xr-x  2 root root       59 5月   7 17:04 kafka
-rw-r--r--  1 root root 71403603 5月   5 16:48 kafka_2.13-2.8.0.tgz
drwxrwxr-x  2 root root       46 5月   7 22:54 kafka-eagle-bin-2.0.5
-rw-r--r--  1 root root 80959303 5月  11 10:11 kafka-eagle-bin-2.0.5.tar.gz
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# ll kafka-eagle-bin-2.0.5
总用量 79052
-rw-rw-r-- 1 root root 80947724 5月   7 22:54 kafka-eagle-web-2.0.5-bin.tar.gz
[root@elk101.oldboyedu.com ~]#

3.安装kafka-eagle-web软件包并创建符号链接

[root@elk101.oldboyedu.com ~]# ll kafka-eagle-bin-2.0.5
总用量 79052
-rw-rw-r-- 1 root root 80947724 5月   7 22:54 kafka-eagle-web-2.0.5-bin.tar.gz
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# tar zxf kafka-eagle-bin-2.0.5/kafka-eagle-web-2.0.5-bin.tar.gz -C /oldboy/softwares/
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# cd /oldboy/softwares/
[root@elk101.oldboyedu.com /oldboy/softwares]# 
[root@elk101.oldboyedu.com /oldboy/softwares]# ll
总用量 0
drwxr-xr-x. 7 root root 145 4月  28 09:31 apache-zookeeper-3.7.0-bin
lrwxrwxrwx. 1 root root  36 4月  22 20:38 jdk -> /oldboy/softwares/jdk1.8.0_201/
drwxr-xr-x. 7   10  143 245 12月 16 2018 jdk1.8.0_201
lrwxrwxrwx  1 root root  16 5月   5 16:50 kafka -> kafka_2.13-2.8.0
drwxr-xr-x  8 root root 117 5月   7 10:51 kafka_2.13-2.8.0
drwxr-xr-x  8 root root  74 5月  11 11:37 kafka-eagle-web-2.0.5
lrwxrwxrwx. 1 root root  50 4月  22 21:00 zookeeper -> /oldboy/softwares/apache-zookeeper-3.7.0-bin/
[root@elk101.oldboyedu.com /oldboy/softwares]# 
[root@elk101.oldboyedu.com /oldboy/softwares]# ln -sv kafka-eagle-web-2.0.5 kafka-eagle
"kafka-eagle" -> "kafka-eagle-web-2.0.5"
[root@elk101.oldboyedu.com /oldboy/softwares]# 
[root@elk101.oldboyedu.com /oldboy/softwares]# ll
总用量 0
drwxr-xr-x. 7 root root 145 4月  28 09:31 apache-zookeeper-3.7.0-bin
lrwxrwxrwx. 1 root root  36 4月  22 20:38 jdk -> /oldboy/softwares/jdk1.8.0_201/
drwxr-xr-x. 7   10  143 245 12月 16 2018 jdk1.8.0_201
lrwxrwxrwx  1 root root  16 5月   5 16:50 kafka -> kafka_2.13-2.8.0
drwxr-xr-x  8 root root 117 5月   7 10:51 kafka_2.13-2.8.0
lrwxrwxrwx  1 root root  21 5月  11 11:38 kafka-eagle -> kafka-eagle-web-2.0.5
drwxr-xr-x  8 root root  74 5月  11 11:37 kafka-eagle-web-2.0.5
lrwxrwxrwx. 1 root root  50 4月  22 21:00 zookeeper -> /oldboy/softwares/apache-zookeeper-3.7.0-bin/
[root@elk101.oldboyedu.com /oldboy/softwares]#

4.为Kafka Eagle配置环境变量

[root@elk101.oldboyedu.com ~]# vim /etc/profile.d/kafka_eagle.sh 
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# cat /etc/profile.d/kafka_eagle.sh 
#ADD kafka-Eagle path by oldboyedu
export KE_HOME=/oldboy/softwares/kafka-eagle
export PATH=$PATH:$KE_HOME/bin
[root@elk101.oldboyedu.com ~]# 
[root@elk101.oldboyedu.com ~]# source /etc/profile.d/kafka_eagle.sh 
[root@elk101.oldboyedu.com ~]#

5.修改kafka eagle的配置文件

[root@elk101.oldboyedu.com ~]# vim /oldboy/softwares/kafka-eagle/conf/system-config.properties 
...

kafka.eagle.zk.cluster.alias=oldboyedu_kafka
oldboyedu_kafka.zk.list=elk101.oldboyedu.com:2181,elk102.oldboyedu.com:2181,elk103.oldboyedu.com:2181/oldboyedu_kafka280
oldboyedu_kafka.kafka.eagle.broker.size=20
kafka.zk.limit.size=32
kafka.eagle.webui.port=8048
oldboyedu_kafka.kafka.eagle.offset.storage=kafka
oldboyedu_kafka.kafka.eagle.jmx.uri=service:jmx:rmi:///jndi/rmi://%s/jmxrmi
kafka.eagle.metrics.charts=true
kafka.eagle.metrics.retain=15
kafka.eagle.sql.topic.records.max=5000
kafka.eagle.topic.token=keadmin
kafka.eagle.driver=com.mysql.cj.jdbc.Driver
kafka.eagle.url=jdbc:mysql://elk101.oldboyedu.com:3306/oldboyedu_kafka?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=con
vertToNullkafka.eagle.username=admin
kafka.eagle.password=oldboyedu
[root@elk101.oldboyedu.com ~]#

6.启动kafka eagle服务

[root@elk101.oldboyedu.com ~]# ke.sh start
[2021-05-11 15:44:34] INFO: Starting  kafka eagle environment check ...
  已创建: META-INF/
  已解压: META-INF/MANIFEST.MF
  已创建: WEB-INF/
  已创建: WEB-INF/classes/

  ...

  已创建: META-INF/maven/org.smartloli.kafka.eagle/
  已创建: META-INF/maven/org.smartloli.kafka.eagle/kafka-eagle-web/
  已解压: META-INF/maven/org.smartloli.kafka.eagle/kafka-eagle-web/pom.xml
  已解压: META-INF/maven/org.smartloli.kafka.eagle/kafka-eagle-web/pom.properties
[2021-05-11 15:44:38] INFO: Port Progress: [##################################################] | 100%
[2021-05-11 15:44:41] INFO: Config Progress: [##################################################] | 100%
[2021-05-11 15:44:44] INFO: Startup Progress: [##################################################] | 100%
[2021-05-11 15:44:34] INFO: Status Code[0]
[2021-05-11 15:44:34] INFO: [Job done!]
Welcome to
    __ __    ___     ____    __ __    ___            ______    ___    ______    __     ______
   / //_/   /   |   / __/   / //_/   /   |          / ____/   /   |  / ____/   / /    / ____/
  / ,<     / /| |  / /_    / ,<     / /| |         / __/     / /| | / / __    / /    / __/   
 / /| |   / ___ | / __/   / /| |   / ___ |        / /___    / ___ |/ /_/ /   / /___ / /___   
/_/ |_|  /_/  |_|/_/     /_/ |_|  /_/  |_|       /_____/   /_/  |_|\____/   /_____//_____/   

Version 2.0.5 -- Copyright 2016-2021
*******************************************************************
* Kafka Eagle Service has started success.
* Welcome, Now you can visit 'http://172.200.1.101:8048'  # 这里是服务的访问地址
* Account:admin ,Password:123456  # 这里是登录的用户名和密码
*******************************************************************
* <Usage> ke.sh [start|status|stop|restart|stats] </Usage>
* <Usage> https://www.kafka-eagle.org/ </Usage>
*******************************************************************
[root@elk101.oldboyedu.com ~]# 

温馨提示:
    [root@elk101.oldboyedu.com ~]# ke.sh start  # 如果启动报错如下不要慌，直接修改该脚本的配置文件，声明JAVA_HOME变量即可.
    [2021-05-11 15:29:42] INFO: Starting  kafka eagle environment check ...
    [2021-05-11 15:29:42] Error: The JAVA_HOME environment variable is not defined correctly.
    [2021-05-11 15:29:42] Error: This environment variable is needed to run this program.
    [root@elk101.oldboyedu.com ~]#

7.kafka_eagle的WebUi登录成功

    如下图所示，输入默认的用户名和密码即可登录成功。

8.监控软件Kafka Eagle的基本使用

    建议顺便讲解一下kafka集群的压力测试，看起来更带劲。

kafka集群的监控软件Kafka_Eagle

一.kafka启动JMX端口

1.修改kafka启动命令

2.修改脚本后，记得分发到集群的其它节点哟~

3.重启kafka集群目的是让上面的修改生效

二.安装MySQL数据存储kafka-eagle-web组件的内容

1.安装MariaDB系列数据库并设置开启自启动

2.创建授权用户

3.测试授权用户

三.监控软件Kafka Eagle的部署

1.下载Kafka Eagle软件包

2.解压安装包

3.安装kafka-eagle-web软件包并创建符号链接

4.为Kafka Eagle配置环境变量

5.修改kafka eagle的配置文件

6.启动kafka eagle服务

7.kafka_eagle的WebUi登录成功

8.监控软件Kafka Eagle的基本使用

发送评论 编辑评论

推荐文章

发送评论编辑评论