简言

通过JMX监控可以获取更详细的监控数据来让我们运维人员方便的分析系统的运行状况和系统存留的风险，这里我们来使用JMX来监控Zk获取更加详细的监控。通过在Zk中埋点来暴露数据，使用Prometheus监控来抓取数据然后借助Grafana来进行数据的展示和告警，即可实现通过Prometheus监控zookeeper的目的。

操作步骤

操作步骤如图所示

步骤一：配置JMX端口

在zookeeper中启用JMX服务以获取资源信息，操作步骤如下：

1.修改/opt/zk/zookeeper-3.4.10/bin/zkServer.sh，在第44行添加JMXPORT=8999。

具体添加位置如下所示：

1
2
3
4
5
6
7
8
9
10
11
if [ "x$JMXLOCALONLY" = "x" ]
then
    JMXLOCALONLY=false
fi

JMXPORT=8999 ## 添加在此处第44行。
if [ "x$JMXDISABLE" = "x" ] || [ "$JMXDISABLE" = 'false' ]
then
  echo "ZooKeeper JMX enabled by default $JMXPORT ..." >&2
  if [ "x$JMXPORT" = "x" ]
  then

2.执行命令进行重启Zookeeper

1
/opt/zk/zookeeper-3.4.10/bin/zkServer.sh start /opt/zk/zookeeper-3.4.10/conf/zoo_sample.cfg &

步骤二: 为应用埋点

启动jmx_exporter，让JMX信息可以直接通过HTTP方式访问，以便Prometheus监控抓取。操作步骤如下：

  1. 下载zookeeper.yaml文件到/opt/exporter_zookeeper/目录下。

  2. 修改下载的/opt/exporter_zookeeper/zookeeper.yaml文件，在第一行添加hostPort: localhost:8999。

    将JMX服务运行的端口暴露给jmx_exporter。

  3. 下载jmx_exporter的可执行文件到opt/exporter_zookeeper/目录下。

  4. 启动jmx_exporter。

1
java -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8998 -cp /opt/exporter_zookeeper/jmx_prometheus_httpserver-0.12.0-jar-with-dependencies.jar io.prometheus.jmx.WebServer 8997 /opt/exporter_zookeeper/zookeeper_exporter.yaml &

通过运行如下命令验证jmx_exporter是否可以正常工作。

1
curl http://<jmx_exporter所在机器的IP>:8997/metrics

zookeeper.yaml配置文件如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
rules:
  # replicated Zookeeper
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
    name: "zookeeper_$2"
    type: GAUGE
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
    name: "zookeeper_$3"
    type: GAUGE
    labels:
      replicaId: "$2"
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(Packets\\w+)"
    name: "zookeeper_$4"
    type: COUNTER
    labels:
      replicaId: "$2"
      memberType: "$3"
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
    name: "zookeeper_$4"
    type: GAUGE
    labels:
      replicaId: "$2"
      memberType: "$3"
  - pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
    name: "zookeeper_$4_$5"
    type: GAUGE
    labels:
      replicaId: "$2"
      memberType: "$3"
  # standalone Zookeeper
  - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+)><>(\\w+)"
    type: GAUGE
    name: "zookeeper_$2"
  - pattern: "org.apache.ZooKeeperService<name0=StandaloneServer_port(\\d+), name1=InMemoryDataTree><>(\\w+)"
    type: GAUGE
    name: "zookeeper_$2"