prometheus应用监控（八）黑盒监控blackbox_exporter

1、介绍

黑盒导出器允许通过 HTTP、HTTPS、DNS、TCP、ICMP 和 gRPC 对端点进行黑盒探测。

• HTTP/HTTPS 测试
  定义 Request Header 信息
  判断 Http status / Http Respones Header / Http Body 内容

• TCP 测试
  探测业务端口存活

• ICMP 测试
  探测主机存活、网络抖动监控

• POST 测试
  探测接口联通性

• GRPC测试
  探测接口联通性

• SSL 证书过期时间

2、配置监控

说明：由于我们的prometheus是通过kube-prometheus的方式部署的，所以添加黑盒监控可以通过additional的方式

修改prometheus-additional.yaml文件

- job_name: "check-tcp"
  metrics_path: /probe
  params:
    module: [tcp_connect]
  static_configs:
    - targets:
      - 127.0.0.1:9115
      - 192.168.96.51:22
      - 192.168.96.53:22
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:19115

- job_name: "check-web-api"
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
      - https://www.baidu.com
      - http://www.baidu.com
      labels:
        group: web-api
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:19115

- job_name: "check-icmp"
  metrics_path: /probe
  params:
    module: [icmp]
  static_configs:
    - targets:
      - 192.168.96.51
      - 192.168.96.53
      - www.baidu.com
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:19115

- job_name: "check-post-api"
  metrics_path: /probe
  params:
    module: [http_post_2xx]
  static_configs:
    - targets:
      - https://oapi.dingtalk.com
      labels:
        group: check-post-api
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter:19115

因为blackbox_exporter的默认配置文件没有添加ICMP模块，所以需要添加一下
更新配置文件blackbox-exporter-configuration，添加如下内容：

"icmp":
  "prober": "icmp"
  "timeout": "5s"
  "icmp":
    "preferred_ip_protocol": "ip4"
    "source_ip_address": "127.0.0.1"

更新部署prometheus-additional.yaml，使prometheus自动加载最新的配置

kubectl delete secret additional-scrape-configs -n monitoring
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n monitoring

查看prometheus中添加的黑盒监控

3、告警规则

通用告警规则

groups:
- name: blackbox_network_stats
  rules:
  - alert: blackbox_network_stats
    expr: probe_success == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "接口/主机/端口连通异常告警"
      description: "接口/主机/端口 {{ $labels.instance }}  连通异常"

SSL告警规则

groups:
- name: check_ssl_status
  rules:
  - alert: "ssl证书过期警告"
    expr: (probe_ssl_earliest_cert_expiry - time())/86400 <30
    for: 4h
    labels:
      severity: critical
    annotations:
      description: '域名{{$labels.instance}}的证书还有{{ printf "%.1f" $value }}天就过期了,请尽快更新证书'
      summary: "ssl证书过期警告"

4、数据展示

在grafana中导入16292模板，生成的图表如下