• 微信公众号:美女很有趣。 工作之余,放松一下,关注即送10G+美女照片!

[prometheus]配置alertmanager和钉钉告警

开发技术 开发技术 2周前 (04-29) 8次浏览

目录
  • prometheus发起告警的逻辑
  • 节点
  • 配置alertmanager
  • 配置钉钉告警插件
  • 配置supervisor守护进程
  • 关联prometheus和alertmanager

prometheus发起告警的逻辑

  1. 假设A服务器和prometheus服务器断联,且已经超过一分钟,匹配上监测存活的告警规则
  2. Prometheus向alertmanager报信,A服务器断联
  3. alertmanager调用钉钉告警插件,发起告警
  4. 钉钉机器人在群里发消息。

节点

  • 172.50.13.101:prometheus server
  • 172.50.13.102:alertmanager和钉钉告警插件

配置alertmanager

首先肯定要去官方下载alertmanager。GitHub – prometheus/alertmanager: Prometheus Alertmanager

安装很简单,解压缩就行了。

alertmanager.yml文件内容:(receiver中的url应该为钉钉告警插件的url)

global:
  resolve_timeout: 5m

route:
  group_by: [alertname]
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 2h
  receiver: webhook
receivers:
- name: webhook
  webhook_configs:
  - url: 'http://172.50.13.102:8060/dingtalk/webhook1/send'
    send_resolved: true

配置钉钉告警插件

插件下载地址:Releases · timonwong/prometheus-webhook-dingtalk · GitHub

只是能用的话,解压缩就行了,不需要修改配置文件。

配置supervisor守护进程

vim /etc/supervisord.d/prometheus.ini

[program:alertmanager]
command=/usr/local/prometheus/alertmanager/alertmanager --storage.path="/home/data/prometheus/alertmanager/" --web.listen-address=":18081" --config.file=/usr/local/prometheus/alertmanager/alertmanager.yml --data.retention=120h --web.external-url=http://172.50.13.102:18081
directory=/usr/local/prometheus/alertmanager
autostart=true
startsecs=10
startretries=3
autorestart=true

[program:dingtalk]
command=/usr/local/prometheus/dingtalk/prometheus-webhook-dingtalk --ding.profile="webhook1=https://oapi.dingtalk.com/robot/send?access_token=xxxx"
directory=/usr/local/prometheus/dingtalk
autostart=true
startsecs=10
startretries=3
autorestart=true

启动参数说明:

  • alertmanager:
    • storage.path:数据存储路径
    • web.listen.addreess:监听端口
    • config.file:alertmanager.yml文件的路径
    • data.retention:数据存储时间
    • web.external-url:启用web页面并配置地址
  • dingtalk:
    • ding.profile:注意webhook1后面替换为实际钉钉机器人的webhook

关联prometheus和alertmanager

prometheus.yml中主要的配置内容:

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['172.50.13.102:18081']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
  - "alertrules/*_rules.yml"

targets为alertmanager的地址。rule_files为告警规则文件,此处为同级目录中alertrules目录下所有带“_rules.yml”后缀的文件。

监测存活的告警规则:

  • alertrules/live_rules.yml
groups:
- name: UP
  rules:
  - alert : node
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.job }} {{ $labels.instance }} 节点断联已超过1分钟!"
      summary: "{{ $labels.instance }} down "

监测负载的告警规则:(监测内存,磁盘占用率和CPU使用率)

  • alertrules/perf_rules.yml
groups:
- name: mem_product 
  rules:
  - alert : mem_product
    expr: (1 - (node_memory_MemAvailable_bytes{job="生产服务器"} / (node_memory_MemTotal_bytes{job="生产服务器"})))* 100  > 90
    for: 5m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.job }} {{ $labels.instance }} 节点的内存使用率超过90%已持续5分钟!"
      summary: "{{ $labels.instance }} 内存使用率超标! "


- name: disk
  rules:
  - alert : disk
    expr: (node_filesystem_size_bytes{fstype=~"ext.*|xfs",mountpoint !~".*pod.*"}-node_filesystem_free_bytes{fstype=~"ext.*|xfs",mountpoint !~".*pod.*"}) *100/(node_filesystem_avail_bytes {fstype=~"ext.*|xfs",mountpoint !~".*pod.*"}+(node_filesystem_size_bytes{fstype=~"ext.*|xfs",mountpoint !~".*pod.*"}-node_filesystem_free_bytes{fstype=~"ext.*|xfs",mountpoint !~".*pod.*"})) > 95
    for: 5m
    labels:
      severity: warning
    annotations:
      description: "{{ $labels.job }} {{ $labels.instance }} 节点的硬盘使用率超过95%已持续5分钟!"
      summary: "{{ $labels.instance }} 硬盘空间使用率超标! "

- name: cpu
  rules:
  - alert : cpu
    expr: ((1- sum(increase(node_cpu_seconds_total{mode="idle"}[5m])) by (instance)/sum(increase(node_cpu_seconds_total[5m])) by (instance)) * 100) >  70
    for: 5m
    labels:
      severity: warning
    annotations:
      description: "{{ $labels.job }} {{ $labels.instance }} 节点的CPU使用率超过70%已持续5分钟!"
      summary: "{{ $labels.instance }} CPU使用率超标!"

程序员灯塔
转载请注明原文链接:[prometheus]配置alertmanager和钉钉告警
喜欢 (0)