windows单节点验证victoriametrics结合AlertManger实现告警推送webhook

BUG弄潮儿

发布于 2025-03-24 13:54:49

6400

代码可运行

文章被收录于专栏：JAVA乐园JAVA乐园

运行总次数：0

代码可运行

安装victoriametrics

https://docs.victoriametrics.com/single-server-victoriametrics/

下载地址

https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.113.0

找到victoria-metrics-windows-amd64-v1.113.0.zip

https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.113.0/victoria-metrics-windows-amd64-v1.113.0.zip

解压并在cmd里启动

victoria-metrics-windows-amd64-prod.exe

启动结果

......
2025-03-21T07:17:44.495Z        info    VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:156  loaded rollupResult cache from "victoria-metrics-data/cache/rollupResult" in 0.010 seconds; entriesCount: 0, sizeBytes: 0
2025-03-21T07:17:44.495Z        info    VictoriaMetrics/app/vmselect/prometheus/prometheus.go:1297      limiting -search.maxUniqueTimeseries to 8547304 according to -search.maxConcurrentRequests=16 and remaining memory=27351374234 bytes. To increase the limit, reduce -search.maxConcurrentRequests or increase memory available to the process.
2025-03-21T07:17:44.496Z        info    VictoriaMetrics/app/victoria-metrics/main.go:105        started VictoriaMetrics in 0.203 seconds
2025-03-21T07:17:44.500Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:169        started server at http://0.0.0.0:8428/
2025-03-21T07:17:44.502Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:171        pprof handlers are exposed at http://0.0.0.0:8428/debug/pprof/

访问

http://127.0.0.1:8428

安装vmutils-windows

vmutils-windows包里包含众多VictoriaMetrics组件

https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.113.0/vmutils-windows-amd64-v1.113.0.zip

解压vmutils-windows-amd64-v1.113.0.zip

启动vmagent

https://docs.victoriametrics.com/vmagent/

启动命令

vmagent-windows-amd64-prod.exe -promscrape.config=prometheus.yml -remoteWrite.url=http://127.0.0.1:8428/api/v1/write

安装victoriametrics

https://docs.victoriametrics.com/single-server-victoriametrics/

下载地址

https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.113.0

找到victoria-metrics-windows-amd64-v1.113.0.zip

https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.113.0/victoria-metrics-windows-amd64-v1.113.0.zip

解压并在cmd里启动

victoria-metrics-windows-amd64-prod.exe

启动结果

......
2025-03-21T07:17:44.495Z        info    VictoriaMetrics/app/vmselect/promql/rollup_result_cache.go:156  loaded rollupResult cache from "victoria-metrics-data/cache/rollupResult" in 0.010 seconds; entriesCount: 0, sizeBytes: 0
2025-03-21T07:17:44.495Z        info    VictoriaMetrics/app/vmselect/prometheus/prometheus.go:1297      limiting -search.maxUniqueTimeseries to 8547304 according to -search.maxConcurrentRequests=16 and remaining memory=27351374234 bytes. To increase the limit, reduce -search.maxConcurrentRequests or increase memory available to the process.
2025-03-21T07:17:44.496Z        info    VictoriaMetrics/app/victoria-metrics/main.go:105        started VictoriaMetrics in 0.203 seconds
2025-03-21T07:17:44.500Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:169        started server at http://0.0.0.0:8428/
2025-03-21T07:17:44.502Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:171        pprof handlers are exposed at http://0.0.0.0:8428/debug/pprof/

访问

http://127.0.0.1:8428

安装vmutils-windows

vmutils-windows包里包含众多VictoriaMetrics组件

https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.113.0/vmutils-windows-amd64-v1.113.0.zip

解压vmutils-windows-amd64-v1.113.0.zip

启动vmagent

https://docs.victoriametrics.com/vmagent/

启动命令

vmagent-windows-amd64-prod.exe -promscrape.config=prometheus.yml -remoteWrite.url=http://127.0.0.1:8428/api/v1/write

其中prometheus.yml为prometheus配置文件；http://127.0.0.1:8428/api/v1/write为victoriametrics服务地址

prometheus.yml配置文件内容如下

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "victoriametrics"
    static_configs:
      - targets: ["127.0.0.1:8429"]
  - job_name: "zookeeper"
    static_configs:
      - targets: ["127.0.0.1:7000"]
  - job_name: "mysql"
    static_configs:
      - targets: ["127.0.0.1:9104"]
  - job_name: "windows"
    static_configs:
      - targets: ["127.0.0.1:9182"]

启动日志如下

2025-03-21T07:32:48.747Z        info    VictoriaMetrics/app/vmagent/main.go:169 started vmagent in 0.380 seconds
2025-03-21T07:32:48.747Z        info    VictoriaMetrics/lib/promscrape/scraper.go:118   reading scrape configs from "prometheus.yml"
2025-03-21T07:32:48.748Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:169        started server at http://0.0.0.0:8429/
2025-03-21T07:32:48.749Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:171        pprof handlers are exposed at http://0.0.0.0:8429/debug/pprof/
2025-03-21T07:32:48.753Z        info    VictoriaMetrics/lib/promscrape/config.go:149    starting service discovery routines...
2025-03-21T07:32:48.754Z        info    VictoriaMetrics/lib/promscrape/config.go:155    started 5 service discovery routines in 0.001 seconds
2025-03-21T07:32:48.754Z        info    VictoriaMetrics/lib/promscrape/scraper.go:439   static_configs: added targets: 5, removed targets: 0; total targets: 5

启动vmalert

https://docs.victoriametrics.com/vmalert/

启动命令

vmalert-windows-amd64-prod.exe -rule=rule-first.yml -datasource.url=http://127.0.0.1:8428 -notifier.url=http://127.0.0.1:9093 -notifier.url=http://127.0.0.1:9093 -remoteWrite.url=http://127.0.0.1:8428 -remoteRead.url=http://127.0.0.1:8428 -external.label=cluster=east-1 -external.label=replica=a

其中rule-first.yml为规则文件；http://127.0.0.1:8428为victoriametrics服务地址；http://127.0.0.1:9093为AlertManger地址。

rule-first.yml规则配置文件内容如下

groups:
  - name: InstanceDown_Rule
    rules:
      - alert: InstanceDown  # 告警名称
        expr: up == 0        # 告警条件
        for: 30s              # 告警触发前需要持续满足条件的时间
        labels:
          severity: critical # 告警的严重程度
          instance: "{{$labels.instance}}"
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "Instance {{ $labels.instance }} has been down for more than 30 seconds."

启动日志如下

......
2025-03-21T07:39:41.952Z        info    VictoriaMetrics/app/vmalert/config/log/logger.go:52     found 1 files to read from "Local FS{MatchPattern: \"rule-first.yml\"}"
2025-03-21T07:39:41.955Z        info    VictoriaMetrics/app/vmalert/config/log/logger.go:52     finished reading 1 files in 889.8µs from "Local FS{MatchPattern: \"rule-first.yml\"}"
2025-03-21T07:39:41.956Z        info    VictoriaMetrics/app/vmalert/rule/group.go:497   group "InstanceDown_Rule" will start in 8.357938405s; interval=1m0s; eval_offset=<nil>; concurrency=1
2025-03-21T07:39:41.959Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:169        started server at http://0.0.0.0:8880/
2025-03-21T07:39:41.960Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:171        pprof handlers are exposed at http://0.0.0.0:8880/debug/pprof/

访问地址

http://127.0.0.1:8880/

部署AlertManger

下载AlertManger

https://github.com/prometheus/alertmanager/releases

下载地址

https://github.com/prometheus/alertmanager/releases/download/v0.28.1/alertmanager-0.28.1.windows-amd64.zip

解压并启动，启动命令如下

alertmanager.exe --config.file=alertmanager.yml --log.level=debug

其中alertmanager.yml为alertmanager的配置文件，配置文件内容如下：

route:
#  group_by: ['alertname', 'instance', 'job']
  group_by: [...]
  group_wait: 10s
  group_interval: 20s
  repeat_interval: 30s
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/user/alert'
        send_resolved: true
#inhibit_rules:
#  - source_match:
#      severity: 'critical'
#    target_match:
#      severity: 'warning'
#    equal: ['alertname', 'instance', 'job']
global:
  resolve_timeout: 5m

http://127.0.0.1:5001/user/alert配置推送到应用服务的服务端点

启动日志如下

......
ts=2025-03-21T07:49:26.950Z caller=main.go:348 level=debug externalURL=http://P7507787A244:9093
ts=2025-03-21T07:49:26.951Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=alertmanager.yml
ts=2025-03-21T07:49:26.954Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=alertmanager.yml
ts=2025-03-21T07:49:26.958Z caller=main.go:495 level=debug routePrefix=/
ts=2025-03-21T07:49:26.960Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093
ts=2025-03-21T07:49:26.962Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093

访问地址

http://127.0.0.1:9093

其中prometheus.yml为prometheus配置文件；http://127.0.0.1:8428/api/v1/write为victoriametrics服务地址

prometheus.yml配置文件内容如下

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "victoriametrics"
    static_configs:
      - targets: ["127.0.0.1:8429"]
  - job_name: "zookeeper"
    static_configs:
      - targets: ["127.0.0.1:7000"]
  - job_name: "mysql"
    static_configs:
      - targets: ["127.0.0.1:9104"]
  - job_name: "windows"
    static_configs:
      - targets: ["127.0.0.1:9182"]

启动日志如下

2025-03-21T07:32:48.747Z        info    VictoriaMetrics/app/vmagent/main.go:169 started vmagent in 0.380 seconds
2025-03-21T07:32:48.747Z        info    VictoriaMetrics/lib/promscrape/scraper.go:118   reading scrape configs from "prometheus.yml"
2025-03-21T07:32:48.748Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:169        started server at http://0.0.0.0:8429/
2025-03-21T07:32:48.749Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:171        pprof handlers are exposed at http://0.0.0.0:8429/debug/pprof/
2025-03-21T07:32:48.753Z        info    VictoriaMetrics/lib/promscrape/config.go:149    starting service discovery routines...
2025-03-21T07:32:48.754Z        info    VictoriaMetrics/lib/promscrape/config.go:155    started 5 service discovery routines in 0.001 seconds
2025-03-21T07:32:48.754Z        info    VictoriaMetrics/lib/promscrape/scraper.go:439   static_configs: added targets: 5, removed targets: 0; total targets: 5

启动vmalert

https://docs.victoriametrics.com/vmalert/

启动命令

vmalert-windows-amd64-prod.exe -rule=rule-first.yml -datasource.url=http://127.0.0.1:8428 -notifier.url=http://127.0.0.1:9093 -notifier.url=http://127.0.0.1:9093 -remoteWrite.url=http://127.0.0.1:8428 -remoteRead.url=http://127.0.0.1:8428 -external.label=cluster=east-1 -external.label=replica=a

其中rule-first.yml为规则文件；http://127.0.0.1:8428为victoriametrics服务地址；http://127.0.0.1:9093为AlertManger地址。

rule-first.yml规则配置文件内容如下

groups:
  - name: InstanceDown_Rule
    rules:
      - alert: InstanceDown  # 告警名称
        expr: up == 0        # 告警条件
        for: 30s              # 告警触发前需要持续满足条件的时间
        labels:
          severity: critical # 告警的严重程度
          instance: "{{$labels.instance}}"
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "Instance {{ $labels.instance }} has been down for more than 30 seconds."

启动日志如下

......
2025-03-21T07:39:41.952Z        info    VictoriaMetrics/app/vmalert/config/log/logger.go:52     found 1 files to read from "Local FS{MatchPattern: \"rule-first.yml\"}"
2025-03-21T07:39:41.955Z        info    VictoriaMetrics/app/vmalert/config/log/logger.go:52     finished reading 1 files in 889.8µs from "Local FS{MatchPattern: \"rule-first.yml\"}"
2025-03-21T07:39:41.956Z        info    VictoriaMetrics/app/vmalert/rule/group.go:497   group "InstanceDown_Rule" will start in 8.357938405s; interval=1m0s; eval_offset=<nil>; concurrency=1
2025-03-21T07:39:41.959Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:169        started server at http://0.0.0.0:8880/
2025-03-21T07:39:41.960Z        info    VictoriaMetrics/lib/httpserver/httpserver.go:171        pprof handlers are exposed at http://0.0.0.0:8880/debug/pprof/

访问地址

http://127.0.0.1:8880/

部署AlertManger

下载AlertManger

https://github.com/prometheus/alertmanager/releases

下载地址

https://github.com/prometheus/alertmanager/releases/download/v0.28.1/alertmanager-0.28.1.windows-amd64.zip

解压并启动，启动命令如下

alertmanager.exe --config.file=alertmanager.yml --log.level=debug

其中alertmanager.yml为alertmanager的配置文件，配置文件内容如下：

route:
#  group_by: ['alertname', 'instance', 'job']
  group_by: [...]
  group_wait: 10s
  group_interval: 20s
  repeat_interval: 30s
  receiver: 'web.hook'
receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/user/alert'
        send_resolved: true
#inhibit_rules:
#  - source_match:
#      severity: 'critical'
#    target_match:
#      severity: 'warning'
#    equal: ['alertname', 'instance', 'job']
global:
  resolve_timeout: 5m

http://127.0.0.1:5001/user/alert配置推送到应用服务的服务端点

启动日志如下

......
ts=2025-03-21T07:49:26.950Z caller=main.go:348 level=debug externalURL=http://P7507787A244:9093
ts=2025-03-21T07:49:26.951Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=alertmanager.yml
ts=2025-03-21T07:49:26.954Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=alertmanager.yml
ts=2025-03-21T07:49:26.958Z caller=main.go:495 level=debug routePrefix=/
ts=2025-03-21T07:49:26.960Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9093
ts=2025-03-21T07:49:26.962Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9093

访问地址