跳到主要内容

Prometheus 联邦集群

· 阅读需 2 分钟

Prometheus 联邦集群

Prometheus Server 环境:

192.168.15.100 #主节点

192.168.15.101 #联邦节点1

192.168.15.102 #联邦节点2

192.168.15.101 #node1,联邦节点1的目标采集服务器

192.168.15.101 #node2,联邦节点1的目标采集服务器

部署 prometheus server

Prometheus 主Server 和 prometheus 联邦 server 分别部署 prometheus

cd /apps
tar xvf prometheus-2.32.1.liunx-amd64.tar.gz

# 创建软连接
ln -sv /apps/prometheus-2.32.1.liunx-amd64 /apps/prometheus

cd /apps/prometheus
# 检测配置文件、检测 metrics 数据等
./promtool check config prometheus.yml

vim /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/apps/prometheus/
ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target

启动 prometheus 服务

systemctl daemon-reload
systemctl restart prometheus
systemctl enable prometheus

node_exporter 部署

下载解压二进制程序

cd /apps 
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xf node_exporter-1.3.1.linux-amd64.tar.gz

# 创建软连接
ln -sv /apps/node_exporter-1.3.1.linux-amd64 /apps/node_exporter

创建 node-exporter service 启动脚本

vim /etc/systemd/system/node-exporter.service

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/apps/node_exporter/node_exporter

[Install]
wantedBy=multi-user.target

启动 node exporter 服务

systemctl daemon-reload
systemctl restart node-exporter
systemctl enable node-exporter.service

配置联邦 server 监控 node_exporter

分别在联邦节点1 监控 node1,在联邦节点2 监控 node2

Prometheus 联邦节点1
vim /apps/prometheus/prometheus.yml

- job_name: 'prometheus-node'
static_configs:
- targets: ['192.168.15.101:9100'] #node_exporter1

# 重启 Prometheus 服务
systemctl restart prometheus.service
Prometheus 联邦节点2
vim /apps/prometheus/prometheus.yml

- job_name: 'prometheus-node'
static_configs:
- targets: ['192.168.15.102:9100'] #node_exporter2

# 重启 Prometheus 服务
systemctl restart prometheus.service

分别查看 prometheus target 中是否存在数据

prometheus server 采集联邦 server

  - job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'
static_configs:
- targets: ['localhost:9090']

- job_name: 'prometheus-federate-2.101'
scrape_interval: 10s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
- '{__name__=~"node.*"}'
static_configs:
- targets:
- '192.168.15.101:9090'

- job_name: 'prometheus-federate-2.102'
scrape_interval: 10s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
- '{__name__=~"node.*"}'
static_configs:
- targets:
- '192.168.15.102:9090'

验证 prometheus server

查看 192.168.15.100:9090/targets 是否存在联邦 prometheus 数据

验证指标数据

在 graph 界面查询 node_load1 查看是否存在数据

Pushgateway 采集数据

· 阅读需 4 分钟

Pushgateway 基本概念

Pushgateway 是采用被动推送的方式,而不是类似于prometheus server 主动连接 exporter获取监控数据

pushgateway 可以单独运行在一个节点,然后需要自定义监控脚本把需要监控的主动推送给pushgateway的API接口,然后 pushgateway 等待 prometheus server 抓取数据,即 pushgateway 本身没有任何抓取监控数据的功能,目前 pushgateway 只是被动的等待数据从客户端推送过来

--persistence.file="" #数据保存的文件,默认只保存在内存中

--persistence.interval=5m #数据持久化的间隔时间

部署 pushgateway

# docker 部署
docker run -d --name pushgateway -p 9091:9091 prom/pushgateway

# 二进制安装
cd /apps
wget https://github.com/prometheus/pushgateway/releases/download/v1.4.2/pushgateway-1.4.2.linux-amd64.tar.gz
tar zxf pushgateway-1.4.2.linux-amd64.tar.gz

# 编写 centos7 控制脚本

cat > /etc/systemd/system/pushgateway.service << EOF
[Unit]
Description=pushgateway
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/apps/pushgateway
ExecStart=/apps/pushgateway/pushgateway # 如果需要持久化pushgateway数据,可以加上--persistence.file="" 和--persistence.interval=5m 两个参数

[Install]
WantedBy=multi-user.target
EOF

prometheus 到 pushgateway 采集数据

验证 pushgateway

curl 192.168.15.100:9091/metrics

prometheus 配置数据采集

vim prometheus-cfg.yaml

  - job_name: 'pushgateway-monitor'
scrape_interval: 5s
static_configs:
- targets: ['192.168.15.100:9091']
honor_labels: true
  • honor_labels 控制 Prometheus 如何处理已经存在于已抓取数据中的标签与 Prometheus 将附加服务器端的标签之间的冲突("job"和"instance"标签,手动配置的目标标签以及服务发现实现生成的标签)
  • 如果 honor_labels 设置为 "true",则通过保留已抓取数据的标签值并忽略冲突的服务器端标签来解决标签冲突
  • 如果 honor_labels 设置为 "false",则通过将已抓取数据中的冲突标签重命名为 "exported_<original-label>" (例如 "exported_instance", "exported_job") 然后附加服务器端标签来解决标签冲突
kubectl apply -f prometheus-cfg.yaml
kubectl delete -f prometheus-deploy.yaml
kubectl applt -f prometheus-deploy.yaml

## 验证一下数据
查看 prometheus 在 target 中是否存在 pushgateway

测试从客户端推送单条数据

Push 数据到 Pushgateway 中,可以通过其提供的 API 标准接口来添加,默认的 URL 地址为:

http://<ip>:9091/metrics/job/<JOBNAME>/<LABEL_NAME>/<LABEL_VALUE>

其中 <JOBNAME> 是必填项,为 job 标签值,后边可以跟任何数量的标签对,一般我们会添加一个 instance/<INSTANCE_NAME> 实例名称标签,来方便区分各个指标

推送一个 job 名称为 mytest_job ,key 为 mytest_metric 值为 2022

echo "mytest_metric 2022" | curl --data-binary @- http://192.168.3.100:9091/metrics/job/mytest_job

echo "mytest_metric 2333" | curl --data-binary @- http://192.168.3.100:9091/metrics/job/mytest_job
pushgateway 验证数据
# 192.168.15.100/#  在 web 界面查看 

除了 mytest_metric 外,同时还新增了 push_time_seconds 和 push_failure_time_seconds 两个标签,这两个是 Pushgateway 自动生成的指标,分别用于记录指标数据的成功上传时间和失败上传时间

# 查看 192.168.15.100:9091/metrics
搜索关键字:mytest ,会发现有此指标
prometheus server 验证数据

在 Graph 页面查询 mytest_metric 可以看到图标即是有数据的

测试从客户端推送多条数据

推送多条数据
cat <<EOF | curl --date-binary @- http://192.168.15.100:9091/metrics/job/test_job/instance/192.168.15.101
#TYPE node_memory_usage gauge
node_memory_usage 4311744512
# TYPE memory_total gauge
node_memory_total 103481868288
EOF
pushgateway 验证数据
# 192.168.15.100/#  在 web 界面查看 
prometheus server

在 Graph 页面查询 node_memory_total 可以看到图标即是有数据的

自定义收集数据

基于自定义脚本实现数据的收集和推送

自定义脚本 vim mem_monitor.sh

#!/bin/bash

total_memory=$(free|awk '/Mem/{print $2}')
used_memory=$(free |awk '/Mem/{print $3}')

job_name="custom_memory_monitor"
instance_name=`ifconfig eth0 | grep -w inet | awk '{print $2}'`
pushgateway_server="htpp://192.168.15.100:9091/metrics/job"

cat <<EOF | curl --date-binary @- ${pushgateway_server}/${job_name}/instance/${instance_name}
#TYPE custom_memory_total gauge
custom_memory_total $total_memory
#TYPE custom_memory_used gauge
custom_memory_used $used_memory
EOF

分别在不同主机执行脚本,验证指标数据收集和推送

bash mem_monitor.sh
Pushgateway 验证数据
192.168.15.100:9091/#  查看数据
prometheus 验证数据

在 Graph 页面查询 custom_memory_total 可以看到图标即是有数据的

删除数据

先对一个组写入多个 instance 的数据

cat <<EOF | curl --date-binary @- http://192.168.15.100:9091/metrics/job/test_job/instance/192.168.15.101
#TYPE node_memory_usage gauge
node_memory_usage 4311744512
# TYPE memory_total gauge
node_memory_total 103481868288
EOF

cat <<EOF | curl --date-binary @- http://192.168.15.100:9091/metrics/job/test_job/instance/192.168.15.102
#TYPE node_memory_usage gauge
node_memory_usage 4311744512
# TYPE memory_total gauge
node_memory_total 103481868288
EOF
通过 API 删除指定组内指定实例的数据
curl -X DELETE http://192.168.15.100:9091/metrics/job/test_job/instance/192.168.15.101

# web浏览 192.168.15.100:9091/#
通过 web 界面删除

直接在数据右边点 Delete Group

二进制安装 Prome 生态

· 阅读需 7 分钟

Prometheus 二进制安装

下载解压二进制程序

mkdir /apps  && cd /apps

#wget https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz
wget https://github.com/prometheus/prometheus/releases/download/v2.35.0/prometheus-2.35.0.linux-amd64.tar.gz
tar xf prometheus-2.35.0.linux-amd64.tar.gz

# 创建软连接
ln -sv prometheus-2.35.0.linux-amd64 /apps/prometheus

# 检查配置文件
./promtool check config prometheus.yml

创建 prometheus service 启动脚本

vim /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/apps/prometheus/
ExecStart=/apps/prometheus/prometheus --config.file=/apps/prometheus/prometheus.yml

[Install]
WantedBy=multi-user.target

启动 prometheus 服务

systemctl daemon-reload
systemctl restart prometheus
systemctl enable prometheus

node export 二进制安装

下载解压二进制程序

cd /apps 
wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz
tar xf node_exporter-1.3.1.linux-amd64.tar.gz

# 创建软连接
ln -sv /apps/node_exporter-1.3.1.linux-amd64 /apps/node_exporter

创建 node-exporter service 启动脚本

vim /etc/systemd/system/node-exporter.service

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
ExecStart=/apps/node_exporter/node_exporter

[Install]
WantedBy=multi-user.target

启动 node exporter 服务

systemctl daemon-reload
systemctl restart node-exporter
systemctl enable node-exporter.service

添加node节点数据收集

vim /apps/prometheus/prometheus.yml

  - job_name: 'prometheus-node'
static_configs:
- targets: ['192.168.15.100:9100']

重启服务

systemctl restart prometheus.service

Alertmanager 二进制安装

下载解压二进制程序

cd /apps 
wget https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz
tar xf alertmanager-0.24.0.linux-amd64.tar.gz

# 创建软连接
ln -sv /apps/alertmanager-0.24.0.linux-amd64 /apps/alertmanager

创建 alertmanager service 启动脚本

vim /etc/systemd/system/alertmanager.service

[Unit]
Description=alertmanager
Documentation=https://github.com/prometheus/alertmanager
After=network.target
[Service]
Type=simple
User=root
ExecStart=/apps/alertmanager/alertmanager --config.file=/apps/alertmanager/alertmanager.yml
Restart=on-failure
[Install]
WantedBy=multi-user.target

编辑配置文件 (邮件)

vim /apps/alertmanager/alertmanager.yml

global:
resolve_timeout: 5m
smtp_from: 'xxxx@qq.com'
smtp_smarthost: 'smtp.qq.com:465'
smtp_auth_username: 'xxxx@qq.com'
smtp_auth_password: 'uuxxxxdvnxzbiaf'
smtp_require_tls: false
smtp_hello: '@qq.com'
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 2m
repeat_interval: 5m
receiver: 'web.hook'
# #receiver: 'default-receiver' #其他的告警发送给default-receiver
# routes: #将critical的报警发送给myalertname
# - reciver: myalertname
# group_wait: 10s
receivers:
- name: 'web.hook'
# webhook_configs:
# - url: 'http://127.0.0.1:5001/'
email_configs:
- to: 'xxxx@qq.com'
inhibit_rules:
- source_match: #源匹配级别,当匹配成功发出通知,但是其他'alertname','dev','instance'产生的warning级别的告警通知将被抑制
severity: 'critical' #报警的事件级别
target_match:
severity: 'warning' #匹配目标为新产生的目标告警为'warning' 将被抑制
equal: ['alertname', 'dev', 'instance']

编辑配置文件 (企业微信)

vim /apps/alertmanager/alertmanager.yml

global:
resolve_timeout: 5m

route:
group_by: ['alertname']
group_wait: 10s # 初次发送告警延时
group_interval: 10s # 距离第一次发送告警,等待多久再次发送告警
repeat_interval: 60m # 告警重发时间
receiver: 'wechat'

receivers:
- name: 'wechat'
webhook_configs:
- url: 'http://172.20.254.138:9080/wechatbot' # adapter
send_resolved: true

inhibit_rules:
- source_match: #源匹配级别,当匹配成功发出通知,但是其他'alertname','dev','instance'产生的warning级别的告警通知将被抑制
severity: 'critical' #报警的事件级别
target_match:
severity: 'warning' #匹配目标为新产生的目标告警为'warning' 将被抑制
equal: ['alertname', 'dev', 'instance']

启动 Alertmanager服务

systemctl daemon-reload
systemctl start alertmanager
systemctl enable alertmanager

安装 Adapter 适配器

git clone https://github.com/lckei/prometheus-wechatbot-webhook.git
编辑 Dockerfile
cat Dockerfile
FROM alpine:latest
LABEL maintainer="kei"
ENV VERSION 1.0
WORKDIR /apps
ADD src/app /apps/app
RUN chmod +x /apps/app
#ADD src/wechatbot.tmpl /apps/wechatbot.tmpl
ADD src/wechatbot2.tmpl /apps/wechatbot.tmpl
EXPOSE 9080
CMD ["/apps/app"]
编辑告警模版文件
{{ define "wechatbot.url.api" }}https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=2730f396-a070-4618-aedb-d290ad132ffc{{end}}
{{- if gt (len .Alerts.Firing) 0 -}}
{{- range $index, $alert := .Alerts -}}
{{- if eq $index 0 }}
==========异常告警==========
告警类型: {{ $alert.Labels.alertname }}
告警级别: {{ $alert.Labels.severity }}
告警详情: {{ $alert.Annotations.description}};{{$alert.Annotations.summary}}
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{- if gt (len $alert.Labels.instance) 0 }}
实例信息: {{ $alert.Labels.instance }}
{{- end }}
{{- if gt (len $alert.Labels.namespace) 0 }}
命名空间: {{ $alert.Labels.namespace }}
{{- end }}
{{- if gt (len $alert.Labels.node) 0 }}
节点信息: {{ $alert.Labels.node }}
{{- end }}
{{- if gt (len $alert.Labels.pod) 0 }}
实例名称: {{ $alert.Labels.pod }}
{{- end }}
============END============
{{- end }}
{{- end }}
{{- end }}
{{- if gt (len .Alerts.Resolved) 0 -}}
{{- range $index, $alert := .Alerts -}}
{{- if eq $index 0 }}
==========异常恢复==========
告警类型: {{ $alert.Labels.alertname }}
告警级别: {{ $alert.Labels.severity }}
告警详情: {{ $alert.Annotations.description}};{{$alert.Annotations.summary}}
故障时间: {{ ($alert.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
恢复时间: {{ ($alert.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
{{- if gt (len $alert.Labels.instance) 0 }}
实例信息: {{ $alert.Labels.instance }}
{{- end }}
{{- if gt (len $alert.Labels.namespace) 0 }}
命名空间: {{ $alert.Labels.namespace }}
{{- end }}
{{- if gt (len $alert.Labels.node) 0 }}
节点信息: {{ $alert.Labels.node }}
{{- end }}
{{- if gt (len $alert.Labels.pod) 0 }}
实例名称: {{ $alert.Labels.pod }}
{{- end }}
============END============
{{- end }}
{{- end }}
{{- end }}
构建 Adapter
docker build -t wechatbot:v1 .
运行 Adapter
docker run -d --name wechatbot --restart=always \
-v /etc/localtime:/etc/localtime \
-v src/wechatbot.tmpl:/apps/wechatbot.tmpl \
-p 9080:9080 wechatbot:v1

rpm 安装 grafana

安装 mysql

# 下载mysql源安装包
wget http://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm

# 安装mysql源
yum localinstall mysql57-community-release-el7-8.noarch.rpm -y

# 检查mysql源是否安装成功
yum repolist enabled | grep "mysql.*-community.*"

# 安装MySQL (5.7需绕过验证)

yum install mysql-community-server -y --nogpgcheck


# 3、启动MySQL服务
systemctl start mysqld

# 查看MySQL的启动状态
systemctl status mysqld


#4、开机启动
systemctl enable mysqld
systemctl daemon-reload

# 5、修改root本地登录密码
# mysql安装完成之后,在/var/log/mysqld.log文件中给root生成了一个默认密码。通过下面的方式找到root默认密码,然后登录mysql进行修改:
grep 'temporary password' /var/log/mysqld.log
mysql -uroot -p

# mysql5.7默认安装了密码安全检查插件(validate_password),默认密码检查策略要求密码必须包含:大小写字母、数字和特殊符号,
# 并且长度不能少于8位。否则会提示ERROR 1819 (HY000): Your password does not satisfy the current policy requirements错误

# 如果不需要密码策略,添加my.cnf文件中添加如下配置禁用即可:
# 配置默认编码为utf8
# 关闭客户端dns反解

echo -e "validate_password = off\ncharacter_set_server=utf8\ninit_connect='SET NAMES utf8'\nskip-name-resolve\n" >> /etc/my.cnf
systemctl restart mysqld

mysql -uroot -p

## 授权
alter user 'root'@'localhost' identified by '123123';

grant all privileges on *.* to root@'%' identified by '123123' with grant option;
flush privileges;

下载安装包

cd /apps
# 地址 https://grafana.com/grafana/download
wget -O /opt/tgzs/grafana-7.5.1-1.x86_64.rpm https://dl.grafana.com/oss/release/grafana-7.5.1-1.x86_64.rpm
yum install grafana-7.5.1-1.x86_64.rpm

# 下载grafana8
wget https://dl.grafana.com/oss/release/grafana-8.5.1-1.x86_64.rpm
yum install -y grafana-8.5.1-1.x86_64.rpm

# 下载grafana10 (建议安装10及以上的,UI更漂亮~)
wget https://dl.grafana.com/oss/release/grafana-10.2.1-1.x86_64.rpm
yum install -y grafana-10.2.1-1.x86_64.rpm

在 mysql 中创建数据库

CREATE DATABASE IF NOT EXISTS grafana DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

修改配置文件

vim /etc/grafana/grafana.ini

type = mysql
host = 127.0.0.1:3306
name = grafana
user = root
password = 123456

启动服务

systemctl start grafana-server
systemctl enable grafana-server
systemctl status grafana-server

# 查看日志
tail -f /var/log/grafana/grafana.log