Prometheus victoria-metrics 存储

Prometheus 本地存储

默认情况下，prometheus 将采集到的数据存储在本地的 TSDB 数据库中，路径默认为 prometheus 安装目录的 data 目录，数据写入过程为先把数据写入 wal 日志并放在内存，然后 2 小时后将内存数据保存至一个新的 block 块，同时再把新采集的数据写入内存并在 2 小时后再保存至一个新的 block 块，以此类推

每个 block 为一个 data 目录中以 01 开头的存储目录，比如说：

[root@yuan ~]# ls -l /apps/prometheus/data/
total 20
drwxr-xr-x 3 root root    68 May  8 13:00 01G2H0KYPSE8MZATVBBG9KPJME
drwxr-xr-x 3 root root    68 May 10 16:08 01G2PG73ZNQ0G1ZGD0EGDV2VMD
drwxr-xr-x 3 root root    68 May 10 16:08 01G2PG740MHR9JC7V772CYARAR
drwxr-xr-x 3 root root    68 May 10 16:08 01G2PG742VRGMXC2Z784YNKW29

block 的特征
block 会压缩、合并历史数据块，以及删除过期的块，随着压缩、合并，block 的数量会减少，在压缩过程中会发生三件事：

定期执行压缩

合并小的 block 到大的 block

清理过期的块

每个块有 4 部分组成

[root@yuan ~]# tree /apps/prometheus/data/01G2H0KYPSE8MZATVBBG9KPJME/
/apps/prometheus/data/01G2H0KYPSE8MZATVBBG9KPJME/
├── chunks
│   └── 000001 #数据目录，每个大小为 512MB 超过会被切分为多个
├── index  #索引文件，记录存储的数据的索引信息，通过文件内的几个表来查找时序数据
├── meta.json  #block 元数据信息，包含了样本数、采集数据数据的起始时间、压缩历史
└── tombstones  #逻辑数据，主要记载删除记录和标记要删除的内容，删除标记，可在查询块时排除样本

本地存储配置参数

--config.file="prometheus.yml"  #指定配置文件
--web.listen-address="0.0.0.0:9090"  #指定监听地址
--storage.tsdb.path="data/"  #指定数据存储目录

--storage.tsdb.retention.size=B,KB,MB,TB,PB,EB  #指定 chunk 大小，默认 512MB
--storage.tsdb.retention.time=  #数据保存时长，默认15天

--query.timeout=2m  #最大查询超时时间
--query.max-concurrency=20  #最大查询并发量

--web.read-timeout=5m  #最大空闲超时时间
--web.max-connections=512  #最大并发连接数
--web.enable-lifecycle  #启用 API 动态加载配置功能

远端存储 victoriametrics

https://github.com/VictoriaMetrics/VictoriaMetrics

https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html

单机版部署

# 下载安装包
wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.71.0/victoria-metrics-arm-v1.71.0.tar.gz

tar xvf victoria-metrics-arm-v1.71.0.tar.gz

参数：
-httpListenAddr=0.0.0.0:8428 #监听地址及端口

-storageDataPath #VictoriaMetrics 将所有数据存储在此目录中，默认为执行启动 victoria 的当前目录下的 victoria-metrics-data 目录中

-retentionPeriod #存储数据的保留，较旧的数据会自动删除，默认保留期为 1 个月，默认单位为m(月)，支持的单位有 h(hour), d(day), w(week), y(year)

设置 service 启动文件

mv victoria-metrics-prod /usr/local/bin

cat /etc/systemd/system/victoria-metrics-prod.service

[Unit]
Description=For Victoria-metrics-prod Service
After=network.target

[Service]
ExecStart=/usr/local/bin/victoria-metrics-prod -httpListenAddr=0.0.0.0:8428 -storageDataPath=/data/victoria -retentionPeriod=3

[Install]
WantedBy=multi-user.target

启动并设置开机自启

1
2
3

systemctl daemon-reload
systemctl restart victoria-metrics-prod.service
systemctl enable victoria-metrics-prod.service

验证页面： 192.168.15.100:8428 查看数据

Prometheus 设置

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).


remote_write:
  - url: http://192.168.15.100:8428/api/v1/write

重启 prometheus，再次验证 192.168.15.100:8428 查看数据

grafana 配置

添加数据源：

类型为 prometheus ，地址及端口为 VictoriaMetrics: http://192.168.15.100:8428

导入指定模版

8919

官方 docker-compose

https://github.com/VictoriaMetrics/VictoriaMetrics/tree/cluster/deployment/docker

git clone https://github.com/VictoriaMetrics/VictoriaMetrics.git
cd VictoriaMetrics/deployment/docker

[root@yuan docker]# ls -l
total 44
-rw-r--r-- 1 root root    61 May 10 19:32 alertmanager.yml
-rw-r--r-- 1 root root 17025 May 10 19:32 alerts.yml
drwxr-xr-x 2 root root    24 May 10 19:32 base
drwxr-xr-x 2 root root    24 May 10 19:32 builder
-rw-r--r-- 1 root root  2843 May 10 19:32 docker-compose.yml
-rw-r--r-- 1 root root  6280 May 10 19:32 Makefile
-rw-r--r-- 1 root root   298 May 10 19:32 prometheus.yml
drwxr-xr-x 4 root root    43 May 10 19:32 provisioning
-rw-r--r-- 1 root root  1495 May 10 19:32 README.md

docker-compose up -d

# 验证 web 界面
192.168.15.100:8428

集群版部署

组件介绍
vminsert #写入组件(写)，vminsert 负责接收数据写入并根据对度量名称及其所有标签的一致 hash 结果将数据分散写入不同的后段 vmstorage 节点之间 vmstorage，vminsert 默认端口 8480

vmstorage #存储原始数据并返回给定时间范围内给定标签过滤器的查询数据，默认端口8482

vmselect #查询组建(读)，连续 vmstorage，默认端口 8481

其他可选组件：

vmagent #是一个很轻量级但功能强大的代理，它可以从 node_exporter 各种来源收集度量指标，并将它们存储在 VictoriaMetrics 或任何其他支持远程写入协议的与 prometheus 兼容的存储系统中，有替代 prometheus server 的意向

vmalert #替换prometheus server，以 VictoriaMetrics 为数据源，基于兼容 prometheus 的告警规则，判断数据是否异常，并将产生的通知发送给 alertmanager

Vmgateway #读写 VictoriaMetrics 数据的代理网关，可实现限速和访问控制等功能，目前为企业版组件

vmctl #VictoriaMetrics 的命令行工具，目前主要用于将 prometheus 、opentsdb 等数据源的数据迁移到VictoriaMetrics

部署集群

分别在各个 VictoriaMetrics 服务器进行安装配置

wget https://github.com/VictoriaMetrics/VictoriaMetrics/releases/download/v1.71.0/victoria-metrics-amd64-v1.71.0-cluster.tar.gz

tar xvf victoria-metrics-amd64-v1.71.0-cluster.tar.gz

[root@yuan victoria]# ls -l
total 35016
-rwxr-xr-x 1 yuan yuan 11312016 Dec 21 01:33 vminsert-prod
-rwxr-xr-x 1 yuan yuan 13026872 Dec 21 01:33 vmselect-prod
-rwxr-xr-x 1 yuan yuan 11512464 Dec 21 01:33 vmstorage-prod

mv vminsert-prod vmselect-prod vmstorage-prod /usr/local/bin

# 主要参数
-httpListenAddr string
    Address to listen for http connections (default ":8482")
-vminsertAddr string
    TCP address to accept connections from vminsert services (default ":8400") 
-vmselectAddr string
    TCP address to accept connections from vmselect services (default ":8401 ")

部署 vmstorage-prod 组件

负责数据的持久化，监听端口:API 8482，数据写入端口：8400，数据读取端口：8401

vim /etc/systemd/system/vmstorage.service

[Unit]
Description=Vmstorage Server
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/tmp
ExecStart=/usr/local/bin/vmstorage-prod -loggerTimezone Asia/Shanghai -storageDataPath /data/vmstorage-data -httpListenAddr :8482 -vminsertAddr :8400 -vmselectAddr :8401

[Install]
WantedBy=multi-user.target

启动服务并设置开机自启

1
2
3

systemctl restart vmstorage.service
systemctl enable vmstorage.service
systemctl status vmstorage.service

配置另外两台服务器

# 将启动文件发送至另两台服务器
scp /etc/systemd/system/vmstorage.service 192.168.15.101:etc/systemd/system/vmstorage.service

scp /etc/systemd/system/vmstorage.service 192.168.15.101:etc/systemd/system/vmstorage.service

scp /usr/local/bin/vm* 192.168.15.101:/usr/local/bin/
scp /usr/local/bin/vm* 192.168.15.102:/usr/local/bin/

# 101
systemctl restart vmstorage.service && systemctl enable vmstorage.service
systemctl status vmstorage.service

# 102
systemctl restart vmstorage.service && systemctl enable vmstorage.service
systemctl status vmstorage.service

部署 vminsert-prod 组件

接收外部的写请求，默认端口 8480

vim /etc/systemd/system/vminsert.service

[Unit]
Description=Vminsert Server
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/tmp
ExecStart=/usr/local/bin/vminsert-prod -httpListenAddr :8480 -storageNode=192.168.15.100:8400,192.168.15.101:8400,192.168.15.102:8400

[Install]
WantedBy=multi-user.target

启动服务并设置开机自启

1
2
3

systemctl daemon-reload
systemctl restart vminsert.service && systemctl enable vminsert.service
systemctl status vminsert.service

配置另外两台服务器

scp /etc/systemd/system/vminsert.service 192.168.15.101:/etc/systemd/system/vminsert.service
scp /etc/systemd/system/vminsert.service 192.168.15.102:/etc/systemd/system/vminsert.service

systemctl daemon-reload
systemctl restart vminsert.service && systemctl enable vminsert.service
systemctl status vminsert.service

部署 vmselect-prod 组件

负责接收外部的读请求，默认端口 8481

vim /etc/systemd/system/vmselect.service

[Unit]
Description=Vminsert Server
After=network.target

[Service]
Restart=on-failure
WorkingDirectory=/tmp
ExecStart=/usr/local/bin/vmselect-prod -httpListenAddr :8481 -storageNode=192.168.15.100:8401,192.168.15.101:8401,192.168.15.102:8401

[Install]
WantedBy=multi-user.target

启动服务并设置开机自启

1
2
3

systemctl daemon-reload
systemctl restart vmselect.service && systemctl enable vmselect.service
systemctl status vmselect.service

配置另外两台服务器

scp /etc/systemd/system/vmselect.service 192.168.15.101:/etc/systemd/system/vmselect.service
scp /etc/systemd/system/vmselect.service 192.168.15.102:/etc/systemd/system/vmselect.service

systemctl daemon-reload
systemctl restart vmselect.service && systemctl enable vmselect.service
systemctl status vmselect.service

验证服务端口

#192.168.15.100
curl http://192.168.15.100:8480/metrics
curl http://192.168.15.100:8481/metrics
curl http://192.168.15.100:8482/metrics

#192.168.15.101
curl http://192.168.15.101:8480/metrics
curl http://192.168.15.101:8481/metrics
curl http://192.168.15.101:8482/metrics

#192.168.15.102
curl http://192.168.15.102:8480/metrics
curl http://192.168.15.102:8481/metrics
curl http://192.168.15.102:8482/metrics

Prometheus 配置远程写入

vim /apps/prometheus/prometheus.yml

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).
  
# 单机写入
#remote_write:
#  - url: http://192.168.15.100:8428/api/v1/write

# 集群写入
remote_write:
  - url: http://192.168.15.100:8480/insert/0/prometheus
  - url: http://192.168.15.101:8480/insert/0/prometheus
  - url: http://192.168.15.102:8480/insert/0/prometheus

grafana 数据源配置

https://github.com/VictoriaMetrics/VictoriaMetrics#grafana-setup

添加数据源

在 grafana settings 中添加Data Sources，

Name：vmselect , URL：http://192.168.15.102:8481/select/0/prometheus

导入指定模版：

13824

import 导入后查看 Dashboard

开启数据复制

https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html#replication-and-data-safety

默认情况下，数据被 vminsert 的组件基于 hash 算法分别将数据持久化到不同的 vmstorage 节点，可以启用 vminsert 组件支持的 -replicationFactor=N 复制功能，将数据分别在各节点保存一份完整的副本以实现数据的高可用