[pbs-devel] [PATCH proxmox/proxmox-backup v5] add metrics server capability
Dominik Csapak
d.csapak at proxmox.com
Wed Feb 2 10:50:09 CET 2022
this series adds support for exporting metrics data to external
metric servers.
for now this includes only data we gather for RRD, though it should
not be hard to extend that functionality
also only influxdb (udp/http(s)) is currently supported, but it should
also not be too hard to include more options here
this also includes a gui now
the 'cannot create with --enable' bug matthias mentioned seems to be a
quirk of the proxmox-backup-debug binary, the api/gui works just fine...
(i'll check it out soon)
this versions has some additional changes that were necessary and
i only noticed that because i wrote the gui ...
ofc, proxmox-backup depends on bumped versions of the proxmox-* crates
changes from v4:
* rebase on master
* move connect_to_udp to udp::connect(), and let it really try every address
* adds 'test_influxdb_http/udp' functions that try to connect for some
sanity checks. (needed a little refactor in the http/udp parts but
mostly code move)
* checks the server connection on create/edit via api when the enable
flag is set
* adds a generic 'list' api call in /admin/metricserver since we need
a place where we return *all* metrics servers regardless of type
(and this is the way we do it e.g. for realms)
* adds the gui to view/add/edit/delete the metric servers
(i put it under configuration, but that gets crowded... maybe there
is a better place?)
changes from v3:
* rebase on master
* introduced helper functions instead of InfluxDBHttp::new
* start tokio task directly in the helper
* combine channel close + join
* fix api description
* combine host/port/protocol in the api types
* introduce a connect_to_udp helper
* use NixPath in the fs_info helper
changes from v2:
* rebase on master
* rustfmt
* clippy (fixed not everything)
* renamed DiskUsage in proxmox-sys and added some more fields
* added 'enable' property for the config (like we have in pve)
* subtracted 50bytes from mtu in the udp variant (for ip header)
changes from v1:
* fixed ipv6 support for udp (tested it this time ;) )
* dropped the 'flush' functionality of the MetricsChannel, but kept the
wrapper struct: it did not do what i intended, and after rethinking it,
turns out it's not necessary (as we autoflush when the data gets to large,
or when we close the channel). kept the struct so that the interface
can stay the same even if we want to implement a manual flush in the future
* improved the influxdb line formatter
* removed variables like 'names2' by reorganizing the code
* used Arc::clone(&foo) instead of foo.clone() (better visibilty)
* used CamelCase for the DeletableProperties
proxmox:
Dominik Csapak (2):
proxmox-async: add udp::connect() helper
proxmox-metrics: implement metrics server client code
Cargo.toml | 1 +
proxmox-async/Cargo.toml | 2 +-
proxmox-async/src/io/mod.rs | 2 +
proxmox-async/src/io/udp.rs | 36 +++++
proxmox-metrics/Cargo.toml | 21 +++
proxmox-metrics/debian/changelog | 5 +
proxmox-metrics/debian/copyright | 16 +++
proxmox-metrics/debian/debcargo.toml | 7 +
proxmox-metrics/src/influxdb/http.rs | 185 ++++++++++++++++++++++++++
proxmox-metrics/src/influxdb/mod.rs | 7 +
proxmox-metrics/src/influxdb/udp.rs | 86 ++++++++++++
proxmox-metrics/src/influxdb/utils.rs | 50 +++++++
proxmox-metrics/src/lib.rs | 117 ++++++++++++++++
13 files changed, 534 insertions(+), 1 deletion(-)
create mode 100644 proxmox-async/src/io/udp.rs
create mode 100644 proxmox-metrics/Cargo.toml
create mode 100644 proxmox-metrics/debian/changelog
create mode 100644 proxmox-metrics/debian/copyright
create mode 100644 proxmox-metrics/debian/debcargo.toml
create mode 100644 proxmox-metrics/src/influxdb/http.rs
create mode 100644 proxmox-metrics/src/influxdb/mod.rs
create mode 100644 proxmox-metrics/src/influxdb/udp.rs
create mode 100644 proxmox-metrics/src/influxdb/utils.rs
create mode 100644 proxmox-metrics/src/lib.rs
--
2.30.2
>From 7dfc34ef9f172f5ecb0554532bdcaa090184d24a Mon Sep 17 00:00:00 2001
From: Dominik Csapak <d.csapak at proxmox.com>
Date: Wed, 2 Feb 2022 10:41:52 +0100
Subject: [PATCH proxmox-backup v5 0/8] *** SUBJECT HERE ***
*** BLURB HERE ***
Dominik Csapak (8):
use 'fs_info' from proxmox-sys
pbs-api-types: add metrics api types
pbs-config: add metrics config class
backup-proxy: decouple stats gathering from rrd update
proxmox-backup-proxy: send metrics to configured metrics server
api: add metricserver endpoints
ui: add window/InfluxDbEdit
ui: add MetricServerView and use it
Cargo.toml | 1 +
pbs-api-types/src/lib.rs | 15 +
pbs-api-types/src/metrics.rs | 138 ++++++++
pbs-config/Cargo.toml | 1 +
pbs-config/src/lib.rs | 1 +
pbs-config/src/metrics.rs | 115 +++++++
src/api2/admin/datastore.rs | 4 +-
src/api2/admin/metricserver.rs | 91 +++++
src/api2/admin/mod.rs | 2 +
src/api2/config/metricserver/influxdbhttp.rs | 314 +++++++++++++++++
src/api2/config/metricserver/influxdbudp.rs | 269 +++++++++++++++
src/api2/config/metricserver/mod.rs | 16 +
src/api2/config/mod.rs | 2 +
src/api2/node/status.rs | 11 +-
src/api2/status.rs | 4 +-
src/bin/proxmox-backup-proxy.rs | 342 +++++++++++++++----
src/tools/disks/mod.rs | 21 +-
www/Makefile | 2 +
www/NavigationTree.js | 6 +
www/config/MetricServerView.js | 145 ++++++++
www/window/InfluxDbEdit.js | 218 ++++++++++++
21 files changed, 1619 insertions(+), 99 deletions(-)
create mode 100644 pbs-api-types/src/metrics.rs
create mode 100644 pbs-config/src/metrics.rs
create mode 100644 src/api2/admin/metricserver.rs
create mode 100644 src/api2/config/metricserver/influxdbhttp.rs
create mode 100644 src/api2/config/metricserver/influxdbudp.rs
create mode 100644 src/api2/config/metricserver/mod.rs
create mode 100644 www/config/MetricServerView.js
create mode 100644 www/window/InfluxDbEdit.js
--
2.30.2
More information about the pbs-devel
mailing list