VPS Monitoring: Prometheus, Grafana and Node Exporter

When a VPS slows down, a current process list cannot show what changed an hour earlier. Prometheus metrics let you compare CPU time, available memory, filesystem space and device activity on a shared timeline. Alert rules can prompt an investigation when a defined condition persists. Monitoring does not prevent failures or guarantee detection before users are affected.

This guide builds a new, native systemd installation on one Linux amd64 VPS using Node Exporter 1.12.1, Prometheus 3.15.1 and Alertmanager 0.34.2, checked on 4 November 2026. Grafana is installed separately on the same host. Metrics and management listeners bind to loopback. Prepare all files before starting services; keep target labels, thresholds and notification routes consistent throughout the installation.

Choose Signals That Support an Investigation

Resource use, latency and customer impact are different signals. High non-idle CPU includes iowait and steal; CPU ready is a separate hypervisor metric, not a standard Node Exporter metric. Low available memory warrants checking workload, swap and limits. High device activity or reported iowait warrants investigating storage latency and queueing. None identifies a particular process, database query or endpoint by itself.

Collect representative quiet and busy periods and annotate deployments and maintenance. Application request latency, errors, order outcomes and queue depth require suitable instrumentation or an additional exporter. Host measurements provide context for these signals. Alerting delay also depends on scrape frequency, evaluation, the rule’s pending duration and delivery; it is not instantaneous.

Architecture: One Host with Local Access

Node Exporter exposes host metrics at 127.0.0.1:9100. Prometheus scrapes and evaluates rules every 15 seconds, stores the data and sends firing alerts to Alertmanager at 127.0.0.1:9093. Native Grafana queries Prometheus at 127.0.0.1:9090. Prometheus owns the alerts in this guide; creating identical Grafana rules would require an explicit plan to avoid duplicate notifications.

A single host is an accessible starting point, but its failure also stops local collection and alert delivery. Use monitoring or a probe in an independent location for host and service availability. Moving components to another host requires corresponding network, TLS/authentication and target changes. In separate containers, localhost refers to each container’s own network namespace; the native addresses below are not a Compose configuration.

The Prometheus alerting documentation, Grafana documentation and Node Exporter repository provide the upstream configuration and installation references. Keep versions, digests and configuration changes in your deployment record. The pinned examples are for a new installation; do not overwrite existing services or monitoring data with these commands.

Install Official Binaries and Node Exporter

Use a separate service account for each component. Keep executables and configuration root-owned; only Prometheus and Alertmanager data directories need daemon write access. Node Exporter reads host information and an optional textfile directory. Its account should not be able to replace its own executable or manufacture textfile metrics.

Step 1: Verify Downloads and Create Accounts

These commands require a new Ubuntu/Debian-style Linux installation with systemd, curl, tar, sha256sum and a root Bash shell; use sudo -i first if necessary. uname -m must report x86_64. ARM64 requires its own release archive and verified checksum. Each digest below belongs only to the specified version and architecture. The script installs all three components and their validation tools, but starts no services.

set -eu
workdir=$(mktemp -d /tmp/vps-monitoring.XXXXXX)
cd "$workdir"
curl --fail --location --proto =https --tlsv1.2 -o node_exporter-1.12.1.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz
printf '%s  %s\n' 'b51d8a76aa2a9156a55d501aca6276fae09e262259a5e4e831d2c2222f084e63' 'node_exporter-1.12.1.linux-amd64.tar.gz' | sha256sum --check -
tar -xzf node_exporter-1.12.1.linux-amd64.tar.gz
install -o root -g root -m 0755 node_exporter-1.12.1.linux-amd64/node_exporter /usr/local/bin/node_exporter
curl --fail --location --proto =https --tlsv1.2 -o prometheus-3.15.1.linux-amd64.tar.gz https://github.com/prometheus/prometheus/releases/download/v3.15.1/prometheus-3.15.1.linux-amd64.tar.gz
printf '%s  %s\n' '6a726885328ad9efebbf36e876ce2fb6225468a71fd1e7ce3ba666d5f4d97a84' 'prometheus-3.15.1.linux-amd64.tar.gz' | sha256sum --check -
tar -xzf prometheus-3.15.1.linux-amd64.tar.gz
install -o root -g root -m 0755 prometheus-3.15.1.linux-amd64/prometheus /usr/local/bin/prometheus
install -o root -g root -m 0755 prometheus-3.15.1.linux-amd64/promtool /usr/local/bin/promtool
curl --fail --location --proto =https --tlsv1.2 -o alertmanager-0.34.2.linux-amd64.tar.gz https://github.com/prometheus/alertmanager/releases/download/v0.34.2/alertmanager-0.34.2.linux-amd64.tar.gz
printf '%s  %s\n' '17ee14acd0f6ba6e25a436ff223d9c17ddd5cf484f543ad12f0932eaa89ba186' 'alertmanager-0.34.2.linux-amd64.tar.gz' | sha256sum --check -
tar -xzf alertmanager-0.34.2.linux-amd64.tar.gz
install -o root -g root -m 0755 alertmanager-0.34.2.linux-amd64/alertmanager /usr/local/bin/alertmanager
install -o root -g root -m 0755 alertmanager-0.34.2.linux-amd64/amtool /usr/local/bin/amtool
useradd --system --no-create-home --shell /usr/sbin/nologin nodeexp
useradd --system --no-create-home --shell /usr/sbin/nologin prometheus
useradd --system --no-create-home --shell /usr/sbin/nologin alertmanager
install -d -o root -g nodeexp -m 0750 /var/lib/node_exporter /var/lib/node_exporter/textfile
install -d -o root -g prometheus -m 0750 /etc/prometheus /etc/prometheus/rules
install -d -o prometheus -g prometheus -m 0700 /var/lib/prometheus
install -d -o root -g alertmanager -m 0750 /etc/alertmanager
install -d -o alertmanager -g alertmanager -m 0700 /var/lib/alertmanager

Step 2: Define the Node Exporter Service

The service keeps default collectors and filesystem exclusions, binds only to 127.0.0.1:9100 and reads the root-owned textfile directory. When adapting exclusions, use the current –collector.filesystem.mount-points-exclude flag and confirm that required mounts still appear. The obsolete ignored-mount-points option is not used. Save this unit now; startup is deferred until all configurations and rules are ready.

cat > /etc/systemd/system/node_exporter.service <<'EOF'
[Unit]
Description=node_exporter
Wants=network-online.target
After=network-online.target

[Service]
User=nodeexp
Group=nodeexp
Type=simple
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100 --collector.textfile.directory=/var/lib/node_exporter/textfile
Restart=on-failure
RestartSec=5s
UMask=0027
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
EOF

Step 3: Verify the Intended Listener

After the startup section, inspect listeners and retrieve metrics locally using the commands below. The explicit loopback bind keeps port 9100 off public interfaces; Prometheus and Alertmanager use the same approach. For remote collection, design private network/VPN and appropriate TLS/authentication access first. A firewall allow rule alone does not remove broader rules; validate both IPv4 and IPv6 while preserving SSH access.

ss -ltnp
curl --fail http://127.0.0.1:9100/metrics

Review service logs and required collector metrics after startup. A successful scrape does not prove every collector succeeded; inspect node_scrape_collector_success and, when applicable, node_textfile_scrape_error. Container deployment requires host namespace, filesystem mounts and matching path settings; published ports and firewall behavior must be checked separately. Do not combine container snippets with this native setup without adapting the architecture.

Configure Prometheus, Rules and Alertmanager

The binaries, accounts and directories already exist from Step 1. Prepare the following configuration and systemd files before enabling services. Prometheus uses job=”node” and instance=”vps-01″ consistently. If the instance name changes, update the expected-target absence rule as well. Each instance must remain uniquely identifiable as you add hosts.

Step 1: Set the Target and Alertmanager Address

This configuration scrapes the local exporter and Prometheus itself, loads rules from /etc/prometheus/rules and explicitly sends alerts through alerting.alertmanagers to 127.0.0.1:9093. The Alertmanager receiver is configured later. An alert visible in Prometheus does not imply that notification routing or SMTP delivery works.

cat > /etc/prometheus/prometheus.yml <<'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - 127.0.0.1:9093
scrape_configs:
- job_name: prometheus
  static_configs:
  - targets:
    - 127.0.0.1:9090
- job_name: node
  static_configs:
  - targets:
    - 127.0.0.1:9100
    labels:
      instance: vps-01
EOF

The unit sets 15 days and 2 GB of block retention; whichever limit is reached first removes older blocks. The size limit is not a hard cap on total disk use: WAL, head data and compaction need extra room. These values are a small test starting point, not capacity guidance for every installation. Measure ingestion and leave headroom; local TSDB storage is a single-node database, not a replicated backup.

cat > /etc/systemd/system/prometheus.service <<'EOF'
[Unit]
Description=prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.listen-address=127.0.0.1:9090 --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=2GB
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
UMask=0027
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
EOF

Step 2: Write Valid Host Alert Rules

The file below has the required groups → name → rules structure. Host CPU uses the complement of the per-core idle-rate average, not process_cpu_seconds_total, which measures the exporter process. Non-idle includes iowait and steal; it is not a pure application-busy ratio. MemAvailable includes reclaimable memory and does not identify a leak. Filesystem space, device busy time and reported CPU iowait are separate measurements.

cat > /etc/prometheus/rules/host.yml <<'EOF'
groups:
- name: vps-host
  rules:
  - alert: CPUHighSustained
    expr: (1 - avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[5m]))) > 0.85
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: High non-idle CPU ({{ $labels.instance }})
      description: Non-idle CPU ratio remains above 85%. Check modes and workload; this includes iowait and steal.
  - alert: MemoryLow
    expr: (node_memory_MemAvailable_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"}) < 0.10
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: Low available memory ({{ $labels.instance }})
      description: Available memory remains below 10%; inspect workload, swap and limits before diagnosing a leak.
  - alert: DiskSpaceLow
    expr: ((node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="node",fstype!~"tmpfs|overlay|squashfs"})
      and (node_filesystem_size_bytes{job="node"} > 0) and (node_filesystem_readonly{job="node"} == 0)) < 0.10
    for: 10m
    labels:
      severity: critical
    annotations:
      summary: Low filesystem space ({{ $labels.instance }} {{ $labels.mountpoint }})
      description: Available space on a writable nonzero filesystem remains below 10%. Check bytes, inodes and growth;
        preserve required data.
  - alert: DiskBusyHigh
    expr: rate(node_disk_io_time_seconds_total{job="node"}[5m]) > 0.60
    for: 15m
    labels:
      severity: warning
    annotations:
      summary: High device busy-time ratio ({{ $labels.instance }} {{ $labels.device }})
      description: Busy-time ratio remains above 60%; correlate latency and queueing. This is not a universal saturation
        percentage.
  - alert: CPUWaitingForIO
    expr: avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="iowait"}[5m])) > 0.20
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: High reported CPU iowait ({{ $labels.instance }})
      description: Reported CPU iowait remains above 20%; inspect storage latency and workload. Kernel accounting
        has limitations.
  - alert: NodeScrapeFailed
    expr: up{job="node"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: Node scrape failed ({{ $labels.instance }})
      description: Prometheus has failed to scrape this target for one minute; inspect exporter, network and host
        separately.
  - alert: ExpectedNodeSamplesMissing
    expr: absent_over_time(up{job="node",instance="vps-01"}[5m])
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: Expected node samples missing ({{ $labels.instance }})
      description: No up samples for the expected target exist in the five-minute window. Check target configuration
        and relabeling.
  - alert: RecentBoot
    expr: (time() - node_boot_time_seconds{job="node"} >= 0) and (time() - node_boot_time_seconds{job="node"} <
      600)
    for: 1m
    labels:
      severity: info
    annotations:
      summary: Recent host boot observed ({{ $labels.instance }})
      description: Observed boot age is less than ten minutes. Compare maintenance records and system logs; this
        is not outage duration.
EOF

Thresholds are examples: non-idle CPU above 85% for ten minutes, available memory below 10% for five, writable filesystem space below 10% for ten, device busy time above 60% for fifteen and reported iowait above 20% for ten. The five-minute rate window differs from for: the condition must remain true at each evaluation for the pending duration. New or sparse series have rate extrapolation effects during warmup. The official alerting guide explains rule evaluation and notification roles; test the actual workload before choosing paging thresholds.

Step 3: Configure the SMTP Receiver and Service

Replace example.com addresses with an authorized recipient, sender and SMTP account. The provider must allow SMTP authentication and STARTTLS on the selected endpoint; confirm valid certificates and outbound port 587 access. Store the password through a secure local editor in /etc/alertmanager/smtp-password with root:alertmanager ownership and mode 0640. Keep it out of shell commands, articles and version control. The protected directory created earlier limits access to the configuration.

cat > /etc/alertmanager/alertmanager.yml <<'EOF'
route:
  receiver: mail
  group_by:
  - alertname
  - instance
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 2h
receivers:
- name: mail
  email_configs:
  - to: alerts@example.com
    from: monitoring@example.com
    smarthost: smtp.example.com:587
    auth_username: monitoring@example.com
    auth_password_file: /etc/alertmanager/smtp-password
    require_tls: true
    send_resolved: true
EOF
cat > /etc/systemd/system/alertmanager.service <<'EOF'
[Unit]
Description=alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml --storage.path=/var/lib/alertmanager --web.listen-address=127.0.0.1:9093 --cluster.listen-address=
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
UMask=0027
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
EOF

All severity levels go to the same mail receiver here; labels alone do not create escalation routes. group_wait, group_interval and repeat_interval affect notification timing in addition to the rule’s for duration. send_resolved explicitly enables resolution emails. The empty cluster listen address disables Alertmanager HA mode for this single-host example. After every file is ready, set ownership and permissions, validate as the service users, and then start:

# Run only after all files and the protected SMTP password exist.
set -eu
chown root:prometheus /etc/prometheus/prometheus.yml /etc/prometheus/rules/*.yml
chmod 0640 /etc/prometheus/prometheus.yml /etc/prometheus/rules/*.yml
chown root:alertmanager /etc/alertmanager/alertmanager.yml /etc/alertmanager/smtp-password
chmod 0640 /etc/alertmanager/alertmanager.yml /etc/alertmanager/smtp-password
chown root:root /etc/systemd/system/{node_exporter,prometheus,alertmanager}.service
chmod 0644 /etc/systemd/system/{node_exporter,prometheus,alertmanager}.service
runuser -u alertmanager -- test -s /etc/alertmanager/smtp-password
runuser -u alertmanager -- test -r /etc/alertmanager/smtp-password
runuser -u prometheus -- /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml
runuser -u alertmanager -- /usr/local/bin/amtool check-config /etc/alertmanager/alertmanager.yml
systemd-analyze verify /etc/systemd/system/{node_exporter,prometheus,alertmanager}.service
systemctl daemon-reload
systemctl enable --now node_exporter alertmanager prometheus
systemctl --no-pager --full status node_exporter alertmanager prometheus
curl --fail http://127.0.0.1:9100/metrics
curl --fail http://127.0.0.1:9090/-/ready
curl --fail http://127.0.0.1:9093/-/ready

For later configuration-only changes, validate before sending SIGHUP through systemctl reload, then inspect the logs. A successful reload command alone is not proof that the new configuration was accepted. Unit changes instead require daemon-reload and a service restart.

runuser -u prometheus -- /usr/local/bin/promtool check config /etc/prometheus/prometheus.yml && systemctl reload prometheus
runuser -u alertmanager -- /usr/local/bin/amtool check-config /etc/alertmanager/alertmanager.yml && systemctl reload alertmanager
journalctl -u prometheus -u alertmanager --since '-5 minutes' --no-pager

Set Up Grafana and Read the Right Units

Build panels around specific questions: what fraction of CPU time is non-idle, how much memory is available, which filesystem is running short, and what changed in device activity? Show instance, device and mountpoint where relevant. Keep missing data distinct from zero and correlate with maintenance, traffic and application measurements.

Step 1: Install Native Grafana and Add Prometheus

Grafana is not included in the download script. Install a supported native stable package on this same host following the official Grafana documentation; the version checked for this guide is 13.3.0. Before its first start, set [server] http_addr = 127.0.0.1 and http_port = 3000 in /etc/grafana/grafana.ini. If the package already started it, restrict external access and restart with the intended settings. Change the initial administrator password and keep anonymous access disabled. Connect through a trusted SSH tunnel; use Connections → Add new connection → Prometheus → Add new data source and URL http://127.0.0.1:9090. Save & test, then query real node samples.

Step 2: Create a Server Overview

Use ratio units for fractions, bytes for capacity and seconds/duration for boot age. Deployment annotations help correlate changes but do not prove causation. Process-level memory and endpoint-level load need additional instrumentation; Node Exporter cannot supply those details merely because a dashboard template contains panels for them.

Non-idle CPU ratio:

1 - avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[5m]))

Select Percent (0.0–1.0). This averages cores and includes iowait and steal in the idle complement; inspect modes and per-core activity separately when needed. Averaging can hide a single-core bottleneck. A five-minute rate is a windowed estimate, not an instantaneous utilization measurement.

Available memory ratio:

node_memory_MemAvailable_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"}

Use Percent (0.0–1.0). MemAvailable estimates memory available to new applications without swapping, including reclaimable cache/slab and kernel watermarks. Low values warrant checking swap, OOM records, cgroup limits and workload; this graph alone cannot identify a memory leak or predict a crash.

Available filesystem space:

(node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|overlay|squashfs"} / node_filesystem_size_bytes{job="node",fstype!~"tmpfs|overlay|squashfs"}) and (node_filesystem_size_bytes{job="node"} > 0) and (node_filesystem_readonly{job="node"} == 0)

Use Percent (0.0–1.0) and show instance, mountpoint and device. The query filters read-only, zero-size and selected temporary/image filesystems; monitor read-only failures and required excluded mounts separately. Also display absolute available bytes and inode capacity. With absolute threshold values, a base red color, 0.1 yellow and 0.2 green represents below 10%, 10–20% and at least 20% respectively.

Reported CPU iowait:

avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="iowait"}[5m]))

Use this separately from rate(node_disk_io_time_seconds_total{job="node"}[5m]), the per-device busy-time ratio. Kernel iowait accounting has limitations, and device busy time is not a universal saturation percentage on parallel storage. Correlate device latency, throughput, IOPS and queueing before blaming a database or backup. Verify how virtual devices map to filesystems; the two label sets are not interchangeable.

Observed time since boot:

time() - node_boot_time_seconds{job="node"}

The result is seconds since the observed boot timestamp; format it as a duration. Add a separate status panel for up{job="node"}: 1 means the latest scrape succeeded, 0 means it failed, and missing data is a third state. None of these is a service-availability percentage or a complete history of application outages.

Inspect raw data when a panel is blank or unexpectedly flat. Confirm the selected source, time range, job and instance labels before replacing missing values with zero. A successful data-source connection test proves neither metric completeness nor alert delivery. Keep an explicit owner for Prometheus rules if Grafana alerting is also introduced later.

Handle Failed Scrapes, Missing Targets and Reboots

for delays firing while a condition remains true at each evaluation. It is not amplitude hysteresis: the example rules resolve at the first evaluation where the condition is false. Different trigger and recovery thresholds require separate logic; keep_firing_for is another option that holds firing state for a duration after the condition ends. Longer waits trade less noise for slower detection.

NodeScrapeFailed triggers after up=0 persists for one minute. The exporter, network, permissions or host may be responsible; this is not proof that the machine is off. A blackbox probe in an independent location can test a specific HTTP endpoint or port. It establishes the outcome from that probe’s location and configured checks, not every user path or the complete business transaction.

ExpectedNodeSamplesMissing checks a specific expected instance. It becomes active when its five-minute range has no up samples and then waits one minute; if no sample has existed since startup, it does not first wait for a full five minutes of history. An up=0 sample is present and therefore is not absence. RecentBoot observes a boot age between zero and ten minutes for one minute; it is informational and can miss restarts first scraped after that window. It does not measure downtime.

Use bounded maintenance silences with exact target labels and confirm their end state. Silences suppress matching notifications; collection and rule evaluation continue. This example groups by alertname and instance and sends all severities to one recipient. Add and verify child routes for separate escalation destinations. A silence expiring does not by itself establish that delivery has recovered.

Protect Access and Plan Recovery

The three monitoring listeners use loopback and should not be publicly reachable. Localhost does not isolate untrusted users or processes on the same host. For remote administration use a trusted SSH tunnel or a properly authenticated TLS proxy; keep Grafana roles least-privileged and accounts personal. A hidden port number is not authentication. Verify IPv4/IPv6 exposure, certificate renewal and access revocation.

Keep instance names unique and stable. Adding environment labels can help routing, but every rule, aggregation and missing-target check must remain consistent with the chosen identity. Do not create unbounded labels from user IDs or raw request paths. More targets, collectors, cardinality and frequent scrapes increase resource requirements; measure before extending retention or collection scope.

A failed monitoring service should not become an application dependency. Back up configuration, rules, required Alertmanager state and Grafana data; use a consistent TSDB backup method and test restoration in a separate environment. Copying a live database directory arbitrarily is not a verified backup. Restarts can interrupt collection and evaluation for an unbounded period if recovery fails; monitor the monitoring system from outside its failure domain.

Tune Thresholds Against Workload and User Impact

Record representative workload and adjust thresholds using application latency, errors, capacity and an agreed response time. An already busy CPU is not automatically a paging incident, and a small filesystem may need an absolute-bytes or growth forecast alert before a percentage rule fires. Avoid simply raising thresholds until warnings disappear; confirm whether the underlying pressure is acceptable.

Recurring I/O activity may align with exports, backups or reports, but confirm that relationship before rescheduling work or changing storage. Cache policy, query optimization and workload separation are possible responses after evidence is gathered. Anycast DNS and automatic failover can support resilience planning; detection, DNS caches, existing sessions and data consistency still affect failover. Monitoring and failover do not guarantee that users never see a disruption.

For example, if reported iowait rises during a scheduled export, compare device latency and queueing with the export timeline. Test a smaller batch or different schedule in a suitable environment, then compare both export completion and application response times. Treat this as a diagnostic workflow rather than proof that a particular workload is responsible whenever iowait rises.

Verify the Full Path and Troubleshoot Missing Data

Inspect Prometheus Targets, Rules and Alerts for the node target, eight loaded rules, pending/firing state and the configured Alertmanager target. Query up and actual node metrics with the same job/instance labels used by rules. Check exporter/collector status and server logs. A job=”node” versus job=”nodes” mismatch can produce an empty vector that never fires a threshold alert.

Use promtool test rules with normal, strict-boundary, short-peak, counter-reset, absent and resolved samples. In a separate test environment, route a short-lived, clearly labeled synthetic test alert to an authorized test recipient and verify rule → Alertmanager → SMTP → received message. Then remove the test rule and verify resolution. Parsing, service readiness and an Alertmanager UI entry are distinct checks; none alone proves delivery. Avoid generating production CPU load just to test a threshold.

Extend Collection and Retention Deliberately

The textfile collector suits metrics tied to a particular machine, such as the last successful run of a host batch job. An authorized producer should write a temporary file in the same directory and atomically rename it to a .prom file; timestamps on samples are unsupported. Add freshness checks because an old file can remain readable. Ongoing service request/error/queue metrics need appropriate application instrumentation or an exporter. External blackbox checks and process metrics are separate additions.

Choose retention from measured ingestion, recovery objectives and available disk; the example’s 15 days/2 GB is not a universal recommendation. A remote_write backend adds its own access control, durability, queueing and failure behavior and is not automatically a backup. Keep credentials separate, test outages and document recovery. Upgrade with pinned versions, release notes and compatibility checks, including how changes affect stored data.

Accept the Installation with Recorded Checks

Before relying on the installation, verify target samples, meaningful dashboard units, all rule conditions, missing-data behavior and delivery to the intended test recipient. Check directory ownership, loopback listeners, disk headroom and recovery after a controlled service restart. Record the versions, date and results; a green dashboard is only one observation.

Start with one target and an actionable set of rules, then observe representative traffic before adding hosts. Document who responds to each alert, how to silence a planned maintenance window and how to test recovery. Add independent monitoring for the host that runs Prometheus and Alertmanager so a local failure cannot silently remove the only notification path.

Use the collected evidence to compare application tuning, workload scheduling, capacity changes and service separation. The useful result is a traceable connection between a metric, its alert condition and a tested response path. Keep those contracts consistent as versions, target labels and architecture evolve.