Centralized VPS Logging with Loki, Alloy and Grafana

When an application fails, useful evidence may be split between Nginx access logs, application files and system services. A central log store on a VPS lets operators compare those records in one time window. It still needs explicit source coverage, correct timestamps and tested queries: a blank dashboard can mean missing telemetry rather than a healthy application. This guide establishes those boundaries before treating a graph or alert as reliable evidence.

The example uses Loki 3.7.9, Grafana Alloy 1.21.3 and Grafana 13.3.0, checked on November 4, 2026, in a fresh single-host Docker Compose deployment. Promtail reached end of life on March 2, 2026, so the collector for new deployments is Alloy. The working scope is selected Nginx access files in combined format, separate error files, newline-delimited application JSON, systemd journal entries with a 12-hour startup lookback and unit labels, and multiline worker logs. A separate native systemd installation is also provided. Automatic container discovery remains outside this file/journal example. Keep existing Loki schema history when migrating an installed system; this fresh-start configuration is not an in-place data migration.

What Centralized Logs Can Explain

Consider an application, queue worker and scheduled jobs spread across several nodes. Separate files can reveal individual failures while obscuring their sequence. Shared host/service identifiers and event timestamps make it easier to inspect a release alongside an increase in HTTP errors. Record the actual source coverage, clock synchronization and retention window, and recognize that logs omitted or lost before ingestion cannot be recovered by a query.

Alloy reads selected sources and forwards their entries. Loki indexes stream labels and stores the log content. Grafana queries Loki and, in this guide, evaluates Grafana-managed alert rules. These are different responsibilities: a successful file read is not durable delivery, a ready Loki endpoint does not prove every source was collected, and a working Explore query does not establish notification delivery.

Useful questions include whether actual 5xx responses increased after a release, whether an application is retrying an operation, and whether an expected heartbeat disappeared. A log-line count is not necessarily a request count, and a per-second error-line rate is not an error percentage. Keep those units visible in panels and alert descriptions so that operators know what changed and what evidence remains missing.

A Single-Host Loki, Alloy and Grafana Stack

The example runs all three services in one Compose network. Loki stores data locally, Alloy reads explicitly mounted host directories, and Grafana reaches Loki by its service name. Only Grafana publishes a host port, bound to 127.0.0.1; Loki has no published HTTP or gRPC port. This is a shared trust and failure domain, not high availability or isolation from the host administrator. Verify actual Docker routing, firewall exposure and access from outside the server.

Start with measured daily log volume, active streams, query concurrency, memory, disk and inode headroom. Seven days is the example retention period, not a universal capacity or compliance recommendation. Each added source and label combination changes the resource profile. Keep a recovery plan and independent monitoring that can still report a failure when the entire logging VPS is unavailable.

Metrics and external availability checks complement the log timeline. The VPS monitoring guide with Prometheus, Grafana and Uptime Kuma explains those additional components. CPU, memory, queue depth and worker health require actual emitted measurements or probes; the file sources below do not automatically provide them.

Prepare the Files and Pinned Compose Services

Prerequisites and file permissions

Use a Linux VPS with Docker Engine, a current Compose plugin, authorized administration and adequate storage. Prepare a separate test directory before using production data. Pin the listed release versions, check release notes and record the tested image digests for deployment. Replacing every tag with latest does not establish configuration compatibility, and filesystem storage on a single VPS remains one copy of the data.

Create docker-compose.yml, loki.yml, config.alloy, state/loki, state/alloy, state/grafana and secrets/grafana_admin_password. The state directories must be writable by the explicitly selected container UIDs 10001, 473 and 472 respectively. Restrict the password file to administrators and Grafana UID472. Set LOG_READ_GID to the real numeric host log-reader group; UID473 plus that supplementary group needs directory traversal and file-read access to the selected logs. Check SELinux policy where applicable. Every bind source must already exist as the correct file or directory; short bind syntax can create a missing source as a directory.

Set JOURNAL_DIR to an existing /var/log/journal or /run/log/journal directory, and set JOURNAL_GID and ADM_GID to the actual host reader-group IDs. UID473 and its supplementary groups must be able to traverse the directory and read the journal files; /etc/machine-id must be the real host file. Ensure worker.log exists in the mounted application directory. Adapt group names and policy to the distribution. Official image digests are pinned below. The Alloy image defaults to root; UID473 is an explicit override, and upstream marks non-root support in this image experimental. Verify this configuration in the target environment or use the separate native alternative below.

Loki TSDB schema and compactor retention

Save this as loki.yml. TSDB, schema v13 and a 24-hour index period form the fresh filesystem configuration. auth_enabled: false selects single-tenant operation; it does not add user authentication. HTTP listens inside the container so Alloy and Grafana can reach it; gRPC uses container loopback. Existing installations must preserve historical schema periods and follow the release-specific storage/upgrade path instead of replacing old entries with this example.

# Loki 3.7.9: fresh single-host Compose deployment, no published Loki ports.
auth_enabled: false
target: all
server:
  http_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 9096
common:
  instance_addr: 127.0.0.1
  path_prefix: /var/lib/loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
  storage:
    filesystem:
      chunks_directory: /var/lib/loki/chunks
      rules_directory: /var/lib/loki/rules
schema_config:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
storage_config:
  tsdb_shipper:
    active_index_directory: /var/lib/loki/index
    cache_location: /var/lib/loki/index-cache
ingester:
  wal:
    enabled: true
    dir: /var/lib/loki/wal
limits_config:
  retention_period: 168h
  reject_old_samples: true
  reject_old_samples_max_age: 168h
compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true
  delete_request_store: filesystem
analytics:
  reporting_enabled: false

The global retention target is 168 hours. Compactor retention, delete_request_store and persistent working data are required for cleanup, which is asynchronous. The old-sample admission limit serves a different purpose from deleting stored entries. Official documentation states retention changes are not retroactive; reducing the duration is not an emergency disk-reclamation plan. Monitor the WAL filesystem and storage warnings as well as chunks, indexes and compactor data.

Compose services and private access

services:
  loki:
    image: grafana/loki@sha256:0db216b630ae1f348a1584c67f3cead8b4c1fc839169a39501db664744158e28
    user: 10001:10001
    command:
    - -config.file=/etc/loki/config.yml
    volumes:
    - ./loki.yml:/etc/loki/config.yml:ro
    - ./state/loki:/var/lib/loki
    restart: unless-stopped
  alloy:
    image: grafana/alloy@sha256:8ab871206bd2d8f89277f53abffee5f452a293980788d52f85fa183a87a505a4
    user: 473:473
    group_add:
    - ${LOG_READ_GID:?Set the host log-reader group GID}
    - ${JOURNAL_GID:?Set actual systemd-journal group GID}
    - ${ADM_GID:?Set actual adm group GID}
    command:
    - run
    - --server.http.listen-addr=127.0.0.1:12345
    - --storage.path=/var/lib/alloy
    - /etc/alloy/config.alloy
    volumes:
    - ./config.alloy:/etc/alloy/config.alloy:ro
    - ./state/alloy:/var/lib/alloy
    - /var/log/nginx:/logs/nginx:ro
    - /var/www/app/storage/logs:/logs/app:ro
    - ${JOURNAL_DIR:?Set existing readable host journal directory}:/journal:ro
    - /etc/machine-id:/etc/machine-id:ro
    depends_on:
    - loki
    restart: unless-stopped
  grafana:
    image: grafana/grafana@sha256:488984b716c0862639f91a698adbe2862677a5a07bad42da5f03f3030883b205
    user: 472:472
    ports:
    - 127.0.0.1:3000:3000
    volumes:
    - ./state/grafana:/var/lib/grafana
    - ./secrets/grafana_admin_password:/run/secrets/grafana_admin_password:ro
    environment:
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
      GF_AUTH_ANONYMOUS_ENABLED: 'false'
      GF_USERS_ALLOW_SIGN_UP: 'false'
    depends_on:
    - loki
    restart: unless-stopped

Save the services as docker-compose.yml and prepare all source paths and configuration files. Set LOG_READ_GID, JOURNAL_GID, ADM_GID and JOURNAL_DIR before running docker compose config. Validate with docker compose run --rm --no-deps loki -config.file=/etc/loki/config.yml -verify-config=true and docker compose run --rm --no-deps alloy validate /etc/alloy/config.alloy. The selected image entrypoints are /usr/bin/loki and /bin/alloy, so Compose replaces their default arguments with these commands; do not add an unrelated shell or entrypoint. Loki’s verification flag checks the config and exits. Start in a test environment with docker compose up -d, inspect logs and trace actual source delivery. depends_on does not establish readiness. Use a trusted SSH tunnel for Grafana; its password file initializes a new admin account rather than rotating an existing database password.

The official Loki documentation covers storage and upgrade behavior. Validate the pinned Loki and Alloy configurations in an isolated test environment, then exercise the selected pipeline with synthetic source files and queries. Configuration validation alone does not establish Docker container execution, actual host permission policy, Grafana UI behavior, TLS, notification delivery or seven-day deletion. Complete those checks in the target test environment before production use.

Collect Selected Logs with Alloy

Alloy replaces Promtail for this new deployment. Limit read access to the files needed for operations; the four selected files and journal directory do not require the Docker socket or a mount of all /var/log. The example preserves log text and uses only bounded source labels. Credentials and unnecessary personal information must be excluded or appropriately redacted at the source; avoiding an indexed label does not remove a secret from the log body.

Define the source data contract

Nginx access.log must use standard combined format; error.log remains separate. orders.log contains one JSON object per line with RFC3339 time, lowercase level and message. worker.log starts each event with a line such as 2026-11-04T12:00:00Z ERROR job failed; following stack frames join that event. The selected journal directory supplies systemd entries. Adapt actual paths and formats explicitly, and set every vps-01 label to a stable host identity; a container hostname is not assumed to identify the VPS.

Alloy file sources and timestamp processing

// Alloy v1.21.3. Set the real, stable VPS identity in both sources.
loki.source.file "nginx" {
  targets = [
    {__path__ = "/logs/nginx/access.log", job = "nginx", log_type = "access", env = "prod", host = "vps-01"},
    {__path__ = "/logs/nginx/error.log", job = "nginx", log_type = "error", env = "prod", host = "vps-01"},
  ]
  file_match { enabled = true }
  forward_to = [loki.process.logs.receiver]
}
loki.source.file "app" {
  targets = [
    {__path__ = "/logs/app/orders.log", job = "app", service = "webshop", env = "prod", host = "vps-01"},
  ]
  file_match { enabled = true }
  forward_to = [loki.process.logs.receiver]
}
loki.process "logs" {
  // Nginx combined access format only; error logs remain untouched.
  stage.match {
    selector = "{job=\"nginx\", log_type=\"access\"}"
    stage.regex {
      expression = `^\S+ \S+ \S+ \[(?P<event_time>[^\]]+)\] "[^"]*" [0-9]{3} [0-9]+ "[^"]*" "[^"]*"$`
    }
    stage.timestamp {
      source = "event_time"
      format = "02/Jan/2006:15:04:05 -0700"
      action_on_failure = "skip"
    }
  }
  // One JSON object per line; time must be RFC3339/RFC3339Nano.
  stage.match {
    selector = "{job=\"app\"}"
    stage.json { expressions = {event_time = "time"} }
    stage.timestamp {
      source = "event_time"
      format = "RFC3339Nano"
      action_on_failure = "skip"
    }
  }
  stage.label_drop { values = ["filename"] }
  forward_to = [loki.write.local.receiver]
}
loki.write "local" {
  endpoint { url = "http://loki:3100/loki/api/v1/push" }
}

loki.relabel "journal_units" {
  forward_to = []
  rule {
    source_labels = ["__journal__systemd_unit"]
    target_label = "unit"
  }
}
loki.source.journal "system" {
  path = "/journal"
  max_age = "12h"
  labels = {job = "systemd", host = "vps-01", env = "prod"}
  relabel_rules = loki.relabel.journal_units.rules
  forward_to = [loki.write.local.receiver]
}

loki.source.file "worker" {
  targets = [
    {__path__ = "/logs/app/worker.log", job = "worker", service = "worker", env = "prod", host = "vps-01"},
  ]
  file_match { enabled = true }
  forward_to = [loki.process.worker.receiver]
}
loki.process "worker" {
  stage.multiline {
    firstline = `^[0-9]{4}-[0-9]{2}-[0-9]{2}T`
    max_wait_time = "3s"
    max_lines = 128
  }
  stage.regex {
    expression = `(?s)^(?P<event_time>[^ ]+) .*`
  }
  stage.timestamp {
    source = "event_time"
    format = "RFC3339Nano"
    action_on_failure = "skip"
  }
  stage.label_drop { values = ["filename"] }
  forward_to = [loki.write.local.receiver]
}

Valid combined-access and JSON timestamps become event timestamps. If extraction fails, skip retains the source timestamp, normally the file-read time; malformed JSON remains in the raw stream. This keeps bad records visible but requires a separate parser-error check. filename is removed from indexed labels, and user IDs, paths, messages and levels are not indexed. JSON fields remain in the original line for query-time parsing. No blanket debug drop or sampling policy is enabled here.

The old Promtail configuration guide remains a legacy migration reference. Use Alloy migration/conversion documentation for existing deployments and inspect all conversion warnings, source paths, labels, timestamps and offsets. Automatic conversion is not proof of correct delivery or zero loss/duplication. The journal and worker components above explicitly collect the selected journal and worker.log. Check the journal mount and reader permissions, and adapt the worker first-line format, wait and line limits to actual input; component validation alone does not establish successful collection.

Separate Native Systemd Alternative

This alternative is for a new Linux amd64 installation. Prepare the preceding loki.yml and config.alloy in the working directory and skip Compose; do not run both installations on the same ports. It requires root Bash, curl, unzip, sha256sum and systemd. Verify adm/systemd-journal groups and read access to the real Nginx/application files; add only the necessary actual log-reader group to SupplementaryGroups where needed. The script verifies official archive digests, installs root-owned binaries and creates separate service accounts. It changes Loki HTTP to loopback, translates Alloy mounts to actual host paths and sets the Loki endpoint to 127.0.0.1. An empty native journal path discovers /var/log/journal and /run/log/journal.

# Root Bash; new native installation. Keep the prepared loki.yml and config.alloy in the current directory.
set -eu
source_dir=$PWD
workdir=$(mktemp -d /tmp/loki-alloy.XXXXXX)
cd "$workdir"
curl --fail --location --proto =https --tlsv1.2 -o loki.zip https://github.com/grafana/loki/releases/download/v3.7.9/loki-linux-amd64.zip
printf '%s  %s\n' '17a5fae28ce733b3560c27c76620920b169bd2aea9473947b733b511f92e81dd' 'loki.zip' | sha256sum --check -
unzip loki.zip
install -o root -g root -m 0755 loki-linux-amd64 /usr/local/bin/loki
curl --fail --location --proto =https --tlsv1.2 -o alloy.zip https://github.com/grafana/alloy/releases/download/v1.21.3/alloy-linux-amd64.zip
printf '%s  %s\n' '50d23af2d0f98c00beac6ce9a365693e60f87fb76b6cad6fe1d161e80588af6d' 'alloy.zip' | sha256sum --check -
unzip alloy.zip
install -o root -g root -m 0755 alloy-linux-amd64 /usr/local/bin/alloy
cd "$source_dir"
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin loki
useradd --system --user-group --no-create-home --shell /usr/sbin/nologin alloy
usermod -aG adm,systemd-journal alloy
install -d -o root -g loki -m 0750 /etc/loki
install -d -o root -g alloy -m 0750 /etc/alloy
install -d -o loki -g loki -m 0700 /var/lib/loki
install -d -o alloy -g alloy -m 0700 /var/lib/alloy
sed 's|^  http_listen_address: 0\.0\.0\.0$|  http_listen_address: 127.0.0.1|' loki.yml > /etc/loki/config.yml
sed -e 's|http://loki:3100|http://127.0.0.1:3100|g' \
    -e 's|/logs/nginx|/var/log/nginx|g' \
    -e 's|/logs/app|/var/www/app/storage/logs|g' \
    -e 's|path = "/journal"|path = ""|' config.alloy > /etc/alloy/config.alloy
chown root:loki /etc/loki/config.yml
chown root:alloy /etc/alloy/config.alloy
chmod 0640 /etc/loki/config.yml /etc/alloy/config.alloy
cat > /etc/systemd/system/loki.service <<'EOF'
[Unit]
Description=loki logging service
Wants=network-online.target
After=network-online.target

[Service]
User=loki
Group=loki
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/config.yml
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536
UMask=0027
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
EOF
cat > /etc/systemd/system/alloy.service <<'EOF'
[Unit]
Description=alloy logging service
Wants=network-online.target
After=network-online.target

[Service]
User=alloy
Group=alloy
SupplementaryGroups=adm systemd-journal
ExecStart=/usr/local/bin/alloy run --server.http.listen-addr=127.0.0.1:12345 --storage.path=/var/lib/alloy /etc/alloy/config.alloy
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536
UMask=0027
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
EOF
chown root:root /etc/systemd/system/loki.service /etc/systemd/system/alloy.service
chmod 0644 /etc/systemd/system/loki.service /etc/systemd/system/alloy.service
runuser -u loki -- /usr/local/bin/loki -config.file=/etc/loki/config.yml -verify-config=true
runuser -u alloy -- /usr/local/bin/alloy validate /etc/alloy/config.alloy
systemd-analyze verify /etc/systemd/system/loki.service /etc/systemd/system/alloy.service
systemctl daemon-reload
systemctl enable --now loki alloy
systemctl --no-pager --full status loki alloy

Install native Grafana separately from its official package, bind it to 127.0.0.1:3000 and use http://127.0.0.1:3100 as its Loki data-source URL. Change the initial administrator password. Inspect journalctl -u loki -u alloy and http://127.0.0.1:3100/ready; readiness does not prove actual source collection. After config changes repeat the Loki/Alloy checks and restart the affected service only on success; unit changes require daemon-reload first. Use systemctl stop for stopping and enable/disable for boot behavior. LimitNOFILE=65536 preserves the example lifecycle setting without guaranteeing capacity. Keep Compose and native data directories separate; migration of existing data needs its own backup and rollback plan.

Explore the Logs and Verify Query Meaning

First send controlled records through the selected source files and find them in Loki with the expected host, timestamp and content. Then create dashboards from queries whose behavior you have checked. A deployment annotation helps establish a timeline, but a nearby error spike alone does not prove the release caused it; correlate application errors and metrics before choosing a corrective action.

Add the Loki data source

In Grafana open Connections → Data sources, add Loki and use http://loki:3100 for this Compose network. Save & test, then select it in Explore and choose the test event time range. No tenant header is needed with this example’s auth_enabled: false. For a different native or remote deployment use its verified address and authentication path; localhost inside Grafana’s container would refer to Grafana itself.

Parse the real status field

A text search for 500 can match a URL, User-Agent or byte count. The anchored line filter below first checks the full combined record shape; pattern then captures the entire request field as one value, so a dash or empty request does not hide a 5xx response. status selects only 500–599. Named captures and literal separators must match the actual source format. These are query-time fields, not new indexed labels.

{job="nginx", log_type="access"} |~ `^\S+ \S+ \S+ \[[^\]]+\] "[^\"]*" [0-9]{3} [0-9]+ "[^\"]*" "[^\"]*"$` | pattern `<remote_addr> <_> <_> [<_>] "<request>" <status> <_> "<_>" "<_>"` | status=~"5[0-9][0-9]"

Pattern can stop with partial captures when the input does not match; the pinned parser does not automatically set __error__ for that mismatch. A missing-status filter catches only failures before status was extracted. Use the full-format negative filter below to find malformed records, including a truncated tail after a valid-looking status. Adapt both expressions if the actual log format changes.

{job="nginx", log_type="access"} !~ `^\S+ \S+ \S+ \[[^\]]+\] "[^\"]*" [0-9]{3} [0-9]+ "[^\"]*" "[^\"]*"$`

The following rate query reports average selected 5xx entries per second over five minutes by host. count_over_time gives a window count instead; neither alone gives an error percentage.

sum by (host) (rate({job="nginx", log_type="access"} |~ `^\S+ \S+ \S+ \[[^\]]+\] "[^\"]*" [0-9]{3} [0-9]+ "[^\"]*" "[^\"]*"$` | pattern `<remote_addr> <_> <_> [<_>] "<request>" <status> <_> "<_>" "<_>"` | status=~"5[0-9][0-9]" [5m]))

The LogQL documentation explains selectors, parsers and metric functions. Extracting raw paths during a query can still create a large aggregation; use an application-defined bounded route template when per-route analysis is required. A latency panel needs a numeric duration field actually present in the source plus the correct units and unwrap operation; standard combined access logs do not include request_time.

Dashboards with explicit data coverage

A useful overview separates access-response rates, application error entries, parser failures and collection health. Include active stream/volume, storage headroom and ingestion errors alongside service panels. A missing-log panel requires an expected source inventory and a regular heartbeat or other known activity; an idle application can correctly produce no logs. Monitor the collector, Loki and Grafana independently where a single-machine failure would otherwise silence every alert.

Record release annotations with the actual deployment identifier and time. Link to the relevant change and service so a reviewer can compare before/after behavior. Keep enough surrounding context to investigate alternative explanations such as dependency failure or traffic changes; an annotation is useful evidence, not a causal conclusion.

Retention, Cardinality and Storage Planning

Manage three distinct retention scopes: original source-file rotation, Loki storage and independent backups. Deleting a source before it is read creates a gap; keeping copies indefinitely can conflict with the intended deletion policy. Size storage for measured ingestion and query patterns, compaction/WAL work, expected outages and restore operations, not just the average compressed chunk volume.

Choose a retention policy you can operate

Seven days is a technical example. Determine the investigation window, privacy requirements and applicable record obligations before changing it. Compactor cleanup is asynchronous, requires persistent markers and is not guaranteed to free space immediately. Moving to S3 needs endpoint/credentials, bucket policy, preserved schema periods and a migration/recovery plan; existing filesystem data does not move automatically. An object lifecycle must not delete objects while Loki still needs them.

Keep indexed labels bounded

Use a small set of meaningful, stable job, host, env, service, log_type and journal unit values. Do not index request IDs, user IDs, raw paths or arbitrary messages; every distinct combination creates a stream. Even a bounded dynamic label can split low-volume streams into small chunks, so add it only for a demonstrated need. Necessary identifiers may remain in controlled log fields or supported structured metadata, subject to data-minimization and access rules; credentials should never be logged.

Drop or sample only with a defined purpose

Reduce unnecessary debug output at the application after identifying records needed for security, accounting and incident investigation. Test any collector filter against both expected matches and important exceptions before deployment. Sampling changes the population used by counts, rates and alerts; document that limitation and preserve required records. Compression cannot compensate for unlimited ingestion, and a dashboard merely hiding a pattern does not reduce stored data.

Rotation, persistent state and recovery

Test rename/create, compression and copytruncate behavior with the actual sources. Persist Alloy’s –storage.path directory; file offsets do not alone prove that Loki durably accepted every line. Removing and re-adding a target can cause it to be read again. Watch disk bytes, inodes, WAL protection and rejected writes. Back up Loki’s consistent index/chunk/WAL state and compactor markers, Grafana’s DB/provisioning, collector state and necessary secrets with controlled access. Verify a restore separately; copying active directories without a consistency method is not a guaranteed recovery point.

Create Grafana-Managed Alerts

Choose a small set of actionable rules based on expected activity and business impact. A burst of 404 responses during a campaign, for example, may need investigation without paging an operator. Define the affected service, unit, evaluation period and first response step so an alert explains both the observation and what should happen next.

Use a numeric query and explicit evaluation behavior

Open Alerts & IRM → Alert rules and create a Grafana-managed rule with a Loki metric query. Specify the evaluation interval, pending period, No Data and Error behavior, then configure a contact point. Choose Advanced notification routing to use the label-based policy tree; Default routes directly to the selected contact point without using that tree. This example defines no Loki ruler rules or external Alertmanager delivery path; localhost:9093 inside the Loki container is not assumed to be a running notification service.

For webshop, the following query counts JSON error lines over ten minutes. An example threshold above 100 is a starting illustration, not a universal production setting. level is extracted at query time; it is not an indexed source label.

sum by (service) (count_over_time({job="app", service="webshop"} | json | __error__="" | level="error" [10m]))

For an expected heartbeat source, absent_over_time can detect that a specific host produced no entries. It does not automatically enumerate every missing host. Retry alerts must parse the JSON message field instead of assuming msg is an indexed label:

sum(count_over_time({job="app", service="webshop"} | json | __error__="" | message="Retrying" [5m]))

Apply a meaningful threshold and pending interval; an empty vector is different from a measured zero. Monitor malformed JSON and collection health separately so parser or transport failure does not silently look like recovery.

The Grafana alerting documentation describes rule evaluation and notification routing. Test the contact point, then trigger a controlled log event to exercise the complete query → rule → policy → destination path. Email requires working SMTP settings and remote integrations need outbound connectivity. A successful contact-point test alone does not prove the rule will fire or route correctly.

Keep the operational response clear

Attach the service, observed value, Explore/panel link and a short initial investigation step. Use maintenance mute timings deliberately and test recovery notifications. Review false positives against the actual data population, not by raising thresholds blindly. Keep an independent check for evaluator availability; a stopped Grafana process cannot reliably notify through its own rules.

Protect Access and Sensitive Log Data

Log text can contain addresses, identifiers and internal stack traces even when those fields are not indexed. Minimize the data at source, restrict retention and grant access according to the investigation need. Read-only source mounts limit container writes but do not make the copied log content nonsensitive. Avoid broadly sharing host directories, credentials or a common administrator account.

Keep Loki unpublished in this local Compose design and access Grafana through a trusted SSH tunnel or a correctly configured HTTPS reverse proxy. For an HTTPS proxy set Grafana’s real domain/root_url and secure cookies while its backend remains HTTP on loopback; support Grafana Live WebSocket Upgrade/Connection forwarding. Use individual authentication and Grafana permissions. Remote collectors require a protected TLS/VPN path with appropriate authentication; an IP allowlist alone does not encrypt traffic. In multi-tenant mode a trusted boundary must set the right X-Scope-OrgID consistently for ingestion and queries. Keep supported binaries updated through tested release changes.

Make Investigations Reproducible

Store nonsecret configuration and version choices in source control, with sample records and expected query results. Keep runbooks beside dashboards so another operator can distinguish “zero errors,” “no requests,” “parser failure” and “collector offline.” These distinctions matter more than adding panels whose underlying source or units are unclear.

Derived fields can link a permitted identifier to an authorized tracing or administration system; verify access and avoid exposing sensitive identifiers in third-party URLs. Add release annotations and concise dashboard instructions. Use a separate investigation view for noisy patterns while deciding whether to change application output or collection policy; moving a pattern to another dashboard does not archive, filter or delete its stored lines.

Troubleshoot Missing or Incorrect Logs

Start from one known synthetic event and follow its path: actual source file, reader permissions, Alloy offset and delivery state, Loki admission, query labels/time window, then Grafana evaluation. Change one layer at a time and keep the evidence. A green component-health indicator does not establish end-to-end delivery.

Check the real mounted path and file format, then Alloy’s own logs and ingestion/retry counters. Test connectivity to the correct Loki endpoint; an ordinary GET to the push URL is not a valid ingestion request. Search for a controlled source line with the expected timestamp and labels. Do not switch a container endpoint to localhost as a generic repair, because it refers to that container. During Promtail migration compare position handling and use explicit gap/duplicate tests before retiring the previous collector.

For Loki, inspect WAL, compactor, index, disk and admission warnings. Loki 3.7.9 defaults to a 90% WAL-filesystem usage threshold for throttling incoming writes; address capacity and retention instead of simply disabling that protection. Distinguish 429 rate limits from rejected timestamps or malformed data. Lowering an ingestion limit is not automatic backpressure on the application; it can increase retries and eventual loss. Narrow query ranges and inspect active stream cardinality, but also measure I/O and query concurrency before attributing every slow query to labels. Parse failures should be visible in their own check.

Use an Incident Timeline Without Inventing Causation

In an illustrative incident, actual access-status 5xx counts rise after a release. Compare those events with application JSON errors and real database/connection-pool metrics. A database error near the release suggests an investigation path; it does not establish that reverting files will safely reverse database migrations or external side effects. Choose a tested rollback or forward fix that preserves current data, then verify that both user behavior and telemetry recover.

In another example, an expected heartbeat disappears after an update. Verify the service state, reader permissions, file path, collector delivery and query selection before restarting anything. If the application is simply idle, missing ordinary log lines may be expected. Record the actual failure and add a targeted check rather than claiming that every silent source is down.

Validate Changes Before Deployment

Alloy reads the selected files and journal entries, applies the documented timestamp handling and forwards bounded-label streams to private Loki storage. Grafana queries those streams and evaluates explicit rules. Validate configuration with the exact supported versions, then run representative normal, malformed, 5xx and missing-data cases. Follow those with permission, restart, rotation, outage and real notification tests in the target environment.

The journal example sets max_age=”12h” and unit labeling from __journal__systemd_unit. The lookback is relative to process start, not a retention policy or a promise to recover unavailable history. Persist Alloy storage.path for cursor state; records without the systemd-unit field may lack unit. The explicit job=”systemd” value is retained. Restrict selected services with matches if transient unit names would create excessive cardinality. Journal realtime timestamps are used when available. The separate worker pipeline uses a date-prefix firstline, with a three-second flush wait and a 128-line block bound; long traces split. It extracts RFC3339 time from the first line and falls back to source time if parsing fails. Test orphan continuation lines, timeouts, splitting and restart behavior with actual inputs; offsets/cursors do not prove durable Loki delivery.

Operate a Verified Logging Workflow

The result should be a traceable chain from a selected source to a retained event and a correctly routed alert. Assign ownership for software updates, access, capacity, source coverage and restore tests. Start with the sources and thresholds you can validate, then expand from observed needs. Neither a single server nor a short retention window guarantees low cost, complete evidence or uninterrupted operation.

Before considering the rollout complete, record at least one normal event, a true 5xx response, an application JSON error and an expected-source silence case. Verify their units, labels and timestamps, and test recovery after the relevant interruption scenarios. Repeat the affected checks when a source format, version, policy or path changes.

Keep the Loki documentation for schema/storage details, the legacy Promtail documentation for interpreting old configurations during Alloy migration, and the Grafana alerting documentation for rule and notification behavior. Use the supported Alloy references for the new collector, and preserve test evidence together with the exact deployed versions.