Nginx Canary Deployment on a VPS
A canary deployment sends limited traffic to a new release while a stable release remains available. On a single VPS, both versions share CPU, memory, storage and the host failure domain. Check capacity for simultaneous processes before rollout. Canary observation complements staging and recovery preparation; it does not prove a release safe.
This guide covers two app versions running side-by-side, Nginx weighted routing to send a small share of requests to the canary, health checks to help detect failures, and traffic rollbacks when needed. Routing changes do not undo database changes or other application side effects, so both versions must remain compatible during the rollout.
What a Canary Deploy Looks Like on a Single VPS
Run stable v1 and canary v2 as separate, identifiable application processes. Start with v2 unavailable to general routing, test it directly, then increase its share while comparing errors, latency and important user operations against v1. Define minimum sample sizes and stop criteria beforehand. A traffic rollback does not undo completed writes, payments or queued jobs.
Nginx receives public traffic and selects a backend. In the weighted pool, configured retry rules may permit another attempt on a different available peer. A forced single-peer canary route has no stable peer to retry. Keep both paths under one enable/disable gate, and verify the actual backend rather than assuming a successful client response means canary succeeded.
For example, a hypothetical rollout could send roughly 5% of requests to v2. A rise in canary 5xx responses would be a reason to pause or roll back traffic, investigate a possible cache-key incompatibility, and resume only after validating a fix.
Prepare Two Application Versions
Two app processes, two ports
This example uses loopback HTTP on 127.0.0.1:5001 for v1 and 127.0.0.1:5002 for v2. Check that those ports and the local configuration probe port 18089 are available. Manage the processes through your own service/container setup and keep backend ports private. The canary /healthz must return exactly HTTP 200 and the body ok version=v2 followed by one LF only when the chosen readiness contract is met.
Nginx Routes Requests
Nginx will sit in front and route traffic with weighted round-robin. You can fine-tune weights to approximate percentages. The weights govern request selection, not sticky user assignment; short samples, retries, and unavailable peers can change the observed share.
Health checks
Open-source Nginx provides passive failure accounting in a multi-peer upstream through max_fails and fail_timeout. These parameters are ignored for a single-server group. Our separate curl checker tests the canary directly and attempts to close the shared gate after three consecutive failures. This external probe does not require NGINX Plus and is not comprehensive application monitoring.
Traffic rollbacks
The helper below validates positive integer weights, serializes cooperating writers, replaces one state file atomically and attempts restoration after validation/reload failure. That state file controls both the weighted peer and the forced-cookie route. The helper reports whether a new configuration generation was observed; it cannot guarantee instant traffic cutover or recover application side effects.
Configure Nginx Weighted Routing
Merge the following fragment once into the existing http context. Create the state file shown later before validating this configuration; do not duplicate upstream or log_format names. Adapt the server to your existing HTTPS configuration. Applications must supply a trusted X-Release: v1 or X-Release: v2 response header. The loopback-only generation endpoint reports Nginx configuration state, not application readiness.
# Merge once into the existing http context; do not duplicate upstream names.
upstream app_stable { server 127.0.0.1:5001; }
upstream app_canary { server 127.0.0.1:5002; }
include /etc/nginx/canary-state.conf;
map "$canary_enabled:$cookie_canary" $backend_upstream {
default app_pool;
"1:1" app_canary;
}
log_format canary_json escape=json
'{"time":"$time_iso8601","status":$status,'
'"route":"$backend_upstream","upstream":"$upstream_addr",'
'"upstream_status":"$upstream_status","release":"$upstream_http_x_release",'
'"request_time":$request_time,"upstream_time":"$upstream_response_time"}';
server {
listen 80;
server_name example.com;
access_log /var/log/nginx/canary.json canary_json;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_connect_timeout 2s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;
proxy_pass http://$backend_upstream;
}
}
# Local configuration-generation probe, not application readiness.
server {
listen 127.0.0.1:18089;
server_name localhost;
access_log off;
location = /__canary_generation { return 200 "$canary_generation\n"; }
location / { return 404; }
}
The initial state keeps canary disabled. After enabling, its target weighted share is canary / (stable + canary): stable 100 with canary 1 is about 0.99%, 5 about 4.76%, 20 about 16.67%, 50 about 33.33%, and 100 about 50%. Cookie overrides, unavailable peers, retries and short samples change observed shares. Weight 0 is invalid as an upstream weight; use the helper’s disable action, which writes down and closes the common gate.
For a hash-bucket alternative, split_clients maps a key into a configured fraction of the hash space. The same key and unchanged split remain deterministic, but shared/changing IPs and changing user agents do not identify stable individual users. Unequal activity also means bucket fractions differ from request-volume fractions. The optional example later replaces the routing map while retaining the shared disable gate.
The default example recognizes an explicitly sent Cookie: canary=1 only while canary is enabled. It does not create that cookie or restrict access: any client can send it. Use separate authentication for team-only testing. Enabling this gate also opens the selected weighted share to ordinary traffic; direct backend or protected staging tests should come first. Browser cookie scope, HTTPS/Secure, SameSite and applicable HttpOnly settings belong to the application’s test flow.
Identify Canary Attempts in Logs
The JSON format records the selected pool separately from actual upstream attempts. release comes from the final upstream response header; it is neither a complete retry history nor proof that an unauthenticated header is trustworthy. The following illustrative records show a direct canary success and a canary failure followed by a stable success:
{"time":"2026-11-04T12:00:00+03:00","status":200,"route":"app_pool","upstream":"127.0.0.1:5002","upstream_status":"200","release":"v2","request_time":0.120,"upstream_time":"0.115"}
{"time":"2026-11-04T12:00:01+03:00","status":200,"route":"app_pool","upstream":"127.0.0.1:5002, 127.0.0.1:5001","upstream_status":"503, 200","release":"v1","request_time":0.025,"upstream_time":"0.010, 0.015"}
Choose one bounded observation window before counting. This jq example assumes every timestamp and both boundaries use the same ISO 8601 format and UTC offset, as in this single Nginx log. Replace the example boundaries with your real window; normalize timestamps first if aggregating mixed offsets. Run analysis from a private directory with permission to read the log.
set -euo pipefail
umask 077
jq -c --arg start_time '2026-11-04T12:00:00+03:00' \
--arg finish_time '2026-11-04T12:10:00+03:00' \
'select(.time >= $start_time and .time < $finish_time)' \
/var/log/nginx/canary.json > ./canary-window.jsonl
Verify the window contains actual canary attempts and enough observations for the rollout decision. Empty results can mean no canary traffic, the wrong window or missing logs; they are not a clean bill of health. Keep the same interval for stable and canary comparisons. Error logs may contain additional details and require their own access and retention controls.
Passive Failure Accounting and Retries
In the weighted pool, the two retry directives below are already present in the main example. They illustrate the existing settings; do not add a second location block. Failure accounting depends on the configured conditions, so HTTP 500 or business-logic errors are not automatically covered by the listed HTTP 502/503/504 conditions.
# Already inside the main location / block; do not duplicate it.
proxy_next_upstream error timeout http_502 http_503 http_504;
proxy_next_upstream_tries 2;
These rules allow a retry on another available peer for the listed failures, in either direction—not only canary to stable. Retries cannot recover a response already sent to the client, and non-idempotent requests already sent upstream are normally not retried. Continue monitoring errors and latency rather than assuming retries hide every failure.
The upstream module documentation defines the single-server exception: max_fails/fail_timeout do not mark the only peer unavailable. Our fixed app_canary route therefore exposes failures directly until the common gate is closed. Two total attempts means at most one retry; choose timeouts and failure thresholds for your workload rather than treating the example values as universal safe limits.
An External Canary Readiness Checker
Save this guard as /usr/local/bin/canary-healthcheck.sh after preparing the helper below. It uses curl without environment proxies or redirect following, requires exactly HTTP 200 plus ok version=v2 and one LF, and returns as soon as any attempt is healthy. Three consecutive failures within a run trigger disable; a helper failure propagates to systemd.
#!/usr/bin/env bash
set -euo pipefail
umask 077
body=$(mktemp)
trap 'rm -f -- "$body"' EXIT
for attempt in 1 2 3; do
if code=$(curl --noproxy '*' -sS --connect-timeout 1 --max-time 2 \
-H 'Host: example.com' -o "$body" -w '%{http_code}' \
'http://127.0.0.1:5002/healthz') &&
[[ "$code" == 200 ]] && cmp -s "$body" <(printf 'ok version=v2\n'); then
exit 0
fi
if (( attempt < 3 )); then sleep 1; fi
done
echo "Three failed readiness probes; attempting canary disable" >&2
/usr/local/bin/canary-weight.sh disable
Save the service and timer below as two separate files at the indicated paths. The timer is relative to completion of the previous service run:
# Save as /etc/systemd/system/canary-healthcheck.service
[Unit]
Description=Canary readiness check and traffic-disable attempt
[Service]
Type=oneshot
ExecStart=/usr/local/bin/canary-healthcheck.sh
TimeoutStartSec=180s
# Save separately as /etc/systemd/system/canary-healthcheck.timer
[Unit]
Description=Run canary readiness check after each completed run
[Timer]
OnBootSec=30s
OnUnitInactiveSec=10s
AccuracySec=1s
[Install]
WantedBy=timers.target
Install both scripts with root-controlled ownership and executable permissions, for example chmod 0750 /usr/local/bin/canary-weight.sh /usr/local/bin/canary-healthcheck.sh. After writing unit files, run systemctl daemon-reload, then systemctl enable --now canary-healthcheck.timer. Verify systemctl status canary-healthcheck.timer and journalctl -u canary-healthcheck.service. OnUnitInactiveSec waits about 10 seconds after a run finishes; AccuracySec, probe duration, lock waits and reloads add delay. This is not a fixed ten-second rollback guarantee. Test with the actual distribution’s unit and Nginx service.
Create /etc/nginx/canary-state.conf with the following initial content before first loading the main configuration. Keep it a regular file in a directory writable only by authorized administrators. It contains the weighted upstream plus both maps, so peer selection and the forced route use one state transition.
# generation: initial
upstream app_pool {
zone app_pool 64k;
keepalive 16;
server 127.0.0.1:5001 weight=100 max_fails=3 fail_timeout=10s;
server 127.0.0.1:5002 down;
}
map $host $canary_enabled { default 0; }
map $host $canary_generation { default "initial"; }
After installing and testing the helper below, open a small share with its set action. Stable remains at weight 100; a canary weight of 5 is about 4.76% of eligible weighted selections. The same action enables the forced-cookie path. It is not an isolated team-only mode.
/usr/local/bin/canary-weight.sh set 5
Disable switches the canary peer to down and closes the common gate, including the optional split route shown later. The helper already validates and reloads; do not add a second uncoordinated reload command:
/usr/local/bin/canary-weight.sh disable
Save this helper as /usr/local/bin/canary-weight.sh. It requires Bash, curl, util-linux flock, GNU coreutils, sed and the guard’s cmp utility (from diffutils on GNU systems). The example assumes a systemd-managed Nginx service, with authorized access to the state file, lock directory and service. It accepts only weights 1–10000 without leading zeros; enable selects weight 1 and disable writes down. All config writers and reloaders must honor this same lock.
#!/usr/bin/env bash
set -euo pipefail
umask 077
STATE="/etc/nginx/canary-state.conf"
LOCK="/run/lock/dchost-canary.lock"
PROBE="http://127.0.0.1:18089/__canary_generation"
positive_weight() {
[[ "$1" =~ ^[1-9][0-9]{0,4}$ ]] && (( 10#$1 <= 10000 ))
}
stable=100
case "${1:-}" in
set)
[[ $# == 2 ]] && positive_weight "$2" || { echo "Usage: canary-weight.sh set INTEGER_1_TO_10000" >&2; exit 64; }
canary=$2 ;;
enable|disable)
[[ $# == 1 ]] || { echo "Unexpected argument" >&2; exit 64; }
if [[ "$1" == enable ]]; then canary=1; else canary=0; fi ;;
*) echo "Usage: canary-weight.sh {set N|enable|disable}" >&2; exit 64 ;;
esac
exec 9>"$LOCK"
flock -x -w 10 9 || { echo "Another canary update holds the lock" >&2; exit 1; }
[[ -f "$STATE" && ! -L "$STATE" ]] || { echo "Expected a regular state file" >&2; exit 1; }
old_generation=$(sed -n 's/^# generation: //p' "$STATE")
[[ "$old_generation" =~ ^[A-Za-z0-9-]+$ ]] || { echo "Invalid old generation" >&2; exit 1; }
backup=""; candidate=""; probe_body=""; dirty=0
wait_generation() {
local wanted=$1 code attempt
for attempt in {1..10}; do
if code=$(curl --noproxy '*' -sS --connect-timeout 1 --max-time 1 \
-H 'Connection: close' -o "$probe_body" -w '%{http_code}' "$PROBE") &&
[[ "$code" == 200 && "$(cat "$probe_body")" == "$wanted" ]]; then
return 0
fi
sleep 0.2
done
return 1
}
cleanup() {
local rc=$? restore
trap - EXIT HUP INT TERM
if (( dirty )); then
echo "Restoring previous state from $backup" >&2
if restore=$(mktemp "${STATE}.restore.XXXXXX") &&
cp -p -- "$backup" "$restore" && mv -fT -- "$restore" "$STATE" &&
nginx -t && timeout 30 systemctl reload nginx && wait_generation "$old_generation"; then
dirty=0
else
echo "RESTORE NOT CONFIRMED. Preserve $backup and reconcile disk/runtime manually." >&2
rc=2
fi
fi
[[ -z "$candidate" ]] || rm -f -- "$candidate"
[[ -z "$probe_body" ]] || rm -f -- "$probe_body"
if (( ! dirty )) && [[ -n "$backup" ]]; then rm -f -- "$backup"; fi
exit "$rc"
}
trap cleanup EXIT
trap 'exit 129' HUP
trap 'exit 130' INT
trap 'exit 143' TERM
backup=$(mktemp "${STATE}.backup.XXXXXX")
cp -p -- "$STATE" "$backup"
candidate=$(mktemp "${STATE}.new.XXXXXX")
probe_body=$(mktemp)
generation="$(date +%s%N)-$$"
enabled=0
canary_line="server 127.0.0.1:5002 down;"
if (( canary > 0 )); then
enabled=1
canary_line="server 127.0.0.1:5002 weight=${canary} max_fails=3 fail_timeout=10s;"
fi
cat >"$candidate" <<EOF
# generation: $generation
upstream app_pool {
zone app_pool 64k;
keepalive 16;
server 127.0.0.1:5001 weight=${stable} max_fails=3 fail_timeout=10s;
$canary_line
}
map \$host \$canary_enabled { default $enabled; }
map \$host \$canary_generation { default "$generation"; }
EOF
chmod --reference="$STATE" "$candidate"
chown --reference="$STATE" "$candidate"
# Every config writer/reloader must obey the same lock during this transaction.
# Mark dirty before rename so a catchable signal cannot skip restoration.
dirty=1
mv -fT -- "$candidate" "$STATE"
if ! nginx -t; then echo "Candidate validation failed" >&2; exit 1; fi
if ! timeout 30 systemctl reload nginx; then echo "Reload command failed" >&2; exit 1; fi
if ! wait_generation "$generation"; then echo "New generation not observed" >&2; exit 1; fi
dirty=0
echo "Accepted generation=$generation stable=$stable canary=$canary; verify traffic and draining workers."
The candidate replaces the state file by a same-directory atomic rename. If validation, reload or generation observation fails, the EXIT handler restores the previous file and attempts to reload/observe its generation. Exit 1 reports a failed change; exit 2 reports unconfirmed restoration and retains the backup for manual disk/runtime reconciliation. A successful generation probe proves only that a worker using that generation replied: old workers can briefly still accept connections and continue in-flight work. This is not a transaction against SIGKILL, power loss or unrelated administrators; monitor convergence and test restoration before rollout. Here GNU timeout sends its default TERM signal after 30 seconds; without -k this is not a hard 30-second completion bound. Terminating the systemctl client does not prove that a manager-owned reload job was canceled. Bash can defer a trapped signal while waiting for a foreground command. If the oneshot service exceeds its 180-second startup limit, systemd normally terminates its control group and may later force-kill remaining processes; restoration may be interrupted. Inspect the service state, retained backup and running generation, and prepare for manual recovery.
Compare Canary and Stable Observations
Track real canary sample count, client-facing errors, upstream attempt errors and latency in the same bounded window. The following commands consume the window file created earlier and require jq. They are inspection tools, not complete alerting or statistical rollout decisions.
Count final client statuses where the last responding application identified itself as v2. This intentionally excludes canary failures that retried successfully on v1; inspect attempt records separately:
jq -r 'select(.release == "v2") | .status' ./canary-window.jsonl | sort | uniq -c
Inspect the largest total request times for final v2 responses, then repeat with v1 over the same interval. These include more than backend execution time; a few maxima neither establish a percentile nor prove a regression:
jq -r 'select(.release == "v2") | .request_time' ./canary-window.jsonl | sort -n | tail
Inspect all records that attempted the exact canary address, retaining the whole address/status sequence. This example splits the comma-separated list for the shown configuration; if your application or error handling adds internal redirects between groups, normalize Nginx’s colon-separated group structure before applying it. A matching line is not itself a retry count:
jq -r 'select(.upstream | split(",") | map(gsub("^ +| +$"; "")) | index("127.0.0.1:5002"))
| [.upstream, .upstream_status, .release, .status] | @tsv' ./canary-window.jsonl
Keep raw upstream addresses and variable-length attempt lists as log fields. Do not export every distinct list, request ID, user, IP or URL as a Prometheus label. If metrics are needed, normalize into bounded values such as stable/canary/unknown and a small status-class set, while retaining appropriate attempt accounting. Each label combination creates a time series with resource cost. The example access log omits paths, query strings, cookies, client IPs and user agents; audit other logs and their retention separately.
A Measured Canary Rollout
Use this as a starting cadence, adjusting observation periods to traffic volume and risk:
Before starting, confirm both process identities, the exact readiness response, trusted release headers, private backend reachability, capacity, logging and the timer. Test failed validation/reload and the retained-backup recovery path in an isolated environment. Directly test v2 before enabling any public traffic; a static proxy health endpoint is insufficient.
Step 1: enable weight 1 against stable 100, approximately 0.99% for eligible weighted requests. Verify actual v2 responses and exercise important flows with controlled test data. Wait for the predetermined sample size and health/latency criteria, rather than assuming a few quiet minutes prove success.
Step 2: consider weight 5, approximately 4.76%, only if the previous stage meets its criteria. Compare with stable during the same window; inspect retry attempts as well as successful final responses.
Step 3: weights 10 and 20 target approximately 9.09% and 16.67%, respectively. Recheck login, forms, pagination, queue processing and database connections as relevant. Use authorized test transactions that avoid unintended real payments, messages or inventory changes.
Step 4: weight 50 against stable 100 targets roughly one-third, not half. Observe enough actual canary requests and important operations; check latency distributions and resource saturation. Stop or roll back when agreed criteria fail, even if the basic readiness endpoint still passes.
Step 5: promote v2 only after sufficient observation. With stable still at weight 100, canary weight 100 is approximately a 50/50 split, not all-in. Before promotion, stop the health-check timer, wait for any already-running guard/helper invocation to finish, and verify its result and the running configuration. The existing helper regenerates the old backend addresses; running it unchanged after promotion can route traffic back to v1. Full promotion requires removing v1 from active routing, validating and reloading the configuration under the shared lock, and allowing existing work to drain before stopping v1. Keep v1 available for rollback while compatibility permits. Before re-enabling the timer, adapt the helper targets, guard release/port contract and initial state file to the next rollout’s stable/canary layout, then test disable and recovery. Low traffic calls for more observation, not automatically skipping stages.
Session, Cache and Database Compatibility
Both versions must understand the same session state. A shared Redis/database store alone is insufficient: verify serialization, cookie names/scopes, signing or encryption keys, authentication and CSRF behavior across versions. An IP hash cannot preserve continuity when addresses or upstreams change. Fix compatibility before relying on routing affinity.
Cache keys and values must remain readable by both versions during coexistence. If the format changes incompatibly, isolate it with versioned keys and test invalidation and fallback behavior. Delay cleanup until old readers and their rollback window are retired; a shared cache alone does not establish compatibility.
Use backward-compatible database changes and postpone destructive cleanup until the rollback window closes. Adding a column/table is not automatically safe: defaults, constraints, migration locks and new write formats can still break v1. Verify both readers and writers. Restoring a pre-release snapshot can discard later writes, so plan data recovery separately from traffic rollback. The application-consistent LVM backup guide covers backup preparation; perform an actual recovery test for your database setup.
Optional Hash Buckets with split_clients
The following alternative replaces the entire cookie/weighted routing map in the main http fragment. Keep the fixed app_stable/app_canary upstreams, shared state include and proxy server; do not add duplicate maps or disconnected v1/v2 pools. Cookie overrides no longer apply. The shared gate still disables all selected canary buckets:
# Alternative to the cookie/weighted routing map, inside http.
# Keep the shared state include, fixed app_stable/app_canary pools and proxy server.
split_clients "$remote_addr|$http_user_agent" $bucket {
5% canary;
* stable;
}
map "$canary_enabled:$bucket" $backend_upstream {
default app_stable;
"1:canary" app_canary;
}
The official split_clients documentation describes MurmurHash2 allocation of hash-space ranges. Here 5% of that space selects canary only while enabled. Identical IP/User-Agent inputs stay in the same bucket under the unchanged rule, but IPs can be shared or change and user agents can change. This does not promise 5% of users or requests. A suitable stable application identifier can improve cohort consistency, with appropriate privacy and authentication controls. Helper weights do not change the 5% bucket size; changing it is a separate controlled config update.
TLS and Graceful Reload Limitations
Adapt the HTTP routing example to your existing HTTPS server and verify certificate renewal, service reload and client connectivity. Graceful reload starts new workers and asks old ones to finish existing clients; it does not guarantee warmed connections or immediate draining. Observing a new generation can precede old listening sockets closing. Monitor the transition, retain old application processes while needed, and coordinate automated reloads with the same lock.
Behind a CDN or tunnel, direct loopback readiness checks help distinguish the canary application from cached or edge responses. Also test the real user path, Host/scheme forwarding, cache isolation and TLS separately. A healthy local response cannot establish end-to-end availability.
Retry and Timeout Tradeoffs
Limited retries may help certain transient failures, but can increase load, mask canary problems behind a successful stable response or duplicate effects if application semantics are unsafe. Observe actual attempt lists and adjust retry and timeout settings to the workload.
The shown configuration allows error, timeout and HTTP 502/503/504 conditions with at most two total attempts. It does not enable non_idempotent retries. A response already sent to the client cannot be repaired by switching peers. Single-peer forced/hash paths cannot retry on stable within their own pool. The guard’s three failed probes initiate a later config transition; they do not cancel requests already executing.
Review the upstream documentation and proxy retry semantics against your actual application. Rate/error budgets, circuit breakers and idempotency controls may complement the routing layer, but none should be assumed from this sample alone.
Protect Control Scripts and Private Endpoints
Keep administration authenticated and backend/config-probe ports private. A canary cookie is a client preference, not authorization. Limit readiness output, protect helper/state/unit ownership, and prevent unauthorized changes to the command search path used by privileged scripts. Configure log access, retention and rotation according to actual operational needs; the minimal access log does not sanitize other application or error logs.
Common Canary Configuration Errors
First, verify weight arithmetic against the currently configured stable baseline. 1:1 is approximately half; stable 100/canary 50 is approximately one-third. Validate exact positive integers and use down for disabling. Do not infer the deployed ratio from a helper command without checking active config and actual release samples.
Second, check keepalive behavior for the installed Nginx version. The weighted app_pool explicitly sets an idle cache of 16; fixed single-peer pools use that version’s defaults. Current documentation describes defaults changing in 1.29.7. HTTP/1.1 and a cleared Connection header support reuse, but do not prove connections remain warm through a reload. The idle cache limit is not a cap on all open upstream connections; measure behavior before tuning.
Third, verify sessions across versions instead of assuming shared storage or a cookie route solves continuity. Check cookie scope, serialization, key rotation and authentication behavior. Hash buckets can change when inputs or the rule change; they are not a substitute for compatibility.
Fourth, delay destructive database changes and test even additive migrations for constraints, defaults, write formats and locking. Keep cache formats mutually readable or version their keys. Rollback is useful only while v1 can handle the current data and session state.
A Canary Rollout Checklist
Adapt this checklist to your application, traffic and recovery requirements:
Prepare: separately managed v1/v2 processes, spare capacity, exact readiness contracts, release identity, private ports, shared state include, JSON logs, compatible data/sessions/cache, and tested helper restoration. Install both executable scripts and separate units, daemon-reload, enable –now the timer, then inspect its status and service journal.
Ship: enable the smallest suitable share, confirm real canary observations, test controlled critical operations and compare with stable. Increase only after the sample/health criteria pass. Monitor resource saturation and side effects as well as HTTP errors.
Hold: run canary-weight.sh disable when criteria fail, inspect its exit status and verify routing convergence and old work draining. The helper already reloads. If restoration is unconfirmed, preserve its backup and reconcile runtime/disk state; investigate and validate a fix before reopening traffic.
Finish: perform a separate controlled promotion that makes v2 the stable destination; stable 100/canary 100 still leaves v1 selected. Keep v1 available during the compatible rollback window, then wait for old work to drain before stopping it. Prepare and test the next rollout state without deleting the required include.
Feature Flags and Other Canary Changes
Feature flags can separately control a risky code path within a canary release. Validate flag targeting and defaults so the observed cohort actually exercises the intended path. Configuration or cache changes can also be tested in the canary process when both versions remain compatible.
Frontend infrastructure changes need their own isolation boundary. A shared TLS or compression setting applied to the public Nginx server may affect every request before upstream canary selection. Use a separate controlled listener, hostname or edge cohort where appropriate; changing backend weights alone does not canary the TLS handshake.
Complete the Rollout with Verified Evidence
A single-VPS canary combines two compatible application versions, explicit routing state, meaningful readiness checks and a tested traffic return path. Its value comes from observing enough actual canary work and acting on agreed stop criteria. Shared-host failures, completed side effects and data recovery remain separate concerns.
Start disabled, validate the exact examples in your own environment, install and inspect the timer, then open a measured share. Keep weighted, cookie and optional hash routes under the same gate. Use bounded log fields for analysis and bounded label values for metrics. Neither one healthy endpoint nor one successful reload proves complete readiness or traffic convergence.
Before retiring v1, confirm full promotion, compatibility and draining. Preserve the recovery procedure and enough operational evidence to investigate errors; choose retention and access rules for the data actually logged. Plan traffic rollback, application rollback and data restoration as distinct operations.
Nginx Reference Documentation
If you want to dive deeper into the Nginx knobs we used, the official docs for upstream configuration and health-related parameters and the split_clients directive are short and clear. For passive retry behavior, the proxying docs include proxy_next_upstream and friends, which are worth a skim before your next rollout.
