How to Read Web Server Logs to Diagnose 4xx–5xx Errors on Apache and Nginx

If you run a website on Apache or Nginx, your access and error logs are the most honest source of truth you have. Every 404, 403, 500 or 502 that users see is already written there line by line. When teams at dchost.com review performance, debug SEO drops or analyze checkout issues, the first place we look is almost always the web server logs. Once you know how to read them, 4xx–5xx errors stop being mysterious; they become concrete, traceable events tied to exact URLs, IPs, and timestamps.

In this guide we’ll walk through how to read Apache and Nginx access and error logs with a focus on diagnosing 4xx–5xx HTTP errors. We’ll look at real log examples, explain what each field means in plain language, and show practical command‑line queries you can copy‑paste. Whether you are on shared hosting, a VPS, or a dedicated server at dchost.com, the workflow is the same: find the right log file, filter for problematic status codes, and connect each error back to configuration or application issues so you can fix them with confidence.

Why 4xx–5xx Errors Start in Your Web Server Logs

HTTP status codes are your web server’s way of describing what happened to each request. 2xx means success, 3xx means redirect, 4xx means a client‑side problem, and 5xx means a server‑side failure. Browser error pages are only a tiny, user‑facing summary. The details live in your logs.

When we review incidents for our customers, almost every meaningful question can be answered from logs:

• “Are these 404s caused by bots, broken internal links, or old external links?” → Check referrer and user agent in access logs.
• “Why are we getting random 500 errors on checkout?” → Correlate 500s in access logs with PHP or application errors in error logs.
• “Is that 503 throttle from the app or from the web server?” → Compare app logs, web server error logs, and upstream timeouts.

We already covered what the main HTTP codes mean for SEO and redirects in our article What HTTP Status Codes Mean for SEO and Hosting. In this article we’ll stay at the server layer and focus on how to read the raw log lines that generate those codes, especially 4xx and 5xx, so you can move from guessing to precise diagnosis.

Quick Refresher: HTTP 4xx and 5xx Status Codes

Before diving into logs, it helps to have a mental map of the most common 4xx–5xx codes you’ll see.

Common 4xx (Client Error) Codes

• 400 Bad Request – The request is malformed (invalid headers, too long URL, or broken syntax). Often logged with extra info in error logs.
• 401 Unauthorized – Authentication required or failed. Usually combined with WWW-Authenticate headers.
• 403 Forbidden – The server understood the request but refuses to authorize it. Common with directory restrictions or security rules.
• 404 Not Found – Requested URL doesn’t exist or rewrite rules mis‑route it.
• 405 Method Not Allowed – Method (POST, PUT, DELETE) is not allowed for this endpoint.
• 408 Request Timeout – Client took too long to send the full request.
• 429 Too Many Requests – Rate limiting by web server, WAF or application.

Common 5xx (Server Error) Codes

• 500 Internal Server Error – Generic error; often PHP, application or configuration failure.
• 502 Bad Gateway – Web server (reverse proxy) couldn’t get a valid response from upstream (PHP‑FPM, Node.js, etc.).
• 503 Service Unavailable – Server is overloaded, under maintenance, or backend pool is down.
• 504 Gateway Timeout – Upstream took too long to respond; often a slow database or external API.

Your Apache or Nginx access log records every one of these responses. The error log will often explain why the server picked that code. Reading both together is the key skill you want to build.

Where to Find Apache and Nginx Logs on Common Systems

Exact paths depend on your OS and control panel, but most dchost.com VPS and dedicated servers follow standard Linux conventions.

Typical Apache Log Locations

• Debian/Ubuntu
  /var/log/apache2/access.log
/var/log/apache2/error.log
• CentOS / AlmaLinux / Rocky Linux
  /var/log/httpd/access_log
/var/log/httpd/error_log
• Per‑vHost logs (common on panels)
  Something like /home/USER/logs/domain.com-access_log and domain.com-error_log

Typical Nginx Log Locations

• All major Linux distros
  /var/log/nginx/access.log
/var/log/nginx/error.log
• Per‑site logs (based on your vhost config)
  Example snippet:
  access_log /var/log/nginx/example.com.access.log main;
error_log /var/log/nginx/example.com.error.log warn;

On shared hosting, you may only see per‑domain logs inside your home directory; on a VPS or dedicated server you have full root‑level access to all logs. If you’re planning a move from shared hosting to a VPS primarily for better observability and control, our checklist for moving from shared hosting to a VPS with zero downtime is a good companion read.

Understanding Access Log Formats (With Real Examples)

Access logs are usually in either Common Log Format (CLF) or Combined Log Format. Combined is more useful for 4xx–5xx diagnosis because it includes referrer and user agent.

Apache Combined Log Format

Definition (from LogFormat):

%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"

Example line:

203.0.113.45 - - [05/Dec/2025:10:23:12 +0000] "GET /checkout HTTP/1.1" 500 512 "https://example.com/cart" "Mozilla/5.0 ..."

Reading it from left to right:

• 203.0.113.45 – Client IP
• – – – Unused identifiers (typically identd and authenticated user)
• [05/Dec/2025:10:23:12 +0000] – Timestamp
• “GET /checkout HTTP/1.1” – Request line: method, URL, protocol
• 500 – Status code (here, a server error)
• 512 – Response size in bytes
• “https://example.com/cart” – Referrer (where the user came from)
• “Mozilla/5.0 …” – User agent (browser, bot, app)

Just from this one line you already know: a real browser (probably) clicked “Proceed to checkout” from the cart page and got a 500. That’s enough to start reproducing and then check error logs or application logs.

Nginx Log Format

Nginx lets you define log_format. A common “combined” style format looks like:

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                '$status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$request_time"';

Example line:

203.0.113.45 - - [05/Dec/2025:10:23:12 +0000] "POST /api/login HTTP/2.0" 401 178 "https://app.example.com/" "Mozilla/5.0 ..." "0.123"

• 401 – Authentication failed
• “0.123” – Request processing time in seconds (very handy when debugging slow 5xx responses)

The key for 4xx–5xx diagnosis is simple: spot the non‑2xx codes, then read the surrounding fields like a story. Who requested what, from which page, with which user agent, and how long it took.

Understanding Error Logs: When Things Really Break

Access logs tell you that a request returned 500. Error logs tell you why. Let’s look at a few common patterns.

Apache Error Log Examples

Example: PHP fatal error causing a 500:

[Fri Dec 05 10:23:12.345678 2025] [proxy_fcgi:error] [pid 12345:tid 140123456789] [client 203.0.113.45:51724] AH01071: Got error 'PHP Fatal error:  Uncaught Error: Call to undefined function wc_get_cart() in /home/user/public_html/wp-content/plugins/custom-checkout.php:42n'

Pieces to notice:

• [proxy_fcgi:error] – The module that logged the error (here, PHP‑FPM proxy).
• [client 203.0.113.45:51724] – Same IP you’ll see in access log.
• PHP Fatal error… – The actual application‑level cause.

Now you can correlate this exact timestamp and client IP with the matching line in access.log to see the URL and referrer.

Example: ModSecurity or access rules causing a 403:

[Fri Dec 05 10:31:00.123456 2025] [authz_core:error] [pid 22345:tid 140223456789] [client 198.51.100.77:38210] AH01630: client denied by server configuration: /home/user/public_html/wp-admin/admin-ajax.php

This tells you the 403 is from Apache’s authorization layer, not from WordPress itself. That matters a lot when tuning WAF rules or .htaccess protection.

Nginx Error Log Examples

Example: PHP‑FPM timeout causing a 504:

2025/12/05 10:45:22 [error] 1234#1234: *56789 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 203.0.113.45, server: example.com, request: "GET /report?month=12 HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.sock", host: "example.com"

Important bits:

• upstream timed out – Nginx didn’t get a response in time.
• request: “GET /report?month=12” – Exact URL causing timeouts.
• upstream: “fastcgi://unix:/var/run/php-fpm.sock” – The backend that is slow (likely CPU/DB issues).

Example: 404 caused by missing file or bad rewrite:

2025/12/05 11:10:00 [error] 1234#1234: *57000 open() "/home/user/public_html/wp-content/themes/theme/assets/js/app.js" failed (2: No such file or directory), client: 198.51.100.10, server: example.com, request: "GET /wp-content/themes/theme/assets/js/app.js HTTP/2.0", host: "example.com"

Here you know instantly: the file truly does not exist at that path. This is very different from a 404 caused by application routing, which might not generate an error log entry at all.

Step-by-Step: Diagnosing Common 4xx Errors from Logs

Let’s walk through concrete workflows for the 4xx errors you’ll see most often.

404 Not Found: Broken Links vs. Bot Noise

First, see how many 404s you have and which URLs are affected.

# Apache
awk '$9 ~ /^404$/' /var/log/apache2/access.log | head

# Nginx (status is field 9 in typical formats)
awk '{ if ($9 == 404) print }' /var/log/nginx/access.log | head

Now, group by URL to find the worst offenders:

awk '$9 == 404 {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

This shows counts and paths: maybe /old-blog-post, /wp-login.php, or random bot paths like /.env.

• If you see many 404s on real pages that used to exist, add redirects (301/410). Our article on HTTP status codes, SEO and hosting explains when 301 vs 410 is the right choice.
• If 404s are bot scans for /wp-admin, /.git, /phpmyadmin etc., that’s normal background noise, but you may want to harden security or add WAF rules.

You can also break down 404s by referrer:

awk '$9 == 404 {print $11}' /var/log/apache2/access.log | sed 's/"//g' | sort | uniq -c | sort -nr | head

This tells you whether 404s are coming from your own pages (internal links to fix) or from external sites.

403 Forbidden: Security Rules or Permissions

For 403s you always want to check both access and error logs.

awk '$9 == 403 {print $1, $4, $5, $7}' /var/log/nginx/access.log | head

Then inspect matching error log entries around the same timestamp:

grep "[client 198.51.100.77" /var/log/apache2/error.log | tail

Common 403 causes:

• Directory or file permissions (e.g. chmod 600 on static assets).
• .htaccess or Nginx rules that deny certain IPs, paths or user agents.
• ModSecurity / WAF rules blocking suspicious patterns in query strings.

If your 403s are mostly WAF‑related, our deep dive on tuning ModSecurity and OWASP CRS to cut false positives will help you reduce noise without weakening security.

401 Unauthorized: Login Troubles

For 401 errors, look at:

• Path – Is it an API endpoint, admin panel, or HTTP Basic Auth area?
• User agent – Real browsers vs. scripts vs. integrations.
• Rate – Is someone brute‑forcing passwords?

awk '$9 == 401 {print $1, $7, $11}' /var/log/apache2/access.log | sort | head

If you see many 401s from the same IP, combine logs with your authentication system’s logs and consider temporary IP blocking (fail2ban, Nginx limit rules, etc.).

429 Too Many Requests: Rate Limiting in Action

429s are usually great news: your rate limiting is working. But they can also hurt real users if thresholds are wrong.

awk '$9 == 429 {print $1, $7}' /var/log/nginx/access.log | sort | uniq -c | sort -nr | head

Ask yourself:

• Are 429s mostly from one or a handful of IPs? → Likely bots or scraping tools.
• Are real users hitting 429 on checkout or cart APIs? → Limits may be too strict.

Our article on monitoring cart and checkout steps with server logs and alerts shows how we use 4xx/5xx patterns to spot real customer friction in e‑commerce flows.

Step-by-Step: Diagnosing Common 5xx Errors from Logs

5xx errors are usually more urgent because they mean your server or app failed. Here’s how to quickly narrow down causes.

500 Internal Server Error: Application or Config Bugs

Start by listing recent 500s:

awk '$9 == 500 {print $4, $5, $7}' /var/log/apache2/access.log | tail -n 50

Then jump to the error log around those times:

tail -n 200 /var/log/apache2/error.log

Typical patterns:

• PHP Fatal error / Uncaught exception – Bug or missing dependency.
• Rewrite rules loop – Misconfigured .htaccess or try_files.
• Permissions problems – App can’t write to cache or storage directories.

If 500s cluster around heavy operations (checkout, reporting, imports), combine this with application and database logs. For PHP apps like WordPress or WooCommerce, our guides on fixing WordPress white screen of death from the hosting side and MySQL/InnoDB tuning for WooCommerce show how server‑side tuning reduces 5xx incidents dramatically.

502 Bad Gateway and 504 Gateway Timeout: Upstream Problems

502/504 almost always mean your reverse proxy (Apache mod_proxy, Nginx, or a load balancer) couldn’t talk to the application backend properly.

In access logs they just show up as status 502/504. The real story is in error logs:

• Apache + PHP‑FPM
  Look for AH01071: Got error 'Primary script unknown' (wrong DOCUMENT_ROOT or script path) or socket connection errors.
• Nginx + PHP‑FPM
  Look for upstream timed out or connect() to unix:/var/run/php-fpm.sock failed.

Checklist when you see many 502/504:

• Is the upstream process running? (php-fpm, node, gunicorn, etc.)
• Is the socket/port correct between web server and app?
• Are timeouts too low for heavy background tasks?
• Is CPU/IO saturated? (Check top, htop, iostat.)

If resource exhaustion is frequent, it may be time to scale resources or separate app and database servers. Our article on 9 server‑side signals it’s time to upgrade your hosting plan explains which log‑level symptoms mean “tuning is not enough anymore.”

503 Service Unavailable: Overload or Maintenance

503 can come from either the web server or the application. Look for hints:

• Nginx – no live upstreams while connecting to upstream indicates all backend servers are down or marked failed.
• Application – Custom HTML saying “We’re under maintenance” while access logs show 503 but error logs are quiet.

Key questions:

• Do 503s correlate with traffic peaks? → CPU/RAM or database contention.
• Do 503s appear after deploys? → App maintenance mode toggled incorrectly.

Reading logs around traffic spikes alongside system metrics is where proper monitoring pays off. We have a dedicated guide on VPS monitoring and alerts with Prometheus, Grafana and Uptime Kuma that fits neatly on top of the log techniques in this article.

Practical Log Analysis Commands You’ll Actually Use

You don’t need a full SIEM stack to start getting value from logs. A handful of shell commands will take you very far on any dchost.com VPS or dedicated server.

Follow Logs in Real Time

# Access logs
sudo tail -f /var/log/nginx/access.log

# Error logs
sudo tail -f /var/log/nginx/error.log

Trigger the error in your browser and watch the line appear in real time. This is the simplest and most underrated technique.

Filter by Status Code Range (4xx/5xx)

With awk you can quickly filter 4xx and 5xx:

# All 4xx
awk '$9 ~ /^4[0-9][0-9]$/' /var/log/apache2/access.log | head

# All 5xx
awk '$9 ~ /^5[0-9][0-9]$/' /var/log/nginx/access.log | head

To see top IPs causing 5xx:

awk '$9 ~ /^5[0-9][0-9]$/ {print $1}' /var/log/nginx/access.log | 
  sort | uniq -c | sort -nr | head

Find Slow Requests that Also Error

If your Nginx log format includes $request_time as last field:

# Slow 5xx (request_time > 2 seconds)
awk '$9 ~ /^5/ && $NF > 2 {print $4, $5, $7, $9, $NF}' /var/log/nginx/access.log | head

This shows which endpoints both fail and are slow, helping you focus optimization on high‑impact areas.

Find Error Spikes by Minute

To see if errors come in bursts:

awk '$9 ~ /^5/ {gsub("[:[]"," ",$4); print $4" "$5}' /var/log/apache2/access.log | 
  awk '{print $1" "$2":"$3}' | sort | uniq -c | sort -nr | head

This aggregates 5xx counts by minute, so you can correlate with deploy times, cron jobs, or traffic surges.

Turn Logs into Monitoring & Alerts (and When to Upgrade Hosting)

Reading logs manually is great during debugging, but long‑term reliability comes from automation: centralizing logs, setting retention, and creating alert rules based on 4xx–5xx patterns.

Centralised Logging on a VPS

On multi‑server architectures (e.g. separate web, app and database servers) we strongly recommend central logging. Instead of SSH‑ing into 3–4 machines and grepping logs, you send everything to a single place for querying and dashboards.

We’ve shared a full playbook in VPS log management without the drama: centralised logging with Grafana Loki + Promtail. The same principles apply whether you run one dchost.com VPS or a fleet of dedicated servers and colocation machines.

Meaningful Alert Rules Based on 4xx–5xx

Instead of “alert on any 500”, it’s more practical to define:

• Error rate thresholds – e.g. “5xx > 1% of requests for 5 minutes”.
• Path‑specific alerts – e.g. “Any 500/502 on /checkout or /payment callbacks”.
• Spike detection – “404s on a single URL > 100/minute” (often broken campaign links or bot waves).

In practice we combine these with business‑level events (orders per minute, signup success rate) so that not every 500 wakes you up, but problems that actually hurt users always get attention.

When Logs Tell You It’s Time to Scale

Sometimes logs make it clear that the problem isn’t a bug, but capacity. Signs include:

• Regular 502/504 timeouts at predictable traffic levels.
• 503s coinciding with CPU at 90–100% and DB load spikes.
• 429 or rate‑limit patterns where you are forced to be overly strict to keep servers alive.

In those cases, optimizing queries and caching is still important, but it’s also time to review your hosting resources (CPU, RAM, NVMe, bandwidth). Our article on capacity planning for WooCommerce in terms of vCPU, RAM and IOPS gives a concrete way to link what you see in logs with what you should provision on your next dchost.com VPS or dedicated server.

Hardening, Performance, and Next Steps

Web server logs are not just for “something is broken” moments. Once you’re comfortable reading 4xx–5xx patterns, you can also use logs to harden security, tune caching, and plan upgrades calmly.

• Security hardening – 403/401/404 patterns reveal brute‑force attempts, path discovery scans and vulnerable plugins. Combine this with a solid baseline from our security checklist for new websites to close obvious doors.
• Performance tuning – High 5xx rates on specific endpoints often point at slow database queries, missing indexes or lack of caching. Use logs to prioritize which pages to optimize first.
• SEO & UX – Persistent 404s on important URLs hurt both search visibility and user trust. Logs help you spot and fix those faster than waiting for search console reports.

At dchost.com our domain, hosting, VPS, dedicated server and colocation customers all benefit from one simple idea: logs first. Before throwing more hardware at a problem or rewriting half the application, we always start by reading what Apache or Nginx are already telling us about 4xx–5xx errors. Most issues become obvious once you see the exact combination of IP, URL, status, response time and backend error.

If you’re currently on a setup where you don’t have full log access, or you’re fighting recurring 500/502/503 errors without clear visibility, this is a good time to rethink your infrastructure. A right‑sized VPS or dedicated server at dchost.com, with proper log retention and monitoring, gives you the control you need to debug calmly instead of guessing. And if you’re unsure where to start, our team is happy to help you design a stack where Apache or Nginx logs, application logs and metrics all line up—so the next 4xx–5xx incident is just another small, well‑understood story in your logs, not a crisis.