Logs, Metrics and Alerts

From the Monitoring Tower you can see the whole compound, but only if the screens show something useful. Once your app has real users, you can't attach a debugger to it; what it writes down is all you get. Logs tell you what happened to one request, metrics tell you how the whole system is doing, and alerts tell you when to look. Beginners usually have too few of the useful ones and too many of the noisy ones.

Structured logs

A log line like console.log('Payment failed for ' + userId) is easy to write and hard to search. A structured log is an object with named fields, usually printed as one JSON line:

logger.error({
  msg: 'payment failed',
  requestId: req.id,
  userId: user.id,
  orderId: order.id,
  provider: 'stripe',
  durationMs: 2140,
  error: err.message,
});

Log tools can now filter by any field: all failures for one user, all Stripe errors, everything slower than two seconds. Libraries such as pino do this for Node with little setup. Log levels help too: error for things that need attention, warn for the unusual, info for normal events, and debug for detail you switch on only while investigating.

Request ids: following one request

A busy server handles hundreds of requests at once, so their log lines are mixed together. Give each request an id when it arrives (or reuse one sent by the caller in a header such as x-request-id), attach it to every log line for that request, and return it in the response. When a user reports "checkout failed at 14:02," you find one line, copy its request id, and filter to see the whole story of that request in order. Pass the same id along when you call other services, and you can follow it across them too.

What never to log

Logs are copied to log services, kept for weeks and read by many people. Never log:

• passwords, even the wrong ones people type
• access tokens, session cookies, API keys, Authorization headers
• full card numbers or other payment details
• more personal data than you need to debug

The common way this happens is logging a whole object, such as logger.info(req.body) on a login route, or req.headers everywhere. Log specific fields instead, and use your logger's redaction option for known sensitive keys as a safety net.

Metrics: error rate and latency

Metrics are numbers counted over time. Two matter most for a web API:

• Error rate: the share of requests that fail with a 5xx, for example 2% over the last five minutes. Use a rate, not a raw count: 50 errors a minute is a disaster at 3am and background noise at peak.
• Latency percentiles: p95 is the time under which 95% of requests finish, so 5% are slower. Averages hide pain. An average of 120ms can come with a p95 of 3 seconds, which means one request in twenty feels broken. Watch p95 or p99 per endpoint.

Add traffic (requests per second) and you can tell "errors went up" apart from "traffic went up."

Alerts on what users feel

An alert should mean "a person needs to act now." The best alerts watch symptoms users feel: error rate above 5% for five minutes, checkout p95 above 2 seconds, no successful signups in 30 minutes. Causes like high CPU or a full queue are worth a dashboard, but CPU at 85% with happy users is not an emergency, and plenty of outages happen with normal CPU.

The for five minutes part matters. One slow request should not wake anyone; a sustained problem should.

Avoiding alert fatigue

If an alert fires thirty times a day and is almost never real, people stop reading it, and the one real page gets ignored with the rest. Treat every alert that fires without needing action as a bug: raise its threshold, lengthen its window, turn it into a dashboard or delete it. Each alert should link to what to check first. A small set of alerts that are always worth reading beats a wall of red that everyone mutes.