Skip to content
BytePatterns

Observability Basics

System Design: lesson 15 of 17

Metrics say something broke; traces say where.

Lesson 15 of 17 · 5 min

Observability Basics

Step 1 of 13

Metrics are cheap numbers over time: request rate, error rate, latency percentiles.

The Idea

Metrics are cheap numbers over time — request rate, error rate, latency percentiles — ideal for alerting and useless for detail. Logs record individual events with context, and get expensive at volume. A trace follows one request across every service it touched, which is the only view that finds the slow hop.

Real-World Example

A bus control room runs all three. The wall board shows headway and punctuality; each vehicle's fault log explains one breakdown; and when a complaint lands, somebody follows that single journey through every depot handover to see where the twelve minutes went.

The Tradeoff

All three cost money and attention, and a dashboard nobody reads is worse than none. Watch percentiles rather than averages, keep metric labels low-cardinality — a label per user ID will bankrupt you — and sample traces instead of recording every single request.

Your turn

Put the steps in the right order.

  1. Open the trace of a slow request to see which hop cost the time
  2. An alert fires when the error-rate metric crosses its threshold
  3. Read that one service's logs around the failing requests
  4. Check the dashboard to see which service the errors come from

Mini quiz

1 / 3

Alerting is best built on:

New lessons land every few weeks

Leave an address and we will tell you when the next one is up. That is the only reason we will use it.

One address, stored so we can email you. Nothing else, ever.