The monitoring stack we wished we had on day one
Most teams build their observability stack backwards — they add monitoring after the fire, not before it. Here's the exact stack I wish we'd had in 2021 when a missing p99 metric cost us 11 hours of downtime and $340k in SLA credits.
In 2021, our team was running a B2B SaaS platform at about 8M daily active users. We had Datadog. We had PagerDuty. We had dashboards that looked incredible in the sales demo. And we had absolutely no idea our checkout service p99 latency had been creeping up for 6 days before it finally fell over at 2am on a Tuesday.
11 hours of downtime. $340k in SLA credits. The root cause was connection pool exhaustion in our Postgres driver — something pg_stat_activity would have screamed about if we'd been watching it. We weren't.
What we actually needed
The stack isn't complex. It's three layers that most teams only half-implement. First, you need RED metrics — Rate, Errors, Duration — per service, not per host. Datadog's APM gives you this out of the box but you have to instrument at the service boundary, not the infrastructure layer. Most people get this backwards.
Second, you need database internals. Not just query latency. pg_stat_statements, connection pool wait times, lock contention. We use pgBadger on top of slow query logs and it's caught three incidents before they became pages. Third, you need synthetic checks that run against production, not staging. We run k6 scripts every 60 seconds hitting real user flows. The moment checkout p99 crosses 800ms, we know before any user does.
The alert that actually wakes you up
PagerDuty with raw threshold alerts is noise. What works is alerting on rate of change, not absolute values. A p99 that jumps 40% in 5 minutes matters more than one that's been sitting at 600ms for a week. Prometheus's increase() and rate() functions make this trivial. Most teams never use them.
What I'd do differently: instrument before you ship, not after you burn.