5 min Reading Time<\/p>\n
Last year, a DevOps engineer friend called me at 2:17 a.m. on a Tuesday – right after his first on-call night. Slack alert: \u201cCRITICAL – API Gateway Response Time > 5s – Production.\u201d He stared at a Grafana dashboard with 14 panels – and understood only three. No one had told him that on-call isn\u2019t about solving problems. It\u2019s about knowing which<\/em> problem to solve first<\/em>. Here are three lessons from that night – lessons every cloud engineer should know before their first shift.<\/strong><\/p>\n Let\u2019s call my friend Max. He\u2019d been on the team for six months. He\u2019d deployed Kubernetes clusters, written Helm charts, and built CI\/CD pipelines. He felt ready. What he didn\u2019t know? Everything he\u2019d learned applied to normal operations<\/em>. On-call begins precisely where normal operations end.<\/p>\n The alert arrived via PagerDuty, routed to Slack, and simultaneously pushed to his phone. Within four minutes, seven more alerts followed – three CRITICAL, two WARNING, two INFO. Max had no idea which of those seven was the root cause – and which were cascading failures. His Grafana dashboard showed red lines everywhere, but he couldn\u2019t tell whether the CPU spike was causing the latency – or merely a symptom of it.<\/p>\n After 20 minutes – and a panicked Slack call with the senior engineer – the issue was resolved: A pod had hit its memory limit and entered a restart loop. The fix? A single line change. But those 20 minutes of panic taught Max more about cloud operations than six months of routine work ever could.<\/p>\n That night, Max received 11 alerts in 15 minutes. Exactly one was relevant. The other ten were either downstream errors – or alerts triggered by thresholds so low they fired on every minor load fluctuation.<\/p>\n What Max wishes he\u2019d known earlier: Alert tuning isn\u2019t a one-time setup task. It\u2019s an ongoing process. Every alert that doesn\u2019t lead to action is noise. Every alert that should<\/em> trigger action – but gets missed – is a risk. Improving your signal-to-noise ratio is the most important work you do before<\/em> your first incident – not building dashboards.<\/p>\n His tip: Ask your team which alerts over the past 30 days actually led to concrete action. All the rest belong under review.<\/p>\n At 2:30 a.m., Max opened the runbook. It described an architecture that had been overhauled three months earlier. The microservice labeled \u201cSingle Point of Failure\u201d no longer existed. The load balancer cited as the \u201cprimary health-check endpoint\u201d had been replaced entirely. He was following documentation that actively misled him.<\/p>\n \n”<\/span><\/p>\n An outdated runbook is more dangerous than no runbook at all. Without a runbook, you know you have nothing. With an outdated one, you think<\/em> you have something – and follow instructions that no longer apply.<\/p>\nTL;DR<\/h2>\n
\n
2:17 a.m.: The moment nothing works like in the tutorial<\/h2>\n
Lesson 1: Signal vs. Noise – the real work happens before<\/em> the incident<\/h2>\n
Lesson 2: A runbook no one has reviewed isn\u2019t a runbook<\/h2>\n