Prometheus Alerting: Lessons from Six Months of False Positives
A workshop on writing rules that wake you only when it matters: verifying metrics before writing thresholds, predict_linear instead of static limits, log-alert pitfalls, and a two-stage AI pipeline that turns JSON into a human sentence.
Three in the morning, the phone buzzes, nothing is on fire
The first version of my alert rules woke me up regularly. Not because the server was failing — because the rules were naive. A memory spike during the nightly media library scan looked like a leak to them. A restart counter that ticked once kept alarming forever. And a metric I trusted was measuring something different than I thought.
Today the server runs over seventy rules, and most weeks none of them speaks up without a reason. This post is a workshop: what specifically changed between the version that woke me and the version that wakes me only for real. A false alarm is not harmless — every one of them teaches you to ignore notifications, and monitoring nobody reads doesn't exist.
Before you write a rule, look at the metric
The most expensive lesson of the whole six months: a metric doesn't always measure what its name suggests. Before you write a threshold, type the metric into the console and check three things: whether it exists under that name at all, what labels it carries, and whether its values match the reality you know from elsewhere.
The classic of this trap: container_memory_usage_bytes includes the filesystem cache, so it inflates "usage" by tens of percent — a container looked like it was eating 433 MB when it was really working with 326 MB. The right metric is container_memory_working_set_bytes. I told that story in full in the monitoring stack case study; here I'll leave just the workshop rule: a threshold on a metric you haven't inspected is guessing with a straight face.
for: — an alert is a state, not a spike
The second fix with the biggest payoff was the simplest one. An alert without a for: clause fires on the first sample that matches the condition — which means on every spike, every scrape restart, and every brief network hiccup.
The distribution across my seventy-plus rules speaks for itself: most rules wait 5 minutes, critical availability ones 2 minutes, and slow trends an hour (the three most common settings are 25 rules at five minutes, 14 at two, and 9 at an hour; the rest spread out from half a minute to six hours). There's also an exception with a history: the container high-memory alert waits half an hour, because the nightly media library scan can legitimately push usage above the threshold for tens of minutes — and the first version of that rule alarmed every night at the same time.
- alert: ServiceDown
expr: up == 0
for: 2m # a scrape restart is not an outage
- alert: ContainerHighMemory
expr: ... # working_set > 85% of the limit
for: 30m # the nightly library scan legitimately raises memory
Choosing for: is always the same trade-off: too short — spikes wake you; too long — you're the last to know. A practical heuristic: how long can this problem last before it actually breaks something? For a dead reverse proxy, minutes. For a memory trend — hours.
predict_linear: alarm before it happens, not when it twitches
A static memory threshold has a built-in dilemma: set it low — it alarms during normal operation; set it high — it wakes you when it's already too late. For memory leaks a trend works better: predict_linear() extrapolates the last few hours and asks whether, at the current pace, the container will hit its limit.
My leak-detection rule went through three versions, and every fix was a lesson:
- alert: ContainerMemoryLeak
expr: |
predict_linear(container_memory_working_set_bytes{...}[6h], 6*3600)
> container_spec_memory_limit_bytes * 0.95
and container_memory_working_set_bytes{...}
/ container_spec_memory_limit_bytes > 0.5
and delta(container_memory_working_set_bytes{...}[1h]) > 50 * 1024 * 1024
Three conditions joined with and, because each one lies on its own. The prediction alone, on a three-hour window, reacted to natural oscillations — a container drifting between 50 and 65% of its limit occasionally looked like a rocket. Widening the window to six hours calmed the trend, the "already using more than half the limit" condition filtered out freshly started containers, and the requirement of real growth (over 50 MB per hour) removed the ones that simply sit high and do nothing wrong.
An alert rule is not a one-time entry — it's code that gets tuned. Every false alarm should leave a trace: a comment in the rule saying what was fixed and why. My best rules carry three dated fixes in their comments — and that's exactly why they're the best.
Log-based alerts: three traps the tutorials skip
Metrics don't see everything — a process killed by the OOM killer leaves a trace in the logs, not on a chart. That's what rules in are for. And here the traps were of an entirely different kind than in Prometheus, because pattern matching against text has its own ways of being spiteful:
Word boundaries. The pattern oom also matches "Room" and "Zoom" — any chat or device name in the logs can fire a memory alert. The rescue: \boom\b. Similarly, "killed" is too broad — it catches innocent sentences in application logs; you need to match specific phrases like "OOM-killer" or "was killed".
The feedback loop. My rule searching container logs for OOM patterns found them… in Loki's own logs, which after all process those patterns. The alert kept firing itself forever. Since then every log rule carries the exclusion container_name!="loki".
Know your labels. A rule is only as good as its selector. In my setup the job is called containerlogs and container names have no leading slash — but with a different promtail configuration the same rules would match nothing and stay silent forever. Before you trust a log rule, check in the console that its selector returns anything at all. Silence can be a configuration error, not health.
The validator passes, the semantics lie: what an audit found
After half a year of tuning I ran a full audit of every rule — not because something was broken, but because everything looked like it was working. What turned up were bugs no syntax validator will ever catch, because syntactically everything was fine.
An inhibition rule dead from day one. Alertmanager lets you suppress related alerts: when a whole service is down, I don't want a separate alarm about its high CPU. My rule used target_match with the value 'HighCPUUsage|HighMemoryUsage' — and target_match matches exactly, character by character. The regex variant is called target_match_re. A literal containing | is a perfectly valid string, so the validator stayed quiet, and for half a year the rule suppressed nothing. On top of that, the alert it was supposed to target was actually named differently. Two bugs in four lines of configuration, zero warnings.
One failure, three critical notifications. Alertmanager groups notifications by alert name, among other things — every name is a separate e-mail. Over six months of adding availability rules I never noticed that a single failing metrics exporter met the conditions of three different rules with three different names. One outage, three critical notifications about the same event. The fix has two parts: rules that were 1:1 duplicates simply went away, and where a dedicated alert carries a better description than the generic one, the dedicated alert now inhibits the generic one (inhibition with equal on a shared job or instance label).
Excluding something that doesn't exist. The generic availability rule excluded the job blackbox-http — which didn't exist; it had been renamed at some point and the exclusion stayed behind. A dead exclusion is worse than no exclusion: it suggests the case is handled, so nobody ever looks there.
Operational takeaways from this round: routing can be tested behaviourally — amtool config routes test with the labels of a sample alert shows which receivers it will reach. Inhibitions need a separate check: whether the alert names and labels in source_match, target_match and equal actually occur in the rules — and in a test environment, on a pair of synthetic alerts as well. Plus a deployment detail that cost me a quarter of an hour: after editing a file mounted into a container as a single bind mount, the container keeps seeing the old version, because the editor swaps the inode — only a container restart helps, not a reload.
A validator checks syntax, not intent. The alert name in an inhibition rule, the job in an exclusion, the label in a selector — these are all strings that must agree with the rest of the configuration, and no tool guards that agreement. A quarterly semantics audit (do the names still exist, do the suppressions actually suppress) finds bugs you otherwise can't see, because they show up as too many notifications, not too few.
The audit also threw in a finding from outside alerting itself: Loki had been writing its index in a format that had meanwhile been marked as deprecated (boltdb-shipper). Migrating to its successor (TSDB) turned out safer than it sounds — a new entry in schema_config with a future date, old data stays readable by the old engine until retention expires, and the rollback is simply removing the entry before that date arrives.
The last mile: from JSON to a human sentence
Even a well-tuned alert reaches your phone as a robot-to-robot message: labels, values, identifiers. The final leg of the pipeline is handled by two local language models via — and the division of labour between them is a lesson of its own, measured rather than assumed. Not every alert wakes the models at all, though: known, recurring patterns carry a hardcoded class and ready-made message, and only the non-obvious ones reach the AI — on a CPU with no GPU, that's a sensible economy.
The plan was simple: one model classifies the priority and writes the message. The tests said otherwise. phi4-mini classified flawlessly and fast, but its Polish turned out crippled. Bielik — the Polish model from SpeakLeash — wrote naturally, but flattened classification: everything landed mid-scale, a critical disk included. Neither could do both at once. The solution: a two-stage pipeline — phi4-mini classifies (stage 1), Bielik writes the final message in Polish (stage 2).
Deployment added traps of its own. Through the raw generate endpoint, Bielik would sometimes echo the prompt back instead of answering — what helped were chat-style calls with example answers in the system prompt (few-shot); without them reliability was a lottery, with them every test passed. A single example, in turn, anchored the style — the model copied its closing line into every message; two examples with different endings solved it. And the overriding rule of the whole stage: fail-open. When Bielik fails or echoes (a simple content guard catches it), the alert goes out with a fallback message. Nice language is optional — delivery is not.
AI sits at the end of the pipeline for a reason: it translates good alerts into human language, but it won't fix bad rules. The order of investment is unambiguous — metrics and rules first, then delivery, language cosmetics last. In the reverse order you get beautifully written false alarms.
The delivery path itself — who gets notified, over which channel, what happens during quiet hours — is covered in the post about n8n patterns, and choosing models for CPU without a GPU in the post about running a local LLM.
Checklist before deploying a rule
- Inspect the metric in the console — it exists, it has those labels, it measures what you think.
- Counter? Compare the increase (
increase(...[5m]) > 0), never the raw value — otherwise the alert never clears. - Add a
for:matched to the question "how long can this problem last before it hurts". - Trend instead of threshold where problems build slowly —
predict_linearwith conditions that cut off oscillations. - Log rules: word boundaries in patterns, exclude the logging system's own logs, a verified selector.
- The rule description = the first diagnostic step, not a restatement of the name. At three in the morning you'll thank yourself for a ready-made command.
- Every false alarm corrects the rule and leaves a dated comment. A silence is not a fix.
- A quarterly semantics audit — do the alert names in inhibition rules and the jobs in exclusions still exist; check routing with
amtool config routes teston a sample of labels.
Six months later, monitoring wakes me a few times a quarter — and every time rightly so. That's the proper measure: not the number of rules or dashboards, but trust in the phone buzzing in the middle of the night.
Frequently asked questions
Where should I start with alerting in a homelab?
With three rules that catch real failures: a service stops responding, disk space is running out, a container is restarting in a loop. Each with a sensible delay (for: 2-5 minutes) and a description of what to do. Only once those three stop producing false alarms is it worth adding more.
Why doesn't my alert clear even though the problem is gone?
Most often because the rule looks at a counter instead of its increase. Prometheus counters only go up — a condition like restarts > 0 stays true forever. Compare the increase over a time window (increase over the last 5 minutes) and the alert will clear together with the problem.
How do I cut false alarms without silencing everything?
Three levers: for (an alert is a sustained state, not a spike), the right metric (check what it actually measures before writing a threshold), and trend prediction instead of a static threshold where problems build up slowly. Silencing is a last resort — a rule you keep silencing is simply written wrong.
Does AI in the alert path make sense without a GPU?
Yes, because it's a background task — nobody is waiting for the answer interactively. A small model classifies the priority in a dozen or so seconds, a second one writes a readable message. But AI sits at the end of the pipeline: it translates good alerts into human language, it won't fix bad rules.