Setting Up Alert Rules

Raw alerts are noisy. A CPU spike at 80% during a deploy is normal. The same spike at 3 AM on a quiet night is a problem. In this lesson, you'll create intelligent alert rules that reduce noise, add context, and tell you why something is happening — not just what.

The Alert Noise Problem

Most monitoring setups suffer from alert fatigue:

Too many alerts: Every minor threshold breach fires a notification
No context: "CPU > 80%" tells you nothing about cause
No correlation: A database alert and an API latency alert fire separately, even though they're the same incident
Wrong channel: Critical alerts go to the same place as informational ones

The goal: fewer alerts, each one actionable.

Define severity tiers

Set up tiered routing so critical alerts wake you up and informational ones wait for morning:

openclaw agent edit "DevOps Monitor" \
  --append-message "

SEVERITY ROUTING:
🔴 CRITICAL (immediate action required):
  - Service down / health check failing
  - Error rate > 5% for 5+ minutes
  - Disk usage > 95%
  - Database connections exhausted
  → Send to #ops-critical + page on-call engineer

🟡 WARNING (investigate soon):
  - CPU > 85% for 10+ minutes
  - Memory > 80%
  - Error rate 1-5%
  - Response time p99 > 2s
  → Send to #ops-alerts

🟢 INFO (review in morning):
  - Deployment completed
  - Auto-scaling event
  - SSL cert expiring in 30+ days
  - Scheduled maintenance completed
  → Send to #ops-info (muted channel)

When in doubt, default to WARNING. Never suppress an alert entirely
— downgrade it to INFO instead."

Configure the channel routing:

openclaw alert-route add \
  --name "Critical" \
  --severity critical \
  --channel slack --to "#ops-critical" \
  --also-page "on-call"

openclaw alert-route add \
  --name "Warning" \
  --severity warning \
  --channel slack --to "#ops-alerts"

openclaw alert-route add \
  --name "Info" \
  --severity info \
  --channel slack --to "#ops-info"

Add AI enrichment to alerts

This is the key difference. Instead of forwarding raw alerts, OpenClaw analyzes them first:

openclaw agent edit "DevOps Monitor" \
  --append-message "

ENRICHMENT RULES:
When you receive an alert, before sending to chat:

1. CHECK RECENT EVENTS (last 2 hours):
   - Any deployments? (from GitHub/Vercel connection)
   - Any config changes?
   - Any other alerts that fired around the same time?

2. CHECK HISTORICAL PATTERNS:
   - Has this alert fired before? When? What fixed it?
   - Is this a known pattern (e.g., daily backup causes CPU spike)?

3. ANALYZE PROBABLE CAUSE:
   - If deploy happened < 30 min ago: likely deploy-related
   - If multiple services affected: likely infrastructure (network, DB, DNS)
   - If single service affected: likely application-level

4. FORMAT THE ALERT:
   🚨 [SEVERITY] Service: [name]
   📊 Metric: [what triggered] → [current value]
   🕐 Started: [time] (duration: [X min])
   🔄 Recent changes: [deploy/config/none]
   🤔 Probable cause: [AI analysis]
   📋 Suggested action: [specific next step]
   📎 Similar incidents: [link to past incidents if any]"

Test with a simulated alert:

openclaw connect simulate Grafana \
  --alert "High error rate" \
  --metric "http_error_rate" \
  --value "4.2%" \
  --host "api-prod-01" \
  --duration "8 minutes"

Expected enriched alert:

🟡 WARNING — api-prod-01

📊 Error rate: 4.2% (threshold: 1%)
🕐 Started: 14:32 UTC (8 minutes ago)

🔄 Recent changes:
  • Deploy #487 at 14:25 UTC (7 min before alert)
    "feat: add user search endpoint" by @sarah

🤔 Probable cause:
  New deployment likely introduced a regression.
  Error rate was 0.1% before deploy, spiked to 4.2% after.
  Most errors are 500s on /api/users/search endpoint.

📋 Suggested actions:
  1. Check logs: /logs api-prod-01 --since 14:25
  2. Roll back if confirmed: /rollback #487
  3. Or investigate: /ssh api-prod-01 "tail -f /var/log/app.log"

📎 Similar: Incident #34 (Jan 2) — same endpoint, fixed by
   adding index on users.email

Set up alert deduplication

Prevent the same issue from flooding your channel:

openclaw alert-rule add \
  --name "Deduplication" \
  --group-by "host,metric" \
  --window "15m" \
  --action "If the same host+metric fires multiple times within 15 minutes,
send only the first alert. Append 'Still ongoing (X occurrences)' to the
original message every 15 minutes until resolved."

Before dedup:

14:32  🟡 CPU > 85% on web-prod-01
14:37  🟡 CPU > 85% on web-prod-01
14:42  🟡 CPU > 85% on web-prod-01
14:47  🟡 CPU > 85% on web-prod-01

After dedup:

14:32  🟡 CPU > 85% on web-prod-01
14:47  🟡 ↳ Still ongoing (4 occurrences, 15 min)
15:02  ✅ Resolved: CPU normalized at 72% on web-prod-01

Create smart correlation rules

Group related alerts into a single incident:

openclaw alert-rule add \
  --name "Incident Correlation" \
  --rule "If 3+ alerts fire within 5 minutes across different services,
group them as a single incident. Create an incident thread in #ops-critical
with:
1. All related alerts
2. Common factors (same host? same time? same recent deploy?)
3. Likely root cause based on correlation
4. Single action plan instead of per-alert suggestions"

Example correlated incident:

🔴 INCIDENT #42 — Multiple services affected

Related alerts (all started within 3 minutes):
  • API error rate: 8.5% on api-prod-01
  • Database connections: 95% utilized on db-prod-01
  • Queue depth: 15,000 (normally <100) on worker-prod-01

🔄 Common factor: Deploy #487 at 14:25 UTC
🤔 Root cause: New search endpoint running unoptimized
   queries, exhausting DB connection pool. API errors
   and queue backlog are downstream effects.

📋 Action plan:
  1. Roll back deploy #487: /rollback #487
  2. Monitor DB connections: /metrics db-prod-01 connections
  3. Once stable, investigate the query in staging

Add time-aware rules

Some alerts mean different things at different times:

openclaw alert-rule add \
  --name "Time Context" \
  --rule "Apply these time-based overrides:

BUSINESS HOURS (9am-6pm weekdays):
  - CPU > 85%: WARNING (normal load)
  - Deployment events: INFO

OFF-HOURS (nights, weekends):
  - CPU > 85%: CRITICAL (unexpected load)
  - Any deployment: WARNING (who's deploying at 2 AM?)

KNOWN MAINTENANCE WINDOWS:
  - Suppress all alerts during scheduled maintenance
  - Auto-resolve alerts that clear within 10 min of maintenance end

DAILY PATTERNS:
  - Ignore backup-related CPU spikes (daily 2-3 AM)
  - Ignore auto-scaling events during known traffic peaks"

Alert Rule Cheat Sheet

Rule	Effect
Severity routing	Critical → page, Warning → alert channel, Info → quiet channel
AI enrichment	Raw metric → context + cause + action
Deduplication	Same alert fires 10x → you see 1 message + updates
Correlation	5 related alerts → 1 incident thread
Time awareness	CPU spike at 2 PM = normal, at 2 AM = critical

Create templates for your most common alert types:

openclaw alert-template add \
  --name "High Error Rate" \
  --format "
🚨 [{severity}] Error rate spike on {host}

Rate: {value} (threshold: {threshold})
Top errors:
{top_5_error_messages}

Recent deploys: {recent_deploys}
Suggested: {ai_suggestion}"

Templates ensure consistent formatting and make alerts scannable at a glance.

Temporarily suppress non-critical alerts during deployments:

openclaw alert-rule add \
  --name "Deploy Quiet" \
  --rule "When a deployment starts, suppress WARNING-level alerts
for 10 minutes. If any alert persists after 10 minutes, escalate
to CRITICAL (the deploy may have caused a problem)."

This prevents false positives from deployment-related metric fluctuations while still catching real issues.

What You Have Now

Your alert system is no longer a fire hose. Alerts are:

Tiered — critical alerts page you, info waits for morning
Enriched — each alert includes probable cause and suggested action
Deduplicated — no more notification spam from flapping metrics
Correlated — related alerts grouped into single incidents
Time-aware — context-sensitive severity based on time of day

The next lesson turns these alerts into action with one-command remediation.

Knowledge Check

Why should you downgrade noisy alerts to INFO instead of suppressing them entirely?

Setting Up Alert Rules

Setting Up Alert Rules

The Alert Noise Problem

Define severity tiers

Add AI enrichment to alerts

Set up alert deduplication

Create smart correlation rules

Add time-aware rules

Alert Rule Cheat Sheet

What You Have Now

Alert routing that does not page you at 3am for a warning

Add the tool, test the tool, then let anything else depend on it