Prometheus + Grafana Monitoring Stack — From Blind to Omniscient

The Problem

Your production app is running. Is it healthy? You check... nothing. Because monitoring is always the thing you'll set up "after launch."

Then at 3 AM, your app goes down. You find out from an angry customer tweet, not from your monitoring system (because you don't have one). You SSH in, check logs manually, restart services, and swear you'll set up monitoring tomorrow.

Tomorrow never comes. Until the next 3 AM incident.

The Solution

The Monitoring skill generates complete Prometheus + Grafana configurations — scrape targets, alert rules, recording rules, and Grafana dashboard JSON — tailored to your specific infrastructure.

The Process

You: Set up monitoring for my infrastructure:
- 5 web servers (nginx + Node.js app)
- 2 PostgreSQL databases (primary + replica)
- 1 Redis cache
- Running on Ubuntu, all have node_exporter
I want alerts for: disk space, CPU, memory, app response time,
database replication lag, Redis memory.

The agent generates the full monitoring stack configuration including:

• prometheus.yml — Scrape configs for all targets with proper intervals
• alert_rules.yml — 15+ alert rules covering infrastructure and application metrics
• recording_rules.yml — Pre-computed queries for dashboard performance
• alertmanager.yml — Slack/PagerDuty/email notification routing
• Grafana dashboard JSON — Pre-built panels for all critical metrics

Alert examples that the agent includes (and you'd never think to add until after the incident):

• Disk filling up (predict when it'll hit 100% based on growth rate)
• SSL certificate expiring within 14 days
• Database replication lag exceeding 30 seconds
• Redis memory approaching maxmemory limit
• Node.js event loop lag exceeding 100ms
• HTTP 5xx error rate exceeding 1% of requests

The Results

Setup on MrChief

skills:
  - monitoring
  - docker     # For containerized stack deployment