Render Backend Monitoring — The Agent Caught 2 Incidents Before I Woke Up

# Health check
response=$(curl -s -o /tmp/health.json -w "%{http_code}|%{time_total}" \
  https://eskimoai-api.onrender.com/health/)

status_code=$(echo "$response" | cut -d'|' -f1)
response_time=$(echo "$response" | cut -d'|' -f2)

# Alert thresholds
if [ "$status_code" != "200" ] || [ $(echo "$response_time > 2.0" | bc) -eq 1 ]; then
  trigger_incident "$status_code" "$response_time"
fi

# Fetch service metrics from Render API
curl -s -H "Authorization: Bearer $RENDER_API_KEY" \
  "https://api.render.com/v1/services/$SERVICE_ID/metrics?period=1h" \
  | jq '{cpu: .cpu, memory: .memory, requests: .requests}'

🚨 INCIDENT — eskimoai-api DOWN
  Time: 03:17 UTC
  Status: 502 Bad Gateway
  Duration: Health check failed (was healthy 5 min ago)

  📋 Render Logs (last 10 lines):
  > [ERROR] Worker pid:12 exited with code 137 (OOM killed)
  > [ERROR] Worker pid:14 exited with code 137 (OOM killed)
  > [INFO] No workers available. Returning 502.

  🔍 Diagnosis: Out-of-Memory kill. Both Gunicorn workers
  exceeded the 512MB memory limit simultaneously.

  Recent deploys: v2.1.8 deployed 6 hours ago — includes
  new PDF export endpoint.

  💡 Suggested fix:
  1. Immediate: Restart service (clears memory)
  2. Investigate: PDF export likely loads full documents into
     memory. Add streaming/chunked response.
  3. Consider: Upgrade from 512MB to 1GB if PDF exports
     are expected usage.

  🔄 Auto-restarting service now...

curl -s -X POST \
  -H "Authorization: Bearer $RENDER_API_KEY" \
  "https://api.render.com/v1/services/$SERVICE_ID/restart"

✅ eskimoai-api recovered
  Time: 03:19 UTC
  Downtime: 2 minutes
  Health check: 200 OK (response: 340ms)

  ⚠️ Root cause unresolved. PDF export endpoint needs memory fix.
  Created issue #67: "PDF export OOM — add streaming response"

⚠️ WARNING — eskimoai-api memory trending up
  Current: 78% of 512MB limit (398MB)
  30 min ago: 62% (317MB)
  1 hour ago: 45% (230MB)

  📈 Growth rate: 2.8MB/minute — will OOM in ~40 minutes

  🔍 Analysis: Access log shows /api/v1/export/report
  called 47 times in the last hour (automated client).
  Each call increases RSS by ~3.5MB without release.

  💡 Diagnosis: Memory not released after PDF generation.
  Likely cause: file handles or BytesIO buffers not closed.

  Suggested:
  1. Restart now (preventive, before OOM)
  2. Rate-limit /export/report to 10 req/min
  3. Fix: ensure `with` context managers on all file
     operations in export views

  🔄 Restarting preemptively...

📊 eskimoai-api Weekly Health (Mar 2-8)

  Uptime: 99.7% (2 incidents, 5 min total downtime)
  Avg response time: 245ms (P95: 890ms)
  Total requests: 47,302
  Error rate: 0.3%

  Memory: Avg 45% | Peak 78% (during incident)
  CPU: Avg 12% | Peak 34%

  📈 Traffic up 40% week-over-week.
  💡 Recommendation: Consider upgrading from Starter ($7/mo)
  to Standard ($25/mo) if growth continues. Current headroom
  is thin for memory-intensive endpoints.