Prd V08 Runbook Creation Skill for Claude

RUN-001: Database Connection Pool Exhaustion
Category: Incident
Trigger: MON-005 alert (connection pool >90%) — from v0.8 Monitoring Setup
Owner: Backend Team
Last Tested: 2025-02-20

## Scope
- **Handles**: Connection pool saturation, slow queries causing pooling
- **Does NOT handle**: Database server crash (see RUN-010), Network failure (see RUN-011)

## Prerequisites
- [ ] Access to AWS RDS console
- [ ] PostgreSQL read credentials in LastPass
- [ ] PagerDuty access for escalation
- [ ] Datadog dashboard access (MON-005 source)

## Procedure

### Step 1: Verify Alert
Check current connection pool status:

Commands:
\`\`\`sql
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
SELECT * FROM pg_stat_activity WHERE state = 'active' ORDER BY query_start;
\`\`\`

Verification:
- [ ] Connection count ≥90% of max pool (check DEP-001 pool config)

### Step 2: Identify Problematic Queries
Find long-running or blocked queries:

Commands:
\`\`\`sql
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE (now() - pg_stat_activity.query_start) > interval '5 minutes';
\`\`\`

Verification:
- [ ] At least one query identified running >5 minutes

### Step 3: Kill Problematic Queries (if safe)
Only kill queries that are clearly stuck:

Commands:
\`\`\`sql
SELECT pg_terminate_backend(pid) FROM pg_stat_activity
WHERE pid = <problematic_pid>;
\`\`\`

Verification:
- [ ] Connection count dropping within 2 minutes
- [ ] MON-005 alert resolving

### Step 4: Investigate Root Cause
- Check recent deployments (last 24h via git log)
- Review application logs for query patterns (Datadog logs)
- Check for missing indexes on recent queries

## Escalation
- **When to escalate**: Issue persists >15 minutes, data integrity concern, cannot kill queries safely
- **Who to contact**: Database Team Lead (Slack: @db-team, PagerDuty)
- **What to provide**: Timeline, queries identified, actions taken, connection count trend

## Post-Incident
- [ ] Document incident timeline (who was paged, when actions taken)
- [ ] File ticket for query optimization if needed
- [ ] Update this runbook if steps were wrong/missing
- [ ] Schedule team drill of this runbook within 1 week if escalated

Linked IDs: MON-005 (alert), DEP-001 (pool config), RISK-008 (data integrity)

---

RUN-002: Production Deployment Procedure
Category: Deployment
Trigger: Scheduled release when all DEP- criteria met
Owner: DevOps Team
Last Tested: 2025-02-18

## Scope
- **Handles**: Standard production deployments (mainline releases)
- **Does NOT handle**: Hotfix deployments (see RUN-003), Database migrations (see RUN-004), Emergency rollback (see RUN-005)

## Prerequisites
- [ ] All DEP- criteria verified
  - [ ] DEP-002: All tests pass in staging
  - [ ] DEP-003: No critical RISK- blockers
  - [ ] DEP-004: Security review complete
- [ ] Staging deployment successful
- [ ] Release approval from Tech Lead in #deployments channel
- [ ] On-call engineer available for rollback (next 30 minutes)

## Procedure

### Step 1: Pre-Deploy Announcement
Notify stakeholders of upcoming deployment:

Commands:
\`\`\`bash
./scripts/notify-deploy.sh --env production --version v${VERSION} --channel #deployments
\`\`\`

Verification:
- [ ] Message posted to #deployments

### Step 2: Create Deployment Checkpoint
Tag current production for rollback (per DEP-003):

Commands:
\`\`\`bash
git tag -a "pre-deploy-$(date +%Y%m%d-%H%M)" -m "Checkpoint before v${VERSION}"
git push origin --tags
\`\`\`

Verification:
- [ ] Tag created and pushed to git

### Step 3: Execute Deployment
Deploy to production using CI/CD:

Commands:
\`\`\`bash
./scripts/deploy.sh --env production --version v${VERSION}
\`\`\`

Verification:
- [ ] Deployment pipeline exits with status 0
- [ ] New version visible: curl https://api.prod.example.com/health | jq .version

### Step 4: Post-Deploy Validation
Run smoke tests (per DEP-004):

Commands:
\`\`\`bash
./scripts/smoke-test.sh --env production --suite critical
\`\`\`

Verification:
- [ ] All smoke tests pass
- [ ] Manual spot-check: key UJ- flows work (test signup, login, create report)
- [ ] Error rate within baseline: curl https://api.prod.example.com/metrics | jq .error_rate_5m

### Step 5: Monitor for 30 Minutes
Watch dashboards for anomalies:

Verification:
- [ ] No new critical/warning alerts (check Datadog/Slack #alerts)
- [ ] Latency (MON-001) within normal range (<500ms p95)
- [ ] Error rate (MON-002) within baseline (<0.5%)
- [ ] Business metrics dashboard shows expected traffic

## Escalation
- **When to escalate**: Smoke tests fail, error rate >2%, user reports critical issue, latency >2s
- **Who to contact**: On-call engineer (PagerDuty), then Tech Lead
- **What to provide**: Version deployed, failure mode, logs from Datadog, timeline

## Post-Deployment
- [ ] Post deployment success to #deployments
- [ ] Update deployment log in runbook folder
- [ ] If issues: Escalate and execute RUN-005 (emergency rollback)

Linked IDs: DEP-001/002/003/004 (release criteria), RUN-005 (emergency rollback)

RUN-XXX: [Runbook Title]
Category: [Incident | Deployment | Maintenance | Recovery | Escalation]
Trigger: [What initiates this runbook]
Owner: [Team or role responsible]
Last Tested: [Date of last drill/use]

## Scope
- **Handles**: [What scenarios this covers]
- **Does NOT handle**: [Explicit exclusions]

## Prerequisites
- [ ] Access to [system/tool]
- [ ] Credentials for [service]
- [ ] Contact info for [team]

## Procedure

### Step 1: [Action Title]
[Detailed instructions]

Commands:

Verification:
- [ ] [How to confirm step succeeded]

### Step 2: [Action Title]
[Detailed instructions]

### Step N: [Final Action]
[Detailed instructions]

## Escalation
- **When to escalate**: [Conditions that require help]
- **Who to contact**: [Name/role, contact method]
- **What to provide**: [Information needed for handoff]

## Post-Incident
- [ ] Document incident timeline
- [ ] Update runbook if steps were wrong/missing
- [ ] Schedule post-mortem if severity > [threshold]

Linked IDs: [MON-XXX, DEP-XXX, RISK-XXX related]

RUN-001: Database Connection Pool Exhaustion
Category: Incident
Trigger: MON-005 alert (connection pool >90%)
Owner: Backend Team
Last Tested: 2025-01-15

## Scope
- **Handles**: Connection pool saturation, slow queries causing pooling
- **Does NOT handle**: Database server crash (see RUN-010)

## Prerequisites
- [ ] Access to AWS RDS console
- [ ] Database read credentials
- [ ] PagerDuty access for escalation

## Procedure

### Step 1: Verify Alert
Check current connection pool status:

Commands:

Verification:
- [ ] Connection count matches alert threshold

### Step 2: Identify Problematic Queries
Find long-running or blocked queries:

Commands:

### Step 3: Kill Problematic Queries (if safe)
Only kill queries that are clearly stuck:

Commands:

Verification:
- [ ] Connection count dropping
- [ ] MON-005 alert resolving

### Step 4: Investigate Root Cause
- Check recent deployments (last 24h)
- Review application logs for query patterns
- Check for missing indexes on recent queries

## Escalation
- **When to escalate**: Issue persists >15 minutes, data integrity concern
- **Who to contact**: Database Team Lead (Slack: @db-team, PagerDuty)
- **What to provide**: Timeline, queries identified, actions taken

## Post-Incident
- [ ] Document incident timeline
- [ ] File ticket for query optimization if needed
- [ ] Update this runbook if steps were wrong/missing

Linked IDs: MON-005, DEP-001, RISK-008

RUN-002: Production Deployment Procedure
Category: Deployment
Trigger: Scheduled release, all DEP- criteria met
Owner: DevOps Team
Last Tested: 2025-01-20

## Scope
- **Handles**: Standard production deployments
- **Does NOT handle**: Hotfix deployments (see RUN-003), Database migrations (see RUN-004)

## Prerequisites
- [ ] All DEP- criteria verified (DEP-002, DEP-003)
- [ ] Staging deployment successful
- [ ] Release approval in deployment channel
- [ ] On-call engineer available for rollback

## Procedure

### Step 1: Pre-Deploy Announcement
Notify stakeholders of upcoming deployment:

Commands:

### Step 2: Create Deployment Checkpoint
Tag current production for rollback:

Commands:

### Step 3: Execute Deployment
Deploy to production using CI/CD:

Commands:

Verification:
- [ ] Deployment pipeline green
- [ ] New version visible in health check endpoint

### Step 4: Post-Deploy Validation
Run smoke tests and verify key flows:

Commands:

Verification:
- [ ] All smoke tests pass
- [ ] Key UJ- flows verified manually
- [ ] Error rate within baseline (MON-001)

### Step 5: Monitor for 15 Minutes
Watch dashboards for anomalies:
- Error rate dashboard
- Latency dashboard
- Business metrics dashboard

Verification:
- [ ] No new alerts
- [ ] Metrics within normal range

## Escalation
- **When to escalate**: Smoke tests fail, error rate >2%, user reports
- **Who to contact**: On-call engineer, then Tech Lead
- **What to provide**: Deployment version, failure mode, logs

## Post-Incident
- [ ] Post deployment success/failure to #deployments
- [ ] Update deployment log
- [ ] Schedule retro if issues encountered

Linked IDs: DEP-001, DEP-002, DEP-003, MON-001