Scaling Agent Skill

Scaling: survive many requests at once

Performance makes one request faster. Scaling makes many requests survive at the same time. Different problem, different toolbox — don't reach for this one when a single endpoint is slow for a single user (that is ../performance/SKILL.md).

The whole job in one line: diagnose the bottleneck, apply the cheapest lever that moves it, re-measure under load. Repeat until the next bottleneck appears or you hit your target.

Prime directive: never add infrastructure without a measurement first. A replica, a queue, or a third app instance you bought on a hunch costs money every month and usually moves the wrong tier. Measure, then add.

Start here — order of operations

Levers are ordered by payoff per dollar. Caching is nearly free and wins biggest; replicas and autoscaling cost forever. Climb the ladder in order; stop the moment the symptom clears.

| Symptom | Likely bottleneck tier | First lever | Sibling that wires it |

|---|---|---|---|

| Slow *only* under load, fine solo | unknown — measure first | USE-method triage (Step 0) | ../monitoring/SKILL.md |

| Same reads recomputed for everyone | app/DB doing repeat work | Lever 1 — cache | ../redis/SKILL.md |

| too many clients already | DB connection slots | Lever 2 — pooler | ../postgresdb/SKILL.md |

| Spiky writes time out / drop | synchronous write path | Lever 2 — async queue | ../redis/SKILL.md |

| Reads dominate, primary CPU hot | DB read capacity | Lever 3 — read replica | ../postgresdb/SKILL.md |

| All tiers healthy, just need throughput | app instance count | Lever 4 — horizontal / autoscale | ../deployment/SKILL.md |

Step 0 — find the bottleneck before you add anything

Use the USE method (Utilization, Saturation, Errors) on every resource — CPU, memory, disk, network, and the DB connection pool. For each one ask: how busy (U), how much is queued/waiting (S), and any errors (E).

• Saturation predicts collapse earlier than utilization. A CPU at 70% util with a growing run-queue is closer to falling over than one at 90% with no queue. Watch queue depth, run-queue length, and connection-pool wait time — those spike before throughput craters.
• Read p95/p99, never the mean. The mean hides the tail your users actually feel; if p50 is 80 ms and p95 is 4 s, a meaningful slice of traffic is having a bad time and the average lies about it.
• The dashboards, alerts, and SLOs that produce these numbers are ../monitoring/SKILL.md / observability's job. Scaling *consumes* USE signals; it doesn't build the collectors.

Output of Step 0 is one sentence: "the bottleneck is the DB connection pool / app CPU / origin cache-miss rate." Don't proceed without it.

Lever 1 — Caching (cheapest, do first)

Cache layers, outermost to innermost — each one removes work the layer behind it would have done:

| Layer | Removes | Typical TTL |

|---|---|---|

| CDN / edge | origin round-trip for static + cacheable HTML | minutes–hours |

| HTTP cache headers (Cache-Control, ETag) | re-downloads; enables 304s | per-resource |

| App cache (in-proc / Redis) | recomputed views, serialized payloads | seconds–minutes |

| Query-result cache | repeated identical DB reads | seconds |

• Cache the expensive read, not the cheap one. Caching a 2 ms lookup adds a network hop and an invalidation bug for nothing; cache the 400 ms aggregate everyone hits.
• Target >80% hit ratio for general traffic (static-heavy/CDN routinely hits 95%+). A ratio consistently <60% means a broken strategy — wrong cache keys, TTL too low, or churny data that shouldn't be cached.
• Beware the cold-cache miss storm. On deploy or cache flush, every request misses at once and stampedes the origin — the cache that was protecting you now amplifies the load. Mitigating that (stampede protection, single-flight, request coalescing) is cache *correctness* → ../redis/SKILL.md.
• If the underlying query is just slow, caching only hides it. Fix the query — N+1, missing index — via ../performance/SKILL.md before papering over it with TTL.

Lever 2 — Connection pooling + queues

Pool DB connections. Each Postgres connection is a backend process with real memory cost; apps that open a connection per request exhaust max_connections fast.

Bad:  app → opens a fresh DB connection per request → "too many clients already"
Good: app → PgBouncer (transaction mode) → small pool of reused server connections

• Use transaction pooling for stateless web apps: a server connection is held only for the duration of a transaction and released on COMMIT/ROLLBACK — reported real-world effect is roughly 100× effective connection capacity.
• Sizing rule: keep (number_of_pools × default_pool_size) < max_connections − ~15 (leave headroom for superuser/admin slots). Set default_pool_size ≈ 1.5–2× vCores for CPU-bound OLTP — more connections than cores just adds context-switch contention, not throughput.
• Gotcha — transaction pooling breaks session-scoped features: prepared statements (pre-PG14 protocol), SET / session GUCs, advisory *session* locks, and LISTEN/NOTIFY. Route those to a session-pooling pool or refactor them out. Don't discover this in production.

Shed spiky writes into a queue. Queue-based load leveling puts a queue between a bursty producer and a constrained consumer so the consumer drains at its own steady rate; the queue absorbs the spike instead of the synchronous tier melting.

• Move anything that doesn't need a synchronous answer — emails, webhooks, thumbnails, exports — off the request path.
• Queue *semantics* (race-free rate limits, stalled-job recovery, fencing tokens, durable SKIP LOCKED queues) belong to ../redis/SKILL.md and ../postgresdb/SKILL.md. Scaling decides *that* you defer work; those decide it's done correctly.

Lever 3 — Read replicas

Reach for a replica only when Step 0 says reads dominate and the primary is read-saturated — not as a reflex.

• Replicas serve reads, never the source of truth for read-after-write. Async streaming replication lags; a user who just wrote and immediately reads from a replica may see stale data. Route post-write reads (or that user's whole session for a window) to the primary, or use replica-lag-aware routing.
• A replica is not a write-scaling story and not, by itself, an HA/backup story. Writes still all hit one primary; durability and recovery are ../backups/SKILL.md.
• Wiring the replication itself — primary_conninfo, slots, promotion — is ../postgresdb/SKILL.md. Scaling decides *add a replica and route reads to it*; postgresdb makes it real.

Lever 4 — Horizontal scaling & autoscaling

• Make the app stateless first. You can't horizontally scale what holds local state — in-memory sessions, sticky uploads on local disk, per-instance caches that must agree. Push session/state to Redis or the DB, then add instances freely.
• Autoscale on the saturation metric that actually binds, not CPU alone. If the real limit is DB connections, scaling app instances on CPU just opens *more* connections and topples the DB faster. Scale on the bottleneck Step 0 found.
• Choosing a host, rolling deploys, and the autoscaling dials are ../deployment/SKILL.md plus the platform skill (../fly-io/SKILL.md, and siblings for railway/render/vercel). Scaling gives the strategy — how many and triggered by what; the platform gives the knobs.

Prove it — load testing with k6

Don't claim the system survives. Measure that it does. k6 (Go core, JS test scripts) reached v1.0.0 on 2025-04-28 under SemVer and is the default OSS load-test tool; the current v1 line is v1.7.x and v2.0.0 shipped in May 2026 (GrafanaCON 2026).

• Thresholds are the test's pass/fail SLO — codify the target so the run goes red on its own instead of you eyeballing a graph.
• Test a prod-like target, never localhost. Localhost has no network latency, no real DB, no CDN — it measures your laptop, not your system.
• Run a ladder, not one shot: smoke → load → stress → spike → soak (find the *knee* where latency turns vertical).

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '1m', target: 50 },   // ramp up to 50 virtual users
    { duration: '3m', target: 50 },   // hold (steady-state load test)
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'], // SLO gate: 95% of requests under 500 ms
    http_req_failed: ['rate<0.01'],   // SLO gate: under 1% errors
  },
};

export default function () {
  const res = http.get(`${__ENV.TARGET_URL}/api/health`);
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

Full ladder (stage configs for each test type), CI gate snippet, and how to read the summary (p95/p99, http_req_failed, spotting the knee) are in references/load-testing-k6.md.

Anti-patterns

| Anti-pattern | Why it bites | Do instead |

|---|---|---|

| Scaling before measuring | You spend on the wrong tier; symptom persists | Step 0 USE triage, name the bottleneck first |

| Scaling a stateful app horizontally | Instances disagree; sessions vanish on routing | Make it stateless, externalize state, then scale |

| Caching cheap work / no TTL strategy | Adds a hop + invalidation bugs for no gain | Cache the expensive read; set deliberate TTLs |

| Load-testing localhost | Measures your laptop, not production | Test a prod-like target over the network |

| Reporting mean latency | Hides the tail users actually feel | Gate on p95/p99 |

| Read replica to absorb writes | Writes still hit one primary; you gain nothing | Replica is read-only; queue/shard writes |

| DB with no connection pooler | too many clients already under any spike | PgBouncer transaction mode + the sizing rule |

| Autoscaling on CPU while DB connections saturate | More instances = more connections = faster DB death | Autoscale on the binding saturation metric |

Stop rule & cost

Scale to the *next* bottleneck, then re-measure — don't pre-buy capacity for traffic you don't have. Every lever has a price: caching is ~free, a pooler is cheap, a read replica and autoscaling cost every month and add operational surface. Climb one rung, re-run the load test, and stop when you clear the target. Survived, proven, no further — that's done.