Dev Benchmark

Benchmarking

Quick start

just exec "capsem-bench snapshot"    # Snapshot benchmarks only
just exec "capsem-bench disk"        # Disk I/O only
just exec "capsem-bench storage"     # Storage split diagnostics
just exec "capsem-bench-rs protocol" # Rust HTTP/model/MCP/DNS protocol benchmark
just test                           # Full validation including benchmarks

capsem-bench

capsem-bench all is the guest benchmark contract. Hot protocol benchmarks are

implemented only by the Rust binary capsem-bench-rs; the Python wrapper may

orchestrate them for all, but must not generate HTTP/protocol/throughput

numbers itself. Do not add Python load generation as release truth for HTTP,

DNS, MCP/tools, model/SSE, or credential-broker paths. Python guest modules

still cover legacy disk/rootfs, storage, startup, and snapshot modes until

those modes are ported.

Structured JSON is saved to /tmp/capsem-benchmark.json for machine

consumption. Hot protocol artifacts must include lane metadata so the same

scenario can be compared as host_direct and guest_capsem.

Locations:

• Rust hot benchmark binary: crates/capsem-bench
• Legacy guest modules: guest/artifacts/capsem_bench/

Benchmark categories

| Category | Command | What it measures |

|----------|---------|-----------------|

| disk | capsem-bench disk | Sequential/random I/O on scratch disk (write/read throughput, IOPS) |

| rootfs | capsem-bench rootfs | Read-only rootfs performance: largest-file sequential read, random 4K reads, large-binary sequential reads, small JS/package reads, and metadata stat-walk throughput |

| storage | capsem-bench storage | Diagnostic split across rootfs reads and writable paths such as /root, /tmp, /var/tmp, /var/log, and /run |

| startup | capsem-bench startup | Cold-start latency for python3, node, claude, gemini, codex |

| protocol/http | capsem-bench-rs protocol | HTTP, model/SSE, credential, MCP/tools, and DNS protocol scenarios against capsem-mock-server |

| snapshot | capsem-bench snapshot | Snapshot create/list/changes/revert/delete via MCP (ms per op at 10/100/500 files) |

| all | capsem-bench all | Default production suite including storage split diagnostics; with CAPSEM_MOCK_SERVER_BASE_URL, calls capsem-bench-rs protocol and merges the Rust network sections |

Abstraction delta

Every hot protocol benchmark needs paired lanes:

• host_direct: host Rust capsem-bench-rs directly against capsem-mock-server
• guest_capsem: guest Rust capsem-bench-rs through the real Capsem network path

The delta artifact must report the cost of Capsem's abstraction for each

scenario: RPS ratio, throughput ratio, p50/p95/p99 latency delta, and error

delta. If a benchmark cannot produce both lanes, it is diagnostic only and not

a release performance gate.

Intentional host-side benchmark runs record artifacts under

benchmarks/lifecycle/, benchmarks/fork/, and benchmarks/route-latency/.

The route-latency artifact measures /stats reads while public profile

mutation routes are writing through the DB boundary. Treat it as the control

plane contention baseline for UI/TUI responsiveness.

Historical host-native benchmark runs may also exist under

benchmarks/host-native/ with local disk I/O, CLI startup, synthetic small-file

reads, metadata-stat throughput, filesystem context, UTC timestamp, host

hardware/OS metadata, and git state. Use this when comparing VM performance

against the hardware that produced the run. The default host I/O directory is

target/host-native-benchmark, not /tmp, so Linux tmpfs does not become the

accidental baseline. Override with CAPSEM_HOST_NATIVE_BENCH_DIR for a specific

disk.

Benchmark history is not managed by a Just convenience command. Run the owning

pytest or Criterion command, review the produced JSON, and commit an intentional

baseline update without overwriting prior evidence.

capsem-bench all includes the storage section. Keep that in the canonical

path so Linux and macOS artifacts both capture rootfs/workspace/tmpfs

attribution data; only the long-running load diagnostics stay opt-in.

Cross-platform comparison

Use python3 scripts/benchmark_report.py <artifacts...> to validate committed

artifacts and optionally render graphs. Compare Linux x86_64 against macOS

arm64 only after both platform lanes rerun the same owning benchmark.

Snapshot benchmarks

Tests the full MCP snapshot pipeline end-to-end (guest CLI -> MCP server -> vsock -> host gateway -> filesystem). Measures at 3 workspace sizes (10, 100, 500 files):

• create: Populate workspace, create named snapshot via MCP
• list: List all snapshots with change diffs
• changes: List changed files since checkpoint
• revert: Revert a single file from snapshot
• delete: Delete the snapshot

Key metrics: per-operation latency in ms. Regressions in create usually mean the clone or hash stage got slower. Use RUST_LOG=capsem=debug to see per-stage breakdown (clone_ws_ms, clone_sys_ms, hash_ms).

JSON output format

{
  "version": "0.3.0",
  "timestamp": 1711561234.5,
  "hostname": "capsem",
  "disk": { "seq_write_mbps": 450, ... },
  "rootfs": { ... },
  "startup": { "python3": { "min_ms": 45, "mean_ms": 48, "max_ms": 52 }, ... },
  "http": { "rps": 120, "p50_ms": 42, ... },
  "throughput": { "throughput_mbps": 85, ... },
  "snapshot": {
    "10_files": { "create_ms": 120, "list_ms": 50, ... },
    "100_files": { "create_ms": 250, ... },
    "500_files": { "create_ms": 800, ... }
  }
}

Environment variables

• CAPSEM_BENCH_DIR: Test directory for disk benchmarks (default: /root)
• CAPSEM_BENCH_SIZE_MB: Write test size in MB (default: 256)
• CAPSEM_STORAGE_BENCH_PATHS: Colon-separated writable paths for storage split diagnostics (default: /root:/tmp:/var/tmp:/var/log:/run)
• CAPSEM_STORAGE_BENCH_SIZE_MB: Write test size in MB for each storage split writable path (default: 64)
• CAPSEM_STORAGE_IO_PROFILE_SIZE_MB: File size in MB for detailed sequential/random storage IOPS profiling (default: 64)
• CAPSEM_STORAGE_IO_PROFILE_RANDOM_OPS: Random read/write operation count for storage IOPS profiling (default: 2000)

Investigating slowness

Snapshot performance

• Run snapshot benchmark: just exec "capsem-bench snapshot"
• Check per-stage timing: RUST_LOG=capsem=debug just exec "capsem-bench snapshot" -- look for snapshot_into_slot timing log lines showing clone_ws_ms, clone_sys_ms, hash_ms
• Check session data: python3 scripts/check_session.py -- MCP tool usage section shows avg duration per snapshot operation
• Query detailed durations: sqlite3 "$HOME/.capsem/sessions/<id>/session.db" "SELECT tool_name, duration_ms FROM tool_calls WHERE origin = 'mcp' AND tool_name LIKE 'snapshot%' ORDER BY duration_ms DESC LIMIT 20"

Common causes:

• clone_ws_ms high: Large workspace, or APFS clonefile falling back to byte copy
• hash_ms high: Many files in workspace (walkdir overhead), or slow filesystem
• compact slow: Merging many snapshots with overlapping files

Disk I/O regression

• Run: just exec "capsem-bench disk"
• Compare sequential write/read throughput against baseline
• Check if VirtioFS mode changed (block mode has different I/O characteristics)

Storage split regression

• Run: just exec "capsem-bench storage"
• Compare /root against /tmp, /var/tmp, /var/log, and /run to separate VirtioFS workspace costs from tmpfs, overlay, and rootfs read costs
• Check storage.kernel for /proc/cmdline, virtio block queue settings, FUSE connection backpressure knobs, and known host-side KVM queue sizes
• Check storage.rootfs.backing.erofs_mounts for the booted EROFS rootfs before comparing Linux/macOS rootfs reads; SquashFS fields are historical diagnostics only, not the 1.3 release gate
• Compare the detailed I/O profile: sequential 4K/64K/1M IOPS/MB/s, random 4K read IOPS, and random 4K sync-write IOPS with p95 latency
• Use the reported mount table to confirm which filesystem backs each path before assigning blame to KVM, VirtioFS, overlayfs, or the host filesystem

Rootfs read regression

• Run: just exec "capsem-bench rootfs"
• Compare rootfs.seq_read for the historical largest-file sequential read gate
• Compare rootfs.large_binary_seq_read to isolate large CLI binary reads
• Compare rootfs.small_js_read for loader-style reads across many small JS/JSON/package files
• Compare rootfs.metadata_stat for thousands of lstat calls across the rootfs tree
• Keep rootfs.rand_read_4k as the broad mixed-file random-read signal

Adding a new benchmark

• Create a new module in guest/artifacts/capsem_bench/ (e.g., mytest.py) with a mytest_bench() function that returns a dict and prints a Rich table
• Add the mode name to VALID_MODES in __main__.py
• Wire it into main() with the if mode in ("name", "all"): pattern (lazy import)
• Update this skill and the benchmarking doc page

Host-side lifecycle benchmark

Profiles individual VM lifecycle operations from the host. Runs outside the guest via pytest, not via capsem-bench.

uv run pytest tests/capsem-serial/test_lifecycle_benchmark.py -xvs

Location: tests/capsem-serial/test_lifecycle_benchmark.py

Operations measured

| Operation | What it times |

|-----------|--------------|

| provision | HTTP POST /provision to service (VM creation + process spawn) |

| exec_ready | First echo ready exec succeeds (VM boot + vsock handshake) |

| exec | Simple echo ok on a running VM |

| delete | HTTP DELETE /delete/{name} (VM teardown + cleanup) |

Output

• Per-run breakdown printed to stdout
• Summary table with min/mean/max per operation
• JSON saved to benchmarks/lifecycle/data_{version}.json (committed to git for historical tracking)

Regression gates

Every operation must complete in under 1.2 seconds. The test runs 3 cycles and asserts each individual operation stays under the gate.

Host-side route latency benchmark

Profiles hot service/gateway read endpoints and DB contention using persistent

HTTP clients instead of curl helpers so process startup does not pollute timing.

This is the TUI/control-plane hot-path gate.

uv run pytest tests/ironbank/test_route_latency.py -q -s
uv run pytest tests/capsem-serial/test_route_latency_benchmark.py -q -s

Gate location: tests/ironbank/test_route_latency.py

Artifact location: tests/capsem-serial/test_route_latency_benchmark.py

Benchmark output: benchmarks/route-latency/data_<version>.json

Endpoint groups

| Group | What it covers | Default gate |

|-------|----------------|--------------|

| service_hot | /status, /vms/list, /stats, profile assets/plugins/enforcement/detection/MCP/security routes | route-specific p95 <= 2-3ms, max <= 5-8ms |

| gateway_hot | Gateway proxy for the same hot control routes | route-specific p95 <= 3-4ms, max <= 8-10ms |

| db_contention | /stats reads while PATCH /profiles/code/mcp/default/edit writes profile mutation ledger rows | Ironbank gate: p95 <= 15ms, max <= 40ms. Release artifact gate: p95 <= 15ms, p99 <= 40ms, max archived for visibility |

When to run

• After changes to /list, /status, /info, history, files, settings,

profile, rule, detection, enforcement, setup, skills, or gateway proxy paths

• After adding TUI polling, dashboard, tray, or gateway aggregation behavior
• Before release when claiming local control-plane responsiveness

Host-side fork benchmark

Profiles fork (image creation) and boot-from-image. Same test file, separate test function.

uv run pytest tests/capsem-serial/test_lifecycle_benchmark.py::test_fork_benchmark -xvs

Operations measured

| Metric | What it measures | Gate |

|--------|-----------------|------|

| fork | POST /fork/{id} — APFS clonefile of rootfs overlay + workspace | < 500ms |

| image_size | Actual disk usage of forked image (blocks, not logical size) | < 12MB |

| boot_provision | POST /provision with image param — clone image into new session | < 1200ms |

| boot_ready | First exec succeeds on the image-booted VM | < 1200ms |

| pkg_survived | Packages installed via apt survive fork (rootfs overlay) | must pass |

| ws_survived | Files written to /root/ survive fork (VirtioFS workspace) | must pass |

Output

• Per-run breakdown with timing + survival status
• Summary table with min/mean/max + gate thresholds
• JSON saved to benchmarks/fork/data_{version}.json (committed to git for historical tracking)

When to run

• After changes to fork/image code (capsem-core/src/image.rs)
• After changes to VirtioFS session layout (capsem-core/src/lib.rs)
• After changes to disk usage reporting (session/maintenance.rs)
• After changes to boot-from-image path in capsem-service or capsem-process
• Before cutting a release

When to run (lifecycle)

• After changes to boot path (capsem-process, capsem-init, capsem-core/vm/boot.rs)
• After changes to VM teardown / delete path
• After changes to the service daemon (capsem-service)
• Before cutting a release

Host-side Security Engine benchmark

Profiles Security Engine hot-path costs with Rust Criterion and VM-originated

enforcement through real service, process, and network transport paths.

cargo bench -p capsem-security-engine --bench security_engine_cel
cargo bench -p capsem-core --bench security_packs

The capsem-security-engine harness measures canonical CEL compile/evaluate,

detection evaluation, backtest evidence dedupe, runtime registry projection,

compiled-plan rebuilds, policy-context projection/materialization, 100-rule

last-match paths, and native lookup comparators. The capsem-core security-pack

harness measures Detection IR V1 JSON parse/validate, Detection IR to CEL

detection-rule lowering, and lower-plus-compile costs.

Intentional Criterion publication archives both harnesses from

target/criterion/**/new/{benchmark,estimates}.json into

benchmarks/security-engine/data_{version}_{arch}_cel_microbench.json and

benchmarks/security-engine/data_{version}_{arch}_security_packs_microbench.json;

do not rely on terminal output as the durable record.

Profiles VM-originated Security Engine enforcement through real service,

process, and network transport paths. This is outside the guest via pytest, not

via capsem-bench.

uv run pytest tests/capsem-serial/test_security_engine_benchmark.py -xvs

Location: tests/capsem-serial/test_security_engine_benchmark.py

Operations measured

| Operation | What it times |

|-----------|---------------|

| blocked_process_exec | Service API exec request -> capsem-process IPC -> process SecurityEvent projection -> CEL enforcement block -> response |

| blocked_http_request | Guest curl -> network transport/MITM -> HTTP SecurityEvent projection -> CEL enforcement block -> response |

| keepalive_blocked_http_request | Guest Python TLS client -> one persistent MITM TLS connection -> repeated HTTP SecurityEvent projection -> CEL enforcement block -> response |

| blocked_dns_request | Guest resolver -> capsem DNS proxy -> DNS SecurityEvent projection -> CEL enforcement block -> NXDOMAIN response |

| blocked_mcp_request | Guest /run/capsem-mcp-server -> framed vsock MCP endpoint -> MCP SecurityEvent projection -> CEL enforcement block -> JSON-RPC denial |

Output

• Per-run blocked exec latencies
• Per-run blocked HTTP request latencies
• Per-run blocked DNS request latencies
• Per-run blocked MCP request latencies
• JSON saved to

benchmarks/security-engine/data_{version}_{arch}_{workload}.json

with command, commit, host, rule, assertion, and latency metadata

Regression gates

The first gross-regression gates assert mean blocked process exec latency stays

under 750ms and mean blocked HTTP request latency stays under 1,000ms. The

artifacts also verify runtime match counters, canonical session.db security

rows, and logs attribution. HTTP artifacts include guest wall-clock timing,

curl phase timing/deltas, and a persistent keep-alive lane. Use the

post-pretransfer first-byte delta and keep-alive first-byte timing to reason

about MITM/Security Engine response cost instead of raw guest curl wall time.

The keep-alive lane also guards against bursty same-millisecond logging

collapsing security_events rows. DNS artifacts additionally verify

dns_events policy fields and security-log qname projection. MCP artifacts

verify tool_calls policy fields and request-id-matched server/tool log

projection.

When to run

• After changes to capsem-security-engine
• After changes to Detection IR parsing/lowering in capsem-core
• After changes to process security event projection or exec dispatch
• After changes to DNS proxy runtime enforcement or dns_events logging
• After changes to runtime enforcement rule propagation/counters
• After changes to security_events logging or capsem logs
• Before making release or marketing claims about Security Engine latency

Tests

• In-VM benchmark test: just exec "capsem-bench all"
• In-VM availability: test_utilities.py::test_utility_available[capsem-bench]
• Host-side lifecycle: uv run pytest tests/capsem-serial/test_lifecycle_benchmark.py::test_lifecycle_benchmark -xvs
• Host-side fork: uv run pytest tests/capsem-serial/test_lifecycle_benchmark.py::test_fork_benchmark -xvs
• Host-side endpoint latency: uv run pytest tests/capsem-serial/test_endpoint_latency_benchmark.py -xvs
• Host-side Security Engine: uv run pytest tests/capsem-serial/test_security_engine_benchmark.py -xvs
• Both host-side: uv run pytest tests/capsem-serial/test_lifecycle_benchmark.py -xvs
• Full release run: just test

Benchmark data directory

Host-side benchmarks save arch-scoped JSON to benchmarks/ (committed to git

for performance baselines). Set CAPSEM_BENCHMARK_RUN_ID for an

intentional named run and CAPSEM_BENCHMARK_OUTPUT_DIR for exploratory runs

that should not dirty the checkout:

benchmarks/
  fork/data_1.2.3_x86_64_linux-rc1.json          # Fork speed, image size, data survival
  lifecycle/data_1.2.3_x86_64_linux-rc1.json     # Provision, exec-ready, exec, delete
  endpoint-latency/data_*.json   # Service/gateway read latency across 8 live VMs
  security-engine/data_*.json    # CEL microbench and VM-originated enforcement

These data files feed the documentation benchmark page at docs/src/content/docs/benchmarks/results.md. Before a release, run both benchmarks and update the results page with the new numbers. See /release-process for the full checklist.