Dev Debugging

Debugging

The rule

Never fix code before you understand why it broke. The temptation to "just make the test pass" or "just patch the symptom" leads to fragile fixes that hide deeper problems. Follow the three-step workflow below every time.

Step 1: Reproduce with a test

Before touching any implementation code, write a test that captures the bug. This test must:

• Fail right now, demonstrating the broken behavior
• Be specific enough to distinguish the bug from correct behavior
• Live in the right test location (see dev-testing for where tests go)

If you can't reproduce it in a test, you don't understand it well enough to fix it. For VM-level issues, use capsem-doctor or write a targeted diagnostic command:

just exec "<command that triggers the bug>"

For telemetry issues, use session inspection:

python3 scripts/check_session.py

Step 2: Diagnose the root cause

With a failing test in hand, investigate. Do not skip this step. Common diagnostic approaches:

MCP triage trio (run FIRST when an investigation is open-ended):

capsem_panics { since: "1h" }       # any Rust panic in any host log? -> rank highest
capsem_triage { id: "vm-1" }        # ranked recent ipc-warns + 4xx/5xx + slow_ops + session.db errors
capsem_timeline { id: "vm-1", trace_id: "<X>" }   # follow ONE logical operation across exec/tool/net/fs/model

These read post-W2 JSON logs (~/.capsem/run/{service,mcp,gateway,tray}.log + capsem-app's latest jsonl) and post-W6 session.db tables. The W4 target=fs op=fsync duration_ms=... markers feed capsem_triage's slow-op rank; the W3 schema_hash check appears in capsem_panics output as IPC handshake failed; refusing connection events. Always start with capsem_panics -- a single panic outranks a hundred warns.

Cross-version mix? The service.start log line emits protocol_version=N, schema_hash=<hex> per binary. If the support bundle (capsem support-bundle) shows two different schema_hash values across binaries, you're hitting the W3 cross-version-mix detection -- rebuild + restart the lagging binary.

Integration-test failures: read the preserved service log. When any integration test fails, the test fixture (tests/helpers/service.py::ServiceInstance, the e2e RealService, and the MCP _start_capsem_service) archives its tmp_dir to test-artifacts/<timestamp>-<worker>-<nodeid>/<tmp-basename>/ before the usual rmtree. The failing test's stderr has the exact path: look for a line ARTIFACT: preserved /var/folders/... -> test-artifacts/.... Inside that directory:

service.log                     host-side capsem-service debug log (RUST_LOG=debug)
logs/gateway.log                gateway stdout/stderr
logs/tray.log                   tray stdout/stderr (if spawned)
sessions/<vm-id>/process.log    per-VM capsem-process log (vsock bridge, IPC, spawn chain)
sessions/<vm-id>/serial.log     VM serial console (kernel boot, capsem-init, agent startup)
sessions/<vm-id>/session.db     SQLite telemetry DB (net_events, model_calls, ...)
persistent/<name>/...           persistent-VM state (checkpoint.vzsave, workspace)

test-artifacts/ is gitignored. Multiple failures sharing a session-scoped service land in different subdirs but the latest run's name tags them by the most recent failing nodeid. First place to look for "VM didn't become exec-ready" style failures: sessions/<id>/serial.log (did the VM boot?) and sessions/<id>/process.log (did the agent come up + IPC handshake?). For "provision hung" or service-side contention: service.log, grep for the VM id.

Rust code: Read the code path the test exercises. Trace the data flow. Add tracing instrumentation if needed (RUST_LOG=capsem=debug). Check if the issue is in capsem-core, capsem-app, or capsem-agent.

Guest VM issues: Boot with targeted commands and inspect behavior:

just exec "capsem-doctor -k <category>"   # Run specific diagnostic category
just exec "<manual investigation command>"

Check boot logs for daemon startup failures, vsock connection issues, or timing problems.

Network/security issues: Check the network intercept path -- SNI parsing,

HTTP/DNS/model normalization, cert minting, SecurityEvent construction,

security rule evaluation, plugin execution, runtime materialization, and ledger

materialization. Do not debug by adding credential handling to formatters,

routes, DB readers, frontend transforms, or harnesses. Use session DB to see

what actually happened:

python3 scripts/check_session.py   # Check net_events for domain, decision, status_code

Frontend issues: Run just dev ui, open Chrome DevTools, check console errors, use take_screenshot to capture state. See dev-testing-frontend for the full visual verification workflow.

Build pipeline issues: Check target/build.log -- all build infrastructure (runner, code signing, generation scripts) logs here. The runner (scripts/run_signed.sh) and _generate-settings recipe both append to this file. Never write diagnostics to stdout from build scripts (it contaminates binary output like mcp-export).

Telemetry pipeline issues: The canonical session ledgers (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events) each have their own boundary. If a table is empty or has wrong data:

• Check if the guest daemon started (boot logs)
• Check if the vsock connection was accepted (host logs)
• Check timing -- did the VM shut down before the debouncer flushed? (add sleep 1)

For route latency or stale stats, do not add service-owned logged-data

projections. Logged-data hot state belongs inside the logger DB object as

table-level mem/disk ownership with DB-layer tests and benchmarks. Service

routes may describe the query they need, but production service code must not

open rusqlite connections or DbReader directly.

Do not "fix" route latency by hardcoding route-specific query helpers in

DbWriter, by adding service caches, or by swallowing missing tables/columns as

empty data. The correct diagnosis target is the DB object: connection/thread

ownership, mem/disk layout, batching, flush, rehydration, and query execution.

If the schema is missing, surface the broken ledger contract.

Write down what you find. The diagnosis should explain *why* the bug exists, not just *where* the symptom appears.

Concurrency flakes are product bugs, not test-tuning problems

just test runs the python suite under pytest -n 4 --dist=loadfile. Four real VMs boot in parallel; this is dogfooding. Capsem ships as a multi-VM sandbox for AI agents -- if the test suite cannot safely run 4 concurrent VMs, real users running an agent farm will hit the same bug. When a test flakes only under concurrency, the diagnosis target is Capsem's product code, not the test:

• "Suspend timed out" appearing only at -n 4 -> handle_suspend IPC race; investigate the with_quiescence path and the Suspend round-trip, not the test timeout
• "Session did not become ready" only with multiple parallel provisions -> Apple VZ resource contention, VirtioFS lock, or service handle_provision serialization gap
• Two tests collide on the same VM/session name -> validate_vm_name / persistent registry has a TOCTOU; UUID prefix in the test is not the bug
• "Connection refused" on a per-VM UDS only at -n 4 -> service spawned the process but didn't wait for the socket to be bound; race in the spawn path
• A test passes serial but hangs at n=4 -> a global lock somewhere (state mutex held across an await, blocking Tokio worker; or a sync std::Mutex on a hot path)

Anti-patterns to avoid:

• Adding time.sleep in the test "to let things settle"
• Bumping a per-test timeout from 30s to 120s "because it's flaky"
• Marking the test serial -- defeats the dogfooding signal
• Adding retries with backoff in the client

Right pattern: capture a service log of the failing run (set RUST_LOG=capsem=trace), find the operation that took unexpectedly long or returned an error, fix the underlying race in capsem-service / capsem-process / capsem-core. Then re-run at -n 4 to confirm.

Step 2.5: Fix the pattern, not the instance

When diagnosis reveals a systemic pattern (the same mistake repeated across the codebase), the fix must cover every instance -- not just the one that was reported.

• Audit the entire codebase for the same pattern. If blocking I/O in async context caused one hang, grep for every other site that does the same thing. A bug is a symptom -- the pattern is the disease.
• Never simplify a fix to the minimum diff. A "quick fix" that patches one call site while 6 others have the identical problem is not a fix -- it's deferred breakage.
• Document the pattern in the relevant skill (e.g., dev-rust-patterns) so it's never reintroduced.
• Add tests that would catch the pattern if it recurs (e.g., a contract test between the frontend and backend response format).

Example: Snapshot MCP hang was caused by blocking I/O (clonefile, walkdir, blake3) on tokio worker threads. The same anti-pattern existed in 7 file tool handlers, the auto-snapshot timer, and asset hash verification. Fixing only the reported snapshots_create call would have left 9 other sites broken.

Step 3: Fix with a comprehensive solution

Now that you understand the root cause, write the fix. The fix should:

• Make your reproducing test pass
• Not break any existing tests (just test)
• Address the root cause, not just the symptom
• Include the test from Step 1 in the same commit

After the fix, run the full validation:

• just test -- unit + cross-compile + frontend
• just exec "capsem-doctor" -- VM smoke test
• If the bug touched telemetry: python3 scripts/check_session.py after a real session

Local/CI execution parity

When a bug appears only in CI, first identify the exact production entrypoint,

runner dependency, architecture, environment variable, permission, device, or

service-manager difference that local testing skipped. Reproduce every portable

Linux difference in Docker and execute the same production entrypoint or shared

predicate that CI executes. A hand-written approximation is not a regression

test.

Keep an executable parity test after the fix and audit sibling workflows for

the same one-sided assumption. If reproduction crosses an unavoidable platform

boundary, document the boundary and preserve the nearest local contract plus

the required owning release-job or physical-machine proof. Do not relabel an

unreproduced CI failure as transient.

What NOT to do

• Do not "fix" a failing test by changing the test assertion. The test is telling you something. Listen to it. If the test is genuinely wrong, explain why in detail before changing it.
• Do not dismiss failures as "pre-existing" or "unrelated." Investigate every failure. If it truly is pre-existing, file it and fix it -- don't leave broken windows.
• Do not guess-and-check. Random changes hoping something sticks waste time and often introduce new bugs. Understand first, then act.
• Do not patch symptoms. If requests fail because gzip content-encoding isn't handled, don't strip the Accept-Encoding header -- implement proper decompression. Fix the system, not the surface.
• Do not apply narrow fixes to systemic problems. If the same anti-pattern exists in 7 places and you fix 1, you haven't fixed the bug -- you've hidden 6 more. Audit first, then fix all instances in a single pass.