Debug

Debug

Goals

• Find why a run is stuck, retrying, or failing.
• Correlate Linear issue identity to a Codex session quickly.
• Read the right logs in the right order to isolate root cause.

Log Sources

• Primary runtime log: log/symphony.log
• Default comes from SymphonyElixir.LogFile (log/symphony.log).
• Includes orchestrator, agent runner, and Codex app-server lifecycle logs.
• Rotated runtime logs: log/symphony.log*
• Check these when the relevant run is older.

Correlation Keys

• issue_identifier: human ticket key (example: MT-625)
• issue_id: Linear UUID (stable internal ID)
• session_id: Codex thread-turn pair (<thread_id>-<turn_id>)

elixir/docs/logging.md requires these fields for issue/session lifecycle logs. Use

them as your join keys during debugging.

Quick Triage (Stuck Run)

• Confirm scheduler/worker symptoms for the ticket.
• Find recent lines for the ticket (issue_identifier first).
• Extract session_id from matching lines.
• Trace that session_id across start, stream, completion/failure, and stall

handling logs.

• Decide class of failure: timeout/stall, app-server startup failure, turn

failure, or orchestrator retry loop.

Commands

# 1) Narrow by ticket key (fastest entry point)
rg -n "issue_identifier=MT-625" log/symphony.log*

# 2) If needed, narrow by Linear UUID
rg -n "issue_id=<linear-uuid>" log/symphony.log*

# 3) Pull session IDs seen for that ticket
rg -o "session_id=[^ ;]+" log/symphony.log* | sort -u

# 4) Trace one session end-to-end
rg -n "session_id=<thread>-<turn>" log/symphony.log*

# 5) Focus on stuck/retry signals
rg -n "Issue stalled|scheduling retry|turn_timeout|turn_failed|Codex session failed|Codex session ended with error" log/symphony.log*

Investigation Flow

• Locate the ticket slice:
• Search by issue_identifier=<KEY>.
• If noise is high, add issue_id=<UUID>.
• Establish timeline:
• Identify first Codex session started ... session_id=....
• Follow with Codex session completed, ended with error, or worker exit

lines.

• Classify the problem:
• Stall loop: Issue stalled ... restarting with backoff.
• App-server startup: Codex session failed ....
• Turn execution failure: turn_failed, turn_cancelled, turn_timeout, or

ended with error.

• Worker crash: Agent task exited ... reason=....
• Validate scope:
• Check whether failures are isolated to one issue/session or repeating across

multiple tickets.

• Capture evidence:
• Save key log lines with timestamps, issue_identifier, issue_id, and

session_id.

• Record probable root cause and the exact failing stage.

Reading Codex Session Logs

In Symphony, Codex session diagnostics are emitted into log/symphony.log and

keyed by session_id. Read them as a lifecycle:

• Codex session started ... session_id=...
• Session stream/lifecycle events for the same session_id
• Terminal event:
• Codex session completed ..., or
• Codex session ended with error ..., or
• Issue stalled ... restarting with backoff

For one specific session investigation, keep the trace narrow:

• Capture one session_id for the ticket.
• Build a timestamped slice for only that session:
• rg -n "session_id=<thread>-<turn>" log/symphony.log*
• Mark the exact failing stage:
• Startup failure before stream events (Codex session failed ...).
• Turn/runtime failure after stream events (turn_* / ended with error).
• Stall recovery (Issue stalled ... restarting with backoff).
• Pair findings with issue_identifier and issue_id from nearby lines to

confirm you are not mixing concurrent retries.

Always pair session findings with issue_identifier/issue_id to avoid mixing

concurrent runs.

Notes

• Prefer rg over grep for speed on large logs.
• Check rotated logs (log/symphony.log*) before concluding data is missing.
• If required context fields are missing in new log statements, align with

elixir/docs/logging.md conventions.