Crash Analysis

Crash Analysis

Systematically analyze a dd-trace-php crash report to identify the root cause.

> Schema note: The event file you receive is the backend-enriched form,

> not the one libdatadog emits.

Input

The user provides a crash event JSON file (path via $ARGUMENTS, or pasted

inline). If no path is given, ask for it. Save pasted JSON to a temporary file.

Phase 1 — Triage (extract all key facts before any analysis)

Extract and print a summary table with these fields. Use jq against the event

file. Do all extractions in parallel where possible.

| Field | Command |

|-------|---------|

| Library version | jq -r '.tracer_version // (.library_version \| "\(.major).\(.minor).\(.patch)")' $EVENT |

| Signal | jq -c .sig_info $EVENT |

| Error summary | jq -r '"\(.error.type // "") \(.error.message // "")"' $EVENT |

| Crash diagnosis | jq -c .crash_diagnosis $EVENT |

| Native stacktrace | jq -c '.error.stack.frames' $EVENT |

| PHP stacktrace | jq -c .experimental.runtime_stack.frames $EVENT |

| Mapped files | jq -c '.files["/proc/self/maps"]' $EVENT |

| Registers | jq -c '.ucontext // .experimental.ucontext' $EVENT \| .claude/scripts/parse_ucontext.py |

| PHP version | jq -r '.language_version // .runtime_version // (.metadata.tags[] \| select(startswith("runtime_version:")) \| split(":")[1])' $EVENT |

| OS / arch | jq -r '(.os_info.os_type // .host.os) + " " + (.os_info.architecture // .host.arch // "unknown") + " (kernel " + (.host.version // "?") + ")"' $EVENT |

> Note: parse_ucontext.py only supports amd64. Check .ucontext.arch

> first; on aarch64, skip the script and read register values directly from

> .ucontext.raw.

crash_diagnosis (schema 1.8+)

crash_diagnosis is computed server-side by the Datadog errors-worker

(DataDog/dd-source,

domains/evp-workers/apps/errors-worker/src/crashtracking/), not by

libdatadog. It consumes sig_info, ucontext.registers, and /proc/self/maps

from the event; if any is absent, the field is omitted. Use it to confirm (not

skip!) manual triage steps:

| Field | Meaning |

|-------|---------|

| category | Crash category — see enum below |

| summary | One-line human-readable description |

| details | Extended description with signal/address details and analysis rationale |

| crashLocation | Optional. The memory mapping containing the instruction pointer at crash time |

| crashLocation.path | Binary where the crashing instruction lives |

| crashLocation.offsetInMapping | Offset within that binary's mapped region (hex) |

| crashLocation.permissions | Mapping permissions (r-xp = executable code) |

| faultAddressMapped | Optional. true = fault address is in a mapped region; false = not mapped (wild pointer); absent if si_addr unavailable |

| faultAddressMapping | If faultAddressMapped is true, the mapping containing the fault address |

| nullRegisters | Registers whose value was < 0x1000 (null page threshold) at crash time — these are the likely null pointer sources for a NullPointerDereference |

| stackPointerValid | Optional. false = SP is outside the [stack] mapping; stack is corrupt; makes native stacktrace unreliable |

DiagnosisCategory enum (complete)

| Value | Signal | Condition |

|-------|--------|-----------|

| NullPointerDereference | SIGSEGV/SEGV_MAPERR | fault addr < 0x1000 (null page) |

| StackOverflow | SIGSEGV/SEGV_MAPERR | fault addr within 8 KB of stack guard page |

| UseAfterFree | SIGSEGV/SEGV_MAPERR | fault addr within 1 MB past heap end |

| WildPointer | SIGSEGV/SEGV_MAPERR | unmapped address, no recognizable pattern |

| WriteToReadOnly | SIGSEGV/SEGV_ACCERR | faulting mapping is non-writable |

| ExecuteNonExecutable | SIGSEGV/SEGV_ACCERR or SIGILL | fault addr == IP and mapping is non-executable |

| MisalignedAccess | SIGBUS/BUS_ADRALN | misaligned memory access (BUS_ADRALN only — BUS_ADRERR, e.g. file-mapped access beyond EOF, maps to Unknown) |

| IllegalInstruction | SIGILL | invalid opcode in executable region |

| IntentionalAbort | SIGABRT | assert(), panic!(), or allocator corruption |

| Unknown | any | no pattern matched |

SSI architecture

When the Datadog SSI package is installed, the process loads four binaries instead of one monolithic ddtrace.so:

| Binary | Typical text size | Role |

|--------|------------------|------|

| dd_library_loader.so | ~28 KiB | Zend extension (loaded via zend_extension=); bootstraps everything else |

| libdatadog_php.so | ~10 MiB | Shared library with sidecar, crashtracker, and Rust components; loaded by the loader with RTLD_GLOBAL |

| ddtrace.so (SSI standalone) | ~750 KiB | PHP extension with most tracer logic; much smaller than monolithic ddtrace.so |

| ddappsec.so | ~630 KiB | AppSec extension; same binary for SSI and non-SSI |

(Sizes vary across versions, but should be of those orders of magnitude)

Loading order (the loader is a Zend extension and fires before any extension= module):

• dd_library_loader.so MINIT fires
• Loader calls dlopen(libdatadog_php.so, RTLD_NOW|RTLD_GLOBAL)
• Loader calls zend_register_internal_module for the SSI ddtrace.so
• PHP processes extension= directives; any extension=ddtrace.so pointing elsewhere is rejected as duplicate

If a non-SSI ddtrace.so (e.g. a vendor-bundled extension) is also listed in extension=, PHP rejects it as a duplicate after the loader already registered the SSI one — and vice versa (if the non-SSI extension loads first, the SSI loader detects the name already in module_registry and unregisters its own injected copy instead; see loader/dd_library_loader.c).

Tag reference

The following tags may be present alongside the event. Use them to attribute the

crash and understand context:

| Tag | Example | Source | Description |

|-----|---------|--------|-------------|

| profiler_inactive | 0 | counters.rs | 1 = profiler was idle at crash time; 0 = it was not. |

| profiler_collecting_sample | 0 | counters.rs | Nonzero = profiler was collecting a sample at crash time. |

| profiler_unwinding | 0 | counters.rs | Nonzero = profiler was unwinding the stack at crash time. |

| profiler_serializing | 0 | counters.rs | Nonzero = profiler was serializing data at crash time. |

| si_signo | 11 | sig_info.rs | Raw signal number (11 = SIGSEGV). |

| si_signo_human_readable | SIGSEGV | sig_info.rs | Signal name (SIGSEGV, SIGBUS, SIGILL, SIGFPE, …). Always uppercase. |

| si_code | 1 | sig_info.rs | Raw signal code; meaning is signal-dependent. |

| si_code_human_readable | SEGV_MAPERR | sig_info.rs | Signal code name (SEGV_MAPERR, SEGV_ACCERR, BUS_ADRALN, ILL_ILLOPC, …). |

| si_addr | 0x00007ff894af86c8 | sig_info.rs | Fault address from siginfo_t.si_addr. |

| is_crash | true | errors_intake.rs / sidecar.c | Always true for crash reports. |

| incomplete | false | errors_intake.rs | true = stack trace is truncated / could not fully unwind. |

| language | php | sidecar.c | Language identifier pushed as language:php. |

| runtime | php | sidecar.c | Runtime identifier pushed as runtime:php. |

| data_schema_version | 1.8 | errors_intake.rs | JSON schema version; current is 1.8. |

| uuid | 2f530826-… | errors_intake.rs | RFC 4122 UUID shared between crash ping and crash report. |

| version | 1.16.0 | sidecar.c | Service version from DD_VERSION or the active APM span. |

| team | telemetry-and-analytics | Datadog backend | Internal routing tag injected by the intake pipeline. Not from PHP code. |

| instrumented_service | web.request | Datadog Agent/backend | Resource/span type at crash time. Not from PHP code. |

| datacenter | us1.prod.dog | Datadog backend | Intake datacenter/region tag. Not from PHP code. |

| datadog.submission_auth | api_key | Datadog intake | Auth method used for submission. |

| datadog.api_key_uuid | 7cacaf92-… | Datadog intake | UUID of the Datadog API key used. |

Check whether any profiler counter (profiler_collecting_sample,

profiler_unwinding, profiler_serializing) is nonzero — this attributes the

crash to profiler activity.

From the mapped files, determine:

• Products loaded: look for ddtrace.so, ddappsec.so,

datadog-profiling.so

• SSI mode: check for libdatadog_php.so and dd_library_loader.so — if

present, the process is running the SSI (Single-Step Instrumentation)

package. See SSI architecture below.

• OS/arch: prefer os_info.architecture (schema 1.8+) over host.arch

(which may be empty), but fall back to reading the mapped ld-linux file name

• libc: GNU (ld-linux-x86-64.so) or musl (ld-musl-x86-64.so)

Finally, print the triage summary before continuing.

Phase 2 — Stacktrace correlation

Checkout the matching version tag in a worktree (tags are like 1.16.0).

For PHP source, use the php-src repository next to this checkout; PHP tags

are like PHP-8.1.33.

PHP runtime frames (experimental.runtime_stack.frames, format: `"Datadog

Runtime Callback 1.0"`) contain:

• file / function / line — source location
• type_name — class name when the frame is a method call (e.g.

"Couchbase\\Collection")

> Note: Ondřej Surý packages for Debian may be slightly modified relative to

> upstream PHP. If discrepancies appear, use apt-get source inside an

> appropriate Docker container to obtain the exact source.

Map each native stacktrace frame to source code:

• Use .claude/scripts/find_map_region.py <address> <event.json> to identify which

binary each instruction pointer belongs to. In schema 1.6+ events, frames

include a module_base_address field that already identifies the binary —

cross-check against the mapped files to confirm the binary name.

• For frames in Datadog binaries, correlate to source using the worktree.
• For frames in PHP, correlate to php-src at the matching tag.

To determine whether a frame in a PHP process belongs to a Datadog product:

first check the function name (it may clearly identify a DD module); if

inconclusive, check whether the instruction pointer falls within the mapped

executable segments of ddtrace.so, ddappsec.so, or

datadog-profiling.so.

If there are no Datadog product stack frames, still check whether a Datadog

product could be the root cause — for example, by having modified the behavior

of PHP functions that are clearly on the crashing call path. If this seems

unlikely, ask the user before continuing with deep analysis.

If frames land in unknown binaries, note them but focus on Datadog frames first.

If you can identify the root cause at this point, stop and report. Only

continue to Phase 3/4 if the analysis is ambiguous or low-confidence.

Note: the authoritative native stacktrace for the crashing thread is

.error.stack.frames (format: "Datadog Crashtracker 1.0", always

populated when a stack could be captured). The crashing thread name is in

.error.thread_name.

error.threads is a per-thread snapshot array present in schema 1.8+. Each

element carries a crashed boolean flag, name, state, and `stack.{frames,

incomplete}. In practice, crashed is often false` on every thread and

stack.frames is null with incomplete: true — the per-thread stacks are

frequently unavailable. Use them as supplementary context only; do not rely on

them as the primary frame source.

Phase 3 — Binary verification (if needed)

If the stacktrace correlation is ambiguous or the crash is in Datadog code:

Datadog binaries

• Download the release binaries:
• SSI (libdatadog_php.so in maps): fetches from ECR public, no credentials needed:

     .claude/scripts/dd_php_release_url --ssi '<version>' '<arch>'

• Monolithic: fetches from GitHub releases:

     .claude/scripts/dd_php_release_url '<version>' '<php_minor>' '<arch>' '<gnu|musl>'

Both print a temp directory with the extracted package. Use the version

exactly as it appears in tracer_version (or reconstructed from

library_version).

• Verify the binary matches the crash by comparing:
• Size of first mapped region (from /proc/self/maps) vs. p_memsz of the

first PT_LOAD segment (readelf -l <binary>). The mapped region spans

from the page floor of p_vaddr to the page ceiling of

p_vaddr + p_memsz, so the mapped size is the segment's virtual address

range rounded outward to page boundaries; mapped size ≥ p_memsz,

they need not be equal.

• Note the base address: map_start - (p_vaddr & ~(PAGE-1)), where PAGE

is typically 0x1000 on x86-64 (aarch64 can be 0x1000, 0x4000, or

0x10000). In schema 1.6+ events, module_base_address in each frame is

the load bias and can be used directly instead of computing it.

• Disassemble around the crashing instruction pointer.

SSI binaries from ECR are stripped but retain exported public symbols —

pass the binary directly to GDB:

   gdb -batch -ex 'add-symbol-file <binary> -o <base_addr>' \
       -ex 'disassemble <ip_addr>-32,<ip_addr>+32'

For full symbols (line numbers, inlined functions), try the S3 tarball —

it includes co-located .debug files but requires a manual CI job to have

been triggered. Check with:

   curl -sI https://dd-trace-php-builds.s3.amazonaws.com/<version>/dd-library-php-ssi-<version>-<arch>-linux.tar.gz

If available, .debug files sit next to each .so in the tarball. Pass

to GDB with add-symbol-file <binary>.debug -o <base_addr>.

For monolithic builds (non-SSI), the binary is unstripped; pass it directly:

   gdb -batch -ex 'add-symbol-file <binary> -o <base_addr>' \
       -ex 'disassemble <ip_addr>-32,<ip_addr>+32'

On macOS, run gdb/readelf/objdump inside a Docker container with

--platform=linux/amd64. Build a Dockerfile with the tools you need

rather than installing packages each time.

PHP binaries (when PHP frames need deeper analysis)

If source code alone is insufficient to understand PHP frames:

• Identify and download the PHP binaries. Common sources:
• Ondřej Surý packages (Debian/Ubuntu)
• PHP Docker images (Alpine or Debian, binaries installed under /usr/local)

• Extract the binaries into the same temp directory used for Datadog binaries.

• Verify each PHP binary matches the crash the same way as Datadog binaries:

compare the size of the first mapped region against p_memsz of the first

PT_LOAD segment (the mapped region spans the segment's virtual address range

rounded outward to page boundaries, so mapped size ≥ p_memsz), note the

base address.

• Disassemble PHP frames using gdb with add-symbol-file at the computed

base address.

Phase 4 — Root cause

Based on the disassembly, register state, and source code:

• Identify what memory access or operation triggered the fault.
• Trace backward to find the bug (NULL deref, use-after-free, bad offset, etc.).
• Check if the fault address (si_addr) and the register state are consistent

with the hypothesis.

Output

Present findings as:

### Crash Summary
- **Signal**: ...
- **Version**: ... | PHP ... | arch ... | libc ...
- **Products**: ddtrace / ddappsec / profiling
- **Profiler active**: yes/no

### Root Cause
<explanation with source file:line references>

### Evidence
<key register values, disassembly snippet, source correlation>

### Confidence
<high / medium / low — and what would raise it>

Rules

• If no Datadog frames are present, check for indirect causation (DD hooks on

the crashing path) before deciding the crash is unrelated. If it still seems

unrelated, ask the user whether to continue before doing deep analysis.

• Use git worktree (not git checkout) to inspect version-specific source.
• On macOS, use Docker with --platform=linux/amd64 for ELF tools.
• Parallelize extractions and lookups wherever possible.