Debug Agent Hw Test

Debug AgentHwTest Failures

End-to-end workflow for building, running, and debugging FBOSS AgentHwTest failures on switches. Drives iterative test-debug-fix cycles.

Prerequisites: Device Access

This skill requires access to a lab switch to deploy and run tests.

Read the device access reference (see Reference Routing below) to learn

the available commands for your environment (upload, download, run

commands on switch).

Invocation Modes

The skill supports two modes:

| Mode | Input | Behavior |

|------|-------|----------|

| Mode A — Single test | A test name (e.g. AgentVoqSwitchTest.addRemoveNeighbor) | Run one test, debug if it fails (today's behavior) |

| Mode B — Batch test list | A file path (e.g. /tmp/test_list.txt) | Read test names from file, run each sequentially |

Mode selection: If the user provides a single test name, run Mode A. If the user provides a file path, read the file and run Mode B.

Test List File Format (Mode B)

One test name per line. Lines starting with # are comments, blank lines are ignored.

# /tmp/test_list.txt
AgentVoqSwitchTest.addRemoveNeighbor
AgentAclCounterTest.AclStatTest

Inputs

Gather these before starting:

| Input | Example | Required? |

|-------|---------|-----------|

| Test name | AgentVoqSwitchTest.addRemoveNeighbor | Yes (Mode A) |

| Test list file | /tmp/test_list.txt | Yes (Mode B) |

| Mode | mono or multi | Yes |

| SDK version | 12.2 DNX | Yes |

| Switch name | fboss325966941.ash6 | Yes |

| Config file path | /root/<user>/<platform>.agent.materialized_JSON (mono) or /root/<user>/<platform>_multi.agent.materialized_JSON (multi) | Yes — see build-and-load.md for how to obtain |

| HW agent binary path | /root/<user>/fboss_hw_agent-brcm-12.2.0.0_dnx_odp | No — derived from SDK version |

| Switch IDs | 0 2 | No — extract from config's switchIdToSwitchInfo (see below) |

Deriving Switch IDs from Config

Switch IDs are platform-specific. Extract them from the materialized config file:

python3 -c "
import json, sys
with open(sys.argv[1]) as f:
    cfg = json.load(f)
for sid, info in cfg['sw']['switchSettings']['switchIdToSwitchInfo'].items():
    print(f'switchId={sid} switchIndex={info[\"switchIndex\"]}')
" /root/<user>/<platform>_multi.agent.materialized_JSON

Known platform switch IDs:

| Platform | Switch IDs | Switch Indices |

|----------|-----------|----------------|

| meru800bfa | 0, 2 | 0, 1 |

| janga800bic | 0, 4 | 0, 1 |

The --switch_id_for_testing flag takes the switch ID (not the switch index).

Build-Before-Run Rule

MANDATORY: Always build binaries on every skill invocation before running any test. Never skip the build step, even if binaries already exist on the switch from a previous session or invocation. Stale binaries produce incorrect results because the source code may have changed.

On every skill invocation:

• Build the required binaries (mono or multi-switch)
• Strip and copy them to the switch (strip_and_copy.sh uses md5 dedup — unchanged binaries skip the network copy automatically)
• Copy the config file and test scripts to the switch
• Then proceed to run the test

The md5 dedup in strip_and_copy.sh makes this safe and fast — if the binary hasn't changed, the copy is skipped. But the build must always run to ensure the binary reflects the current source.

Debug Loop

Follow this iterative cycle for each failing test:

• Build the test binary (mono or multi-switch) — never skip this step
• Load the binary onto the switch
• Copy firmware if testing on a Broadcom DNX switch (see build-and-load.md)
• Run the test (cold or warm boot)
• Analyze output — logs, crashes, vendor diag shell
• Fix or adjust — code change, feature flag, XLOG insertion
• Re-run to verify

Broadcom DNX firmware prerequisite: All tests on Broadcom DNX switches (Jericho3, Ramon3, etc.) require the firmware db/ directory at /tmp/db/ on the switch. See build-and-load.md for details. Without this, hw_agent processes abort with FW: ... is not accessible error:-1. This does not apply to Broadcom XGS or Leaba/Cisco platforms.

Discipline: Work on one test at a time. Try up to 5 iterations before categorizing and moving to the next test.

Batch Execution Loop (Mode B)

When running in batch mode, follow this loop for each test in the list:

for each test in list:
  1. RUN the test (cold boot + warm boot, across all switch_ids)
  2. if PASS on all switch_ids:
       - Record as PASS_NO_CHANGE
       - Proceed immediately to next test (no debug needed)
  3. if FAIL on any switch_id:
       - Enter debug loop (up to 5 iterations)
       - Use debug references: analyze-logs, crash-debug, vendor-diag-shell, etc.
       - Categorize result (PASS_FEATURE_FLAG, PASS_FBOSS_FIX, FAIL_FBOSS, etc.)
       - After 5 iterations or categorization, MOVE ON to next test
  4. Update the running results table after each test

Key principle: Passing tests get zero debug time. Failing tests get bounded debug time (max 5 iterations). The batch always makes forward progress — never get stuck on one test.

Session Tracking

Maintain a results table at /tmp/agent_hw_test_results.md to track progress across the debug session. Update after each test attempt.

Mode A — Single Test Results Table

| Test Name                              | Switch ID | Cold Boot | Warm Boot |
|----------------------------------------|-----------|-----------|-----------|
| AgentVoqSwitchTest.addRemoveNeighbor   | 0         | PASS      | PASS      |
| AgentVoqSwitchTest.addRemoveNeighbor   | 4         | PASS      | PASS      |

Mode B — Batch Results Table

| # | Test Name                              | Switch ID | Cold Boot | Warm Boot          | Category         | Iterations | Notes                        |
|---|----------------------------------------|-----------|-----------|--------------------|------------------|------------|------------------------------|
| 1 | AgentVoqSwitchTest.addRemoveNeighbor   | 0         | PASS      | PASS               | PASS_NO_CHANGE   | 1          |                              |
| 1 | AgentVoqSwitchTest.addRemoveNeighbor   | 4         | PASS      | PASS               | PASS_NO_CHANGE   | 1          |                              |
| 2 | AgentAclCounterTest.AclStatTest        | 0         | FAIL      | SKIPPED (cold fail) | FAIL_VENDOR      | 4          | SAI_STATUS_NOT_SUPPORTED     |
| 2 | AgentAclCounterTest.AclStatTest        | 4         | FAIL      | SKIPPED (cold fail) | FAIL_VENDOR      | 4          |                              |

At the end of a batch run, also print a summary:

=== Summary ===
Total tests: 2
  PASS_NO_CHANGE:    1
  PASS_FEATURE_FLAG: 0
  PASS_FBOSS_FIX:    0
  PASS_VENDOR_FIX:   0
  FAIL_FBOSS:        0
  FAIL_VENDOR:       1
  TODO_ASIC_CONFIG:  0

Pass rate: 1/2 (50%)

Test Categorization Taxonomy

Categorize every test into one of these 7 outcomes:

| Category | Code | Meaning |

|----------|------|---------|

| PASS without code change | PASS_NO_CHANGE | Test passes as-is on the platform |

| PASS after HwAsic feature flag change | PASS_FEATURE_FLAG | Needed to enable/disable an ASIC feature flag |

| PASS after FBOSS code change | PASS_FBOSS_FIX | Required a fix in FBOSS agent code |

| PASS after Vendor SDK code change | PASS_VENDOR_FIX | Required a fix in vendor SAI/SDK code |

| FAIL: FBOSS needs manual debug/fix | FAIL_FBOSS | Identified as FBOSS issue, needs more work |

| FAIL: Vendor SDK needs manual debug/fix | FAIL_VENDOR | Identified as vendor SDK issue, needs more work |

| FAIL: Test hung or timed out | FAIL_TIMEOUT | Test did not complete within allowed time, possible deadlock |

| TODO: assess ASIC chip config change | TODO_ASIC_CONFIG | May need ASIC config (lane map, MMU, etc.) change |

For FAIL_VENDOR (or PASS_VENDOR_FIX), produce a vendor-escalation package — SAI Replayer log + hw_config + a case report the vendor can reproduce from — using references/vendor-escalation.md. It is vendor-generic (Broadcom CSP / Cisco TAC / NVIDIA support).

Scripts

On-Switch Scripts (uploaded and run on the switch)

These scripts run on the switch. Upload them once per session, then run in the background.

| Script | Purpose | Args |

|--------|---------|------|

| scripts/run_mono_test.sh | Mono cold+warm boot cycle | <binary> <config> <filter> <user> |

| scripts/run_multi_test.sh | Multi-switch cold+warm boot for one switch_id | <hw_agent> <test_binary> <config> <filter> <switch_id> |

| scripts/collect_vendor_escalation.sh | Collect a vendor-escalation package (SAI replayer log + hw_config), with optional packet-send logging | <mono\|multi> <hw_agent\|-> <test_binary> <config> <filter> <switch_id> <out_dir> <pkt_log:0\|1> [get_attr_log:0\|1] [suffix] |

Local Helper Scripts (run on build host only)

| Script | Purpose | Args | Output |

|--------|---------|------|--------|

| scripts/strip_and_copy.sh | Strip binary, print stripped path + md5 | <source_path> <dest_name> | Stripped binary at /tmp/<dest_name> |

> Vendor firmware scripts: See build-environment.md

> for environment-specific SDK path resolution and firmware preparation scripts.

Remote Execution Pattern

Upload scripts to the switch, run tests in the background, and fetch

logs when complete. First resolve the device-access reference via the

Reference Routing table below, then follow

run-tests.md for the full execution pattern.

Reference Routing

For each reference pair below, load the facebook/ version first if it

exists in your checkout. Otherwise load the references/ version. Treat

the selected file as the source of truth for that topic.

This routing is client-agnostic:

• internal checkouts typically provide facebook/ overrides with lab-specific

device access and build instructions

• OSS checkouts fall back to references/ with standard ssh/scp and

open-source build instructions

• client-specific command forms are optional conveniences only; the resolved

reference must always contain a direct runnable path as fallback

| Need | Try first | Fallback |

|------|-----------|----------|

| Device access (upload, download, run on switch) | facebook/device-access.md | references/device-access.md |

| Build commands, SDK paths, config locations | facebook/build-environment.md | references/build-environment.md |

| Build mono/multi binaries, copy to switch | — | references/build-and-load.md |

| Run tests (cold/warm, mono/multi), parse results | — | references/run-tests.md |

| Analyze SAI Replayer logs, read code for root cause | — | references/analyze-logs.md |

| Enable SAI logging, replayer logging, packet tx logs | — | references/enable-logging.md |

| SAI Replayer — capabilities/flags (packet/get/elapsed-time log), capture, build, run | facebook/sai-replayer.md | references/sai-replayer.md |

| Debug crashes — non-stripped binaries, GDB, stack traces | — | references/crash-debug.md |

| Vendor diagnostic shell — counters, routes, neighbors, techsupport dumps | facebook/vendor-diag-shell.md | references/vendor-diag-shell.md |

| Insert XLOGs, re-run, validate hypotheses | — | references/hypothesis-driven-debug.md |

| Full categorization guide and tracking template | — | references/test-categorization.md |

| Build a vendor-escalation package (SAI replayer log + hw_config + case report) for FAIL_VENDOR | — | references/vendor-escalation.md |