Validation Diagnostics

Validation & Diagnostics

Overview

Structured process for comparing planned vs implemented work, generating diagnostic reports, and identifying gaps. Used by validation agents to provide concrete, evidence-based assessments.

Key Concepts

The Trust Problem

Agents will claim work is complete when it isn't. Validation must be:

• Observable — Based on test output, not agent claims
• Deterministic — Same input produces same verdict
• Evidence-based — Every finding references concrete code or test results
• Structured — Consistent format for easy human review

Patterns & Techniques

Planned vs Implemented Comparison

Step 1: Extract planned items

From the spec, issue, or plan, extract a checklist of:

• Acceptance criteria
• Required endpoints/functions
• Expected model fields
• Required test coverage
• UI components specified

Step 2: Verify each item

For each planned item, check:

• Does the code exist? (file search, grep)
• Does it match the specification? (read and compare)
• Is it tested? (find corresponding test)
• Does the test pass? (run and capture output)

Step 3: Generate drift report

## Drift Analysis: [Feature]

| Planned Item | Status | Evidence |
|-------------|--------|----------|
| [spec item] | ✅ Implemented | [file:line] |
| [spec item] | ⚠️ Partial | [what's missing] |
| [spec item] | ❌ Not found | [searched in...] |
| [unplanned] | ⚡ Scope creep | [file:line] |

Diagnostic Report Template

## Diagnostic Report: [Component/Feature]

### Summary
[1-2 sentence verdict]

### Test Results

[Actual test output — copy/paste, not paraphrased]

### Code Quality
| Metric | Result |
|--------|--------|
| Type hints present | ✅ / ❌ |
| Pydantic models used | ✅ / ❌ |
| Config class used (no hardcoded secrets) | ✅ / ❌ |
| Error handling present | ✅ / ❌ |
| Logger used (not print) | ✅ / ❌ |

### Findings
| # | Severity | Finding | Location | Recommendation |
|---|----------|---------|----------|----------------|
| 1 | [High/Med/Low] | [what] | [file:line] | [fix] |

### Coverage Gaps
[Code paths without tests]

### Verdict
✅ PASS | ⚠️ CONDITIONAL PASS | ❌ FAIL
[Explanation with evidence]

HASTE-Specific Validation Checks

| Component | Must Verify |

|-----------|-------------|

| New API endpoint | Auth level, Pydantic validation, error codes, CORS |

| New processor | Config injection, logger usage, error handling |

| New data model | Pydantic BaseModel, field types, validation rules |

| New data layer | Abstract interface compliance, connection handling |

| New UI component | FluentUI usage, no alt frameworks, responsive |

| Geospatial code | CRS preservation, COG compliance, GDAL/rasterio usage |

Feedback Loop

When validation reveals a miss:

• Document the miss in the diagnostic report
• Identify the pattern (was it a convention violation? missing test? spec ambiguity?)
• Propose a skill update or instruction addition to prevent recurrence
• Flag to the Orchestrator for tracking

Decision Framework

| Validation Result | Action |

|-------------------|--------|

| All checks pass, tests green | ✅ Approve |

| Minor issues, tests pass | ⚠️ Conditional — list issues for human review |

| Tests fail | ❌ Block — must fix before proceeding |

| Scope creep detected | ⚠️ Flag — human decides if extra work is acceptable |

| Spec ambiguity found | ⚠️ Flag — needs clarification before validation |

Common Pitfalls

• Accepting "tests passed" without seeing output — Always run and capture
• Validating only happy path — Check error handling and edge cases
• Skipping convention checks — HASTE has specific patterns that must be followed
• Not checking for scope creep — Extra changes can introduce regressions