Falsify The Problem Agent Skill

falsify-the-problem

> Falsify the problem before solving it.

Answer Are we solving the right problem? Challenge the current formulation,

seek disconfirming evidence, and hand the result back to the host agent.

Do not design or implement the solution inside this skill.

Use the same protocol across domains; examples are not domain-specific workflows.

1. Decide whether to enter

• Explicit invocation of falsify-the-problem: always enter. Strong existing

evidence can justify a brief KEEP without alternatives or a new test.

• Automatic activation: require BOTH meaningful formulation uncertainty AND

non-trivial intervention cost or risk. Neither alone is enough.

• Useful signals include solution-shaped requests, a hypothesis treated as fact,

repeated failed attempts, and costly or hard-to-reverse proposed changes.

Signals are reasons to check the gate, not unconditional triggers.

• Usually do not auto-activate for a verified diagnosis, obvious syntax or

compilation error, typo, translation, formatting, mechanical transformation,

deterministic local edit, or very cheap reversible intervention.

• Mere logical possibility of another explanation is insufficient.

BYPASS is an activation decision, never an evidence verdict. When bypassing,

leave the original task with the host; do not manufacture an adversarial report.

If explicitly invoked on a supported diagnosis, use KEEP, not BYPASS.

2. Separate the claims and preserve their sources

Distinguish these five objects, even when the user combines them in one sentence:

| Object | Meaning | Example |

| --- | --- | --- |

| Observation | Something observed, measured, or reported | User reports P99 rose from 300 ms to 2.8 s |

| Interpretation | Meaning assigned to an observation | The database is slow |

| Causal Hypothesis | A specific mechanism proposed to explain it | Lock contention delays requests |

| Problem Formulation | What kind of problem is being solved | Database service time dominates the slow tail |

| Solution | A proposed intervention | Add Redis |

Observation != Interpretation != Causal Hypothesis != Problem Formulation != Solution.

For each consequential observation, retain the source and relevant scope:

• tool-observed: name the file, command, trace, or other inspected artifact.
• user-reported: retain attribution, including measurements supplied by a user.
• document-reported: identify the document and its claim.
• experimentally observed: identify the experiment and conditions.
• measurement-derived: identify inputs, transformation, and sampling limits.

These labels can combine. Reading a report with a tool verifies what the report

says, not that its underlying claim is true. A local simulated fixture is evidence

about that fixture, not independent verification of a real incident.

Never upgrade user reports into independently verified facts.

Record uncertainty about coverage, freshness, comparability, or missing data

where it could change the conclusion. Do not fabricate observations or execution.

State the Current Problem Formulation explicitly. If inferred from a proposed

solution, mark it as inferred; do not attribute it to the user as a stated belief.

Keep this original formulation as the reference for the overall verdict through

this pass. A replacement is REFORMULATE, not a silently relabeled KEEP.

Preserve claim strength: a possible contributor, a primary explanation, and an

exclusive explanation are different claims. Do not infer exclusivity from a

proposed solution or refute a stronger claim than the request requires.

3. Build materially different live formulations

Ask what would change the kind of problem being investigated. Competing causes

within one formulation are not automatically competing problem formulations.

Two formulations should materially differ in at least one consequential dimension:

failing variable, mechanism class, abstraction level, system boundary, expected

evidence, investigation path, or intervention target.

If two formulations predict nearly the same evidence and lead to nearly the same

investigation path, they are probably superficial variants. Merge them.

For example, weak embeddings, small embeddings, and old embeddings usually belong

to one representation-quality formulation. Retrieval construction, ranking,

generation grounding, evaluation validity, and query distribution can change the

framing when they predict meaningfully different evidence.

Keep only plausible, decision-relevant alternatives. Do not mechanically fill a

quota. Let evidence and the pending decision determine the live set; more

candidates are not evidence of better reasoning. Presentation limits in section 9

do not justify dropping a materially different, decision-relevant candidate.

Candidates need not be mutually exclusive. If several mechanisms coexist, test

which framing explains the material failure at the relevant scope; do not force a

winner from evidence that supports a mixed or narrower formulation.

4. Find load-bearing assumptions and distinct predictions

Prioritize assumptions whose failure would materially weaken or collapse a live

formulation. Avoid exhaustive lists or generic questions about everything.

Include measurement validity only when it bears on the decision.

For each important live formulation, connect:

Formulation -> load-bearing assumption -> observable prediction -> contrary evidence

A prediction must distinguish live formulations, not restate their names.

Example: a service-time framing predicts elevated service spans in slow requests;

a capacity framing predicts rising pre-service wait with concurrency; a measurement

framing predicts disagreement between raw samples and reported aggregation.

State the population, comparison, and meaningful difference when available.

Do not invent numerical thresholds without a basis; explain qualitative criteria.

Before choosing the test, make one lightweight Shared-Frame / Coverage Check:

do the live formulations all rely on one consequential, untested premise about

the relevant boundary, measurement path, causal layer, population, time scope,

or optimization target? Select what matters here; do not audit a category checklist.

If that premise failing would undermine the whole set, and a plausible alternative

would change the decision and predict different evidence, add at most **ONE

Outside-Frame Challenger**. Name the shared premise and its contrasting prediction

briefly when decision-relevant. Generic "measurement could be wrong" is insufficient;

ground the challenger in this task. If none is warranted, proceed without one.

Do not chain coverage audits or expand the set merely to satisfy a quota.

5. Select ONE Primary Discriminating Test for this round

When beginning an investigation round under unresolved uncertainty, choose

exactly ONE Primary Discriminating Test. This is one per round, not one forever.

Choose the cheapest currently available test with substantial discriminatory

information relative to cost, risk, and execution time.

Ask: would plausible different outcomes materially change the relative support

of the live formulations? If not, choose a better test.

An outcome predicted by several live formulations may localize the failure without

distinguishing them. Prefer a matched comparison that separates the relevant

factor; otherwise retain those formulations rather than assigning the outcome

to one by default.

Specify:

• Test: one bounded comparison or observation, its source, and its scope.
• Why this first: which live formulations it separates and why its value

exceeds the cost of the next plausible option.

• Possible outcomes: what would strengthen, weaken, or kill which candidates.

Include inconclusive or mixed outcomes where plausible.

• Remaining uncertainty: what this test cannot establish.

One test can require several retrieval or computation steps if they serve one

predeclared comparison and outcome map. It cannot hide independent investigations

inside a bundle such as "check logs, metrics, configuration, and tests."

Include an unaccounted or outside-boundary outcome when the observation method

could omit relevant stages; an artifact's label does not define its boundaries.

Handle such a result through the evidence update below, not a forced winner.

Do not call it a "Decisive Test" or imply that every result must settle the issue.

6. Gather evidence instead of deflecting it

STOP solutioning != STOP investigating.

Inspect evidence directly when it is available, permitted, low-risk, and reasonably

cheap. Use the host's tools as appropriate: local artifacts, code, history, logs,

traces, metrics, configurations, documentation, existing tests, safe diagnostics,

low-risk experiments, or benchmarks. No specific tool or runtime is required.

Choose the investigation for its discrimination, not because a tool is available.

Read-only exploration needed to locate the selected evidence is allowed. Bound it

to the comparison; do not turn discovery into a generic diagnostic checklist.

Record what was actually inspected or run and the result, including failures.

If the selected source is inaccessible, prefer an available lower-cost proxy only

if it can discriminate; record the limitation. If no useful proxy exists, state

the one pending Primary Discriminating Test and request only the minimum missing

discriminating evidence or access. Do not ask users to do work the host can safely

perform. Do not claim the pending test ran or loop through identical requests.

Investigation does not authorize implementation. Before formulation validation,

do not implement the requested solution, migrate architecture, introduce

infrastructure, make expensive production changes, take irreversible action, or

optimize around an unvalidated premise. Calling an intervention a "test" does not

make it low-risk. Respect the host's existing permissions and user task scope.

7. Update from evidence, then decide whether to continue

Evidence beats rhetoric and the agent's previous conclusion. Accept contradictory

evidence; do not reinterpret it merely to preserve an earlier position.

After the selected test:

• Attribute the result and compare it with the predicted outcomes.
• If a consequential result fits no live formulation, mark it **OUTSIDE CURRENT

FRAME**: the set may be incomplete. This is an evidence diagnostic, not a sixth

verdict. Preserve the unexplained residual; do not award it to the closest

candidate. Repeat the lightweight coverage check once for this new evidence,

adding at most one grounded challenger if warranted. Otherwise leave it unresolved.

• Update affected candidates using the labels below with a short evidence reason,

and re-check load-bearing assumptions invalidated by the test.

• Merge redundant candidates; retain uncertainty and scope limits.
• Apply the STOP conditions. If a further discriminating observation is worth its

cost, start the next round with the updated live set and one new Primary Test.

| Candidate update | Meaning |

| --- | --- |

| STRENGTHEN | New evidence raises support, without proving the candidate |

| WEAKEN | New evidence reduces support; the candidate remains live |

| KILL | Reliable evidence contradicts a necessary assumption at the stated scope |

| UNRESOLVED | Evidence does not discriminate enough or is unavailable |

| MERGE | Combine with a named candidate because the distinction is redundant |

Missing evidence is not contrary evidence. An absent signal kills a candidate only

when coverage and detection were sufficient for that signal to be expected.

If new evidence undermines a prior kill, explicitly reopen the candidate and give

the reason; labels are revisable conclusions, not permanent states.

Candidate updates and the overall verdict have different meanings and scopes.

8. STOP, give an overall verdict, and hand back control

End the adversarial pass when any of these holds:

• The current formulation has sufficient evidence for the pending decision.
• No meaningful materially different alternative remains, or remaining

alternatives are too weak to justify delaying the decision.

• Further investigation has poor information gain relative to cost, risk, or time.
• Another formulation is sufficiently supported to replace the original.
• The original formulation is dead and no useful cheap investigation remains.
• Essential discriminating evidence is unavailable and no useful proxy exists;

hand off the minimal evidence request rather than stall indefinitely.

Sufficiency is proportional to the pending intervention and evidence quality.

Running out of tests, access, time, or alternatives is not itself evidence for KEEP.

Do not become contrarian for its own sake. KEEP is a normal successful outcome.

Use exactly one overall verdict about the original Current Problem Formulation:

| Overall verdict | Meaning | Solutioning / handoff |

| --- | --- | --- |

| KEEP | Current framing remains best-supported and sufficiently validated for the pending decision | ALLOWED for that framing; release to the downstream solver |

| WEAKEN | Current framing remains live but has materially lost support; no replacement is sufficiently established | NOT YET; continue useful investigation or hand off the specific uncertainty |

| KILL | Current framing is no longer viable, with no sufficiently supported replacement | BLOCKED for the killed premise; investigate another framing if worthwhile |

| REFORMULATE | A materially different framing is sufficiently better supported to replace the original | NOT YET; return the Reformulated Problem to the downstream workflow; the old solution has no release |

| INSUFFICIENT EVIDENCE | Available evidence cannot discriminate enough to support a stronger verdict | NOT YET; investigate if worthwhile, otherwise state the minimum missing evidence |

For REFORMULATE, always output Reformulated Problem: with the replacement and

its supporting evidence. KILL does not assert a replacement; REFORMULATE does.

If several descriptions fit, choose the most informative supported verdict:

an established replacement takes REFORMULATE over KILL or WEAKEN; absent one,

contradicted necessary assumptions justify KILL, reduced support justifies WEAKEN,

and non-discrimination justifies INSUFFICIENT EVIDENCE.

Skill STOP != Host Agent STOP. KEEP ends this skill's responsibility. When the

original request already authorizes implementation, the host can continue in the

same turn without asking again. Validation is not new permission for extra work.

KEEP validates the problem framing, not the effectiveness of the proposed solution.

Mark any subsequent diagnosis or implementation as host/downstream continuation

so a completed framing pass does not silently turn into a root-cause workflow.

After REFORMULATE, the host may run a brief validation pass with the replacement

as the new Current Problem Formulation, reusing the evidence already gathered.

If it warrants KEEP, release it without redundant testing; then the host can solve

within the user's authorized scope. Do not implement the original solution merely

because an alternative explanation was found. The skill itself never designs it.

9. Default to compact output; disclose detail as needed

Complete the relevant checks, but show only decision-relevant evidence and

rationale, not an exhaustive reasoning transcript or hidden chain of thought.

Choose the shortest path that preserves the decision, discrimination, and scope:

| Path | When | Visible output |

| --- | --- | --- |

| Fast | BYPASS, strong KEEP, verified deterministic diagnosis, or simple low uncertainty | Usually 2-5 sentences: source-backed observation, framing, verdict/release if entered, then host continuation. No invented alternative or new test. BYPASS needs no adversarial report. |

| Normal (default) | Ordinary unresolved coding, debugging, architecture, research, or product framing | Usually 120-220 English words: observation/provenance, current framing, live alternatives, one Primary Test with contrasting outcomes, result or pending evidence, verdict and handoff. Fold each key assumption into its prediction. |

| Deep | Stakes or complexity require more detail, including when 5+ live formulations materially affect the decision | Usually 250-450 English words: expand assumptions, predictions, candidate updates, and scope limits only where needed. Domain alone does not require this path. |

These are soft presentation guides, not word quotas or correctness gates; use

equivalent brevity in other languages. Normal output usually shows 2-4 live

formulations including the current one, at most 3-4 by default. Do not conceal a

consequential challenger to fit that range: merge genuine redundancies, or expand

the output when needed. A supported single formulation needs no padding.

Keep provenance compact (for example, "report.txt, supplied summary"), never omit

it or upgrade it. State whether the one primary comparison ran or is pending;

preliminary localization is not a discriminating test when all candidates predict

it. Give one clear verdict and release status, plus Reformulated Problem: when

required. Headings are optional; do not mechanically reproduce all workflow steps.

On later turns, use delta output: new evidence/source, what changed, affected

candidate updates and scope, a new Primary Test only if useful, then verdict/handoff.

Do not repeat unchanged observations, formulations, assumptions, or predictions.

Reference prior candidates briefly so the update remains intelligible.

Do not repeat the user prompt, narrate internal exploration, stage a debate,

explain the methodology unless asked, or list discarded/merged candidates unless

their change matters now. Stop once the framing decision is supported; mark any

authorized downstream work separately and let the host continue.