The report arrived with nine sections, four comparison tables, a risk matrix, and forty-one named fields. Every one of them read N/A.
Not zero. Not blank. N/A — defined in its own glossary as "cannot be assessed because input is missing," with an explicit warning that it must not be conflated with "assessed as neutral." The pipeline ran its structural checks and passed. Every header present. Every table rendered. Every section terminated with a conclusion. It looked like an audit. It contained nothing.
The failure was upstream and clean. A Stage 1 decomposer was supposed to reduce a source document into numbered information points — the atomic, verifiable facts that every downstream conclusion cites. It returned an empty list. Stage 2 then had two legal moves: emit nothing, or emit a typed refusal. It chose refusal, and it was right to. The interesting part is not the refusal. It is that the refusal was formatted identically to a finding.
I audit pipelines for a living, and this is the bug class I check first now: not a wrong answer, but an answer whose provenance has been severed. A conclusion with no information point behind it is not analysis. It is decoration with a schema.
The architecture is standard in crypto due diligence. Stage 1: decompose. Extract facts — a contract address, a TVL figure, an unlock date, an auditor's name. Number them. Stage 2: map nine analytical dimensions onto those numbered facts and require every claim to cite the number it descends from. The contract between stages is provenance, and it is the only thing separating a research pipeline from a text generator.
When Stage 1 returns zero points, that contract is vacuously satisfied. Every claim cites nothing, and nothing contradicts. A schema validator sees a well-formed document. A human skimming it sees a well-formed document. The only entity that could catch the failure is a check that counts information points before rendering — and that check is almost never written, because in every demonstration the input is never empty.
Here is the first mechanical failure. Risk levels in these schemas are string enums: LOW, MEDIUM, HIGH, N/A. Downstream aggregators parse them with a default branch.
def aggregate(report):
score = 0
for dim in report.dimensions:
level = dim.risk_level # "LOW" | "MEDIUM" | "HIGH" | "N/A"
score += WEIGHTS.get(level, 0) # "N/A" -> 0, identical to "LOW"
return scoreWEIGHTS.get("N/A", 0) and WEIGHTS.get("LOW", 0) return the same integer. The absence of evidence and the evidence of absence collapse into one number. This is not a logic error. It is a type error, and no amount of weight-tuning repairs it. The correction is structural: risk must be Optional[Risk], where None is not zero, and any aggregation that touches None returns None rather than a total. A nine-dimension report with nine None values must aggregate to None. It must not aggregate to a reassuring 0.
The second failure is that completeness metrics measure shape.
| Validator check | Empty report | Verdict | |---|---|---| | All sections present | 9 / 9 | pass | | All enum values valid | 41 / 41 | pass | | Tables render | 4 / 4 | pass | | Information points ≥ 1 | 0 | not checked |
A validator that never counts its inputs will pass every empty document it is ever shown, forever, at 100% confidence. The report's own self-assessment — "this output is a validation-failure report, not investment analysis" — was correct, and it was buried below forty-one rows of N/A that looked like findings.
The third failure is the missing provenance hash.
EMPTY = sha256(b"").hexdigest()def emit(source_bytes, report): if sha256(source_bytes).hexdigest() == EMPTY: raise TypedRejection("no_input") # never render return render(report) ```
Four lines. It costs nothing. It is absent from nearly every pipeline I have reviewed, because the condition it guards only occurs when someone forgets to attach the source — and the person who forgets is usually the person who wrote the guard.
There is a fourth failure, subtler than the other three: framework weight exceeding input weight. A nine-dimension framework applied to a one-line input does not produce nine dimensions of insight. It produces one dimension of insight and eight dimensions of fiction, because a framework that must render will render. The refusal report I was reading was honest enough to name this — it flagged "over-analysis risk" for short inputs and proposed a degraded lightweight mode. That is the correct instinct, and it is the instinct that gets overruled in production, because a lightweight mode returns less than the full template and less is indistinguishable from worse.
Now the confidence field. The report carried confidence: none on every hidden-inference row. That is a category mistake at the schema level. A record whose confidence is none should not be annotated; it should be invalidated. A confidence field that accepts "none" as a legal value converts an epistemic failure into a formatting detail, and formatting details get carried forward.
Which brings the failure to the execution layer. In March I audited the smart contract interface between an AI trading agent and its oracle feed. The agent read a price, computed a position, executed. The vulnerability was temporal: when the oracle update lagged, the agent did not read "unknown." It read the last known price and treated it as current. A stale value was coerced into a valid value by the execution layer's default branch — the same WEIGHTS.get(level, 0) pattern, one layer down and denominated in dollars. I wrote the fix as a time-lock: reject any settlement where block.timestamp - price.updated_at > threshold, and fail closed rather than falling back to the last quote.
Both bugs are null coercion. In the research pipeline, "no input" becomes "no risk." In the trading agent, "no update" becomes "last price." One costs you a document. The other costs you a vault. The class is identical, and the industry files the first under formatting and the second under critical — which tells you the taxonomy is organized around consequences rather than causes.
Here is the counter-intuitive part, and it is where I expect disagreement.
The refusal was the correct output, and it is the output most likely to be patched out. Benchmarks score completeness. A pipeline that returns a blank document scores zero on every evaluation harness ever built, regardless of whether the input deserved a blank document. So the pressure is one-directional: add a retry, add a fallback prompt, add an instruction to produce a best-effort analysis even when information is thin. Each of those changes raises the benchmark and degrades the system, because the system's single most valuable behavior — refusing to fabricate — is precisely the behavior the benchmark cannot see. Verification > Reputation. A pipeline that has never emitted a rejection has not been verified. It has been untested against the only condition that separates analysis from generation.
Silence before the breach is not a metaphor in this context. It is the observable state of a system that will return nine sections of N/A with full confidence and no warning, and will do it in the same format it uses when it is right.
Code is law, until it isn't — and here the code did exactly what it was told. The schema permitted N/A. The validator permitted N/A. The aggregator priced N/A at zero. Three permissive defaults, each individually defensible, composing into a document that reads as a clean bill of health. One unchecked loop, one drained vault; one unchecked field, one fabricated thesis.
A better refusal schema is not difficult. It needs three fields: a rejected boolean set true, a missing_inputs array naming what was absent, and a minimum_viable_input spec describing exactly what would make analysis possible. It should not carry the nine dimensions at all. An empty report that still has nine dimension headers is an invitation to fill them.
The forward-looking question for anyone running research agents against on-chain state is not whether the model is accurate. It is whether the pipeline has a rejection rate at all. Ask for it. A number near zero does not mean the system is good. It means the system has never been permitted to say no, and the first time it should have, it will instead hand you forty-one rows of N/A and call it due diligence.
