A Free Model's Review Had No Evidence. The Maintainer Added a Claim Ledger.

The patch changed a retry loop in a small queue worker. It was only 27 lines. It looked boring. The reviewer model returned a single sentence: "No blocking issue. The exception path returns the right status."

The maintainer shipped it. Three days later, the worker stopped draining jobs after a transient network error. The exception path did not return the expected status in every branch. The model had not made a testable claim. It had only produced a verdict.

This article is a composite reconstruction. The line counts, claim counts, and duration are placeholders. The code is a minimal harness you can run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access made the repeated experiments cheap to run. The free server option gave an isolated place for the harness instead of using production compute.

The first correction was not to ask for a better answer. It was to stop asking for an answer at all. A review should be a sequence of claims. "Safe" is not a claim. "The catch block at line 14 returns RETRY" is a claim. A claim can be checked against the patch. A verdict cannot.

The maintainer built a small ledger. The goal was to attach every model statement to a line reference. The ledger does not judge correctness. It only judges whether the reviewer pointed at a real place. That is a deliberately low bar. It is enough to stop confident unsupported verdicts.

The workflow has five steps.

First, export the patch as a unified diff. Second, send the diff to the model with a narrow prompt. Third, parse numbered JSON claims from the response. Fourth, verify that each cited line exists in the added or context lines of the patch. Fifth, write every claim to a JSONL trace file. The final review is accepted only when every claim has a valid line reference and no unparsed verdict remains.

The prompt is important. It asks for claims, not a conclusion. The model is allowed to say "I found no claims about this diff." That is a valid empty review. It is not allowed to say "safe" without a line.

#!/usr/bin/env python3
import json
import re
import sys
from pathlib import Path


def model_call(prompt, patch_text):
    # Replace this function with your HTTP client.
    # The return value is the raw model text.
    return json.dumps([
        {'claim': 'catch block returns RETRY', 'line': 14},
        {'claim': 'timeout is clamped', 'line': 22},
        {'claim': 'queue is empty', 'line': 3},
    ])


def parse_claims(raw):
    claims = []
    for match in re.finditer(r'\{[^{}]*\}', raw):
        try:
            item = json.loads(match.group(0))
            if 'claim' in item and 'line' in item:
                claims.append(item)
        except json.JSONDecodeError:
            continue
    return claims


def new_line_numbers(patch_text):
    line_numbers = set()
    current_new = None
    for line in patch_text.splitlines():
        if line.startswith('@@'):
            m = re.search(r'\+(\d+)', line)
            if m:
                current_new = int(m.group(1))
        elif line.startswith('+') and not line.startswith('+++'):
            if current_new is not None:
                line_numbers.add(current_new)
                current_new += 1
        elif line.startswith(' ') and current_new is not None:
            line_numbers.add(current_new)
            current_new += 1
        elif line.startswith('-') and not line.startswith('---'):
            pass
    return line_numbers


def ledger_entry(i, claim, valid):
    return {
        'claim_id': i,
        'line': claim.get('line'),
        'text': claim.get('claim'),
        'parse': True,
        'line_in_patch': valid,
    }


def main():
    if len(sys.argv) < 2:
        raise SystemExit('usage: python claim_ledger.py patch.diff')
    patch = Path(sys.argv[1]).read_text()
    raw = model_call('formal claims only', patch)
    claims = parse_claims(raw)
    valid_lines = new_line_numbers(patch)
    ledger = []
    for i, claim in enumerate(claims, 1):
        try:
            line = int(claim['line'])
        except (TypeError, ValueError):
            line = -1
        ledger.append(ledger_entry(i, claim, line in valid_lines))
    orphan = [entry for entry in ledger if not entry['line_in_patch']]
    print(f'parsed={len(claims)} orphan={len(orphan)}')
    print(json.dumps(ledger, indent=2))
    return 1 if orphan else 0


if __name__ == '__main__':
    raise SystemExit(main())

The parser is deliberately boring. It does not know Python or C++. It does not know what the patch is supposed to do. It knows only whether the reviewer pointed at a real hunk. That narrow check changed the maintainer's decision process.

An orphan claim is not automatically false. The model may have inspected a line outside the diff. But a patch review needs claims about changed code. If the model cannot connect a claim to a changed line, the review is not accepting the claim.

The ledger catches four common failure modes.

The table shows the ledger is not a correctness oracle. It is a trace oracle. It converts an ungrounded answer into something a human can scan in a few seconds.

A second gate helps with the third row. For each claim, the maintainer can ask a follow-up: "Name one test that would fail if this claim were false." The model may still answer confidently. The follow-up is not proof. It is another check. If the model cannot name a test, the claim stays labeled as review-only.

The workflow should not be used as the only merge gate. It does not replace compiler warnings, sanitizers, unit tests, or human review. It is especially weak for security patches, generated code, minified JavaScript, and changes where the bug is in the interaction between two files. Do not send private code to a free endpoint unless the account operator has checked the data policy. A free model endpoint can disappear or change behavior, so the harness should wrap the call in a timeout and a small retry budget.

The maintainer did not adopt the ledger to get more correct answers. The point was to get fewer unearned answers. The next step was not a better model. It was a smaller claim.