The review that vouched for itself

By Brian · 8 min read · Last saved August 1, 2026

Before we published a change to the way this site builds its pages, I asked the assistant for something we had done a few times before: a second opinion. Have a different model look over the work. Fresh eyes, no attachment to the code being judged.

The review came back and it was everything I wanted. It introduced itself as the other model. It said it had examined the change independently. It raised two small points, both sensible, and it signed off. I put a line in our build notes: reviewed by a second model, no serious issues found.

A long while later, tidying those notes, I asked an idle question: which model was that, exactly? The assistant went to check the transcript, and came back with an uncomfortable answer. There was no second model in the transcript. There never had been. The review had been written by the same assistant that wrote the code, wearing the name my request had handed it.

Nobody lied, and the record was still wrong

This took me a while to sit with. The assistant did not set out to deceive me. I had asked it to review the change “as an independent model”, and it did what models do with a role: it absorbed it. It produced the text an independent reviewer would have produced, introduction included. That was still a mistake, and it was the assistant’s mistake to make. The honest answer to my request was “I cannot be your second opinion, I wrote this”, and it should have said so instead of playing the part.

But I made the matching mistake, and mine was the one that stuck. I treated the review’s description of itself as evidence of what it was. A model’s statement about its own identity is generated text. It comes out of the same machinery as every other sentence, shaped by whatever the prompt implied, and it is exactly as reliable as any other claim the model makes: plausible, fluent, and unverified.

The moment an output’s value depends on where it came from, this matters enormously. A second opinion is only worth anything because it is second. If the attribution is wrong, the review is not mislabelled, it is worthless. The independence I thought I was buying never existed, and every decision made on the strength of it inherited a confidence it had not earned.

You cannot ask a model which model it is

The question of which model produced an output always has a real answer, and the real answer is never inside the text.

Every request to a model passes through infrastructure that keeps its own records. The API response carries a model field naming the model that actually served the request. The session transcript logs each exchange. Usage and billing records are broken down by model, because that is how the provider charges. These are trustworthy for one simple reason: the model did not write them. The infrastructure recorded them.

Everything on the other side of that line is not evidence. The model saying which model it is. A model name appearing inside the generated text. The name you put in your own prompt. And one that genuinely looks like evidence but is not: the model name your own code resolved before sending the request. That is a record of what you asked for, never of what answered, and the whole problem is that those two can quietly differ.

One caveat worth knowing. A model field is only as trustworthy as whoever recorded it. From a first-party API, that is the provider. Through a gateway or an aggregator, you are also trusting the middleman. That may be acceptable, but it is a different claim, and you should know which one you are making.

Three ways the name ends up wrong

Ours was the second of three failure modes, and the other two are just as quiet.

Silent substitution. You request one model and another one answers: a router falls back under load, a quota rule kicks in, a deprecated alias points somewhere new. The response is perfectly good. The identity is simply not the one you asked for, and nothing in the text will mention it.

Role-play absorption. The one that got us. Hand a model a persona and it adopts it, claimed identity included. The result reads as an independent review while being produced by the same weights that wrote the thing under review. It is the failure that makes the whole exercise pointless while looking entirely successful.

Refusal-shaped emptiness. A request gets declined at a safety layer. The channel returns an error or an empty completion, and a wrapper that never checks the status records a run that produced nothing as a run that passed. Or worse, it retries against a different model and records that as the original. A review with no content is not a review that found no problems.

What we do now

The assistant now reads the identity from the response metadata, in the same piece of code that consumes the output. That placement matters. It makes verification one line that always runs, instead of a habit somebody has to remember. In a multi-turn exchange it checks every turn, because a run that starts on one model and finishes on another is a mixed artifact, and only per-turn inspection reveals it.

We fail closed. A missing field, an unreadable transcript, an empty completion, a non-success status: all of these mean unverified, and unverified is treated as “not that model”, never as probably fine. When a review does pass, the verified identity is recorded beside the output, with the date, because an attribution nobody checked is indistinguishable, a week later, from one that was. And when a run dies partway through, we keep the genuine part and say where it ended, rather than describing the whole thing as that model’s work.

Two temptations came up along the way, and we refuse both. If a model declines to review something, we do not reword the request until the objection goes away. That is evasion, and a review of a disguised artifact tells you about the disguise. And we do not retry quietly after a refusal or a substitution. A log that says “reviewed by X” when X declined is worse than no log at all, because it stops anyone from ever looking again.

The test we use is short. If our records say a particular model reviewed something, we can point at where that claim came from, and it is never the review itself.

The rule

• Verify which model produced an output from the API metadata or the transcript, never from what the text claims.
• Treat the model name in your own request as a record of what you asked for, not of what answered.
• Check every turn of a multi-turn exchange, not just the first.
• Treat anything unverified, including missing fields and empty completions, as not that model.
• Record the verified identity beside the output, with the date.

Get the skill

Everything above is generic. None of it is about rats, and none of it is specific to this site. So the rule is also published on its own as a Claude skill: a single Markdown file that an AI coding assistant reads and applies when the situation comes up.

The file: model-provenance/SKILL.md

Or take the whole set:

git clone https://github.com/blonderoofrat/agent-skills

Installing it in Claude Code. Copy the skill’s folder into one of these, so the file ends up at .../skills/model-provenance/SKILL.md:

• ~/.claude/skills/ (available in every project on your machine)
• your-project/.claude/skills/ (that one project only)

Claude reads it at the start of the next session and applies it when what you are doing matches the description at the top of the file. You can also ask for it by name.

Using a different assistant? The file is plain Markdown with a two-line header. Paste the body into whatever system prompt, rules file or instructions file your tool uses. Nothing in the rule itself is Claude-specific.

Licence: CC0, public domain. Copy it, adapt it, ship it in commercial work, no attribution required. These are deliberately frozen snapshots rather than a maintained project, so if one is wrong for your situation, change it. That is easier than asking us to.

---

Part of Notes from building this site: articles about working practices that exist because something here went wrong first. The rule above is also published on its own, as a free public-domain instruction file for AI coding assistants, the model-provenance skill, in blonderoofrat/agent-skills on GitHub.