Team Operations: Setting Up a Disagreement Ledger for Shared AI Prompts
Most teams that use AI models in daily operations discover the same problem sooner or later. Someone asks a model a question, gets a confident answer, and passes it along. A week later a colleague asks the same question in a slightly different way and receives a different answer, and nobody notices until a decision has already been made. The fix is rarely a better model. It is a better record of how answers were produced and where they disagreed. This article describes a disagreement ledger: a simple, shared log that treats model disagreement as operational information rather than noise. It applies to any team, and it works whether you run one model or several.
Why disagreement deserves its own record
When two answers match, there is little to write down. When they differ, the difference usually tells you something about the question itself. The prompt may be ambiguous, the required context may be missing, or the model may be extrapolating beyond what it actually knows. Each of these conditions matters to an operator, and each is invisible if you only keep the answer you happened to accept.
Open Harness is built around this idea: running one prompt through up to six models side by side exposes drift, hallucinations, and weak outputs before they cost time, money, or credibility. Whatever tool you use, the operating principle is the same. Disagreement is a signal to slow down and look, and a team needs a place to put what it saw.
Building the ledger: fields that earn their place
A disagreement ledger does not need special software. A shared spreadsheet or a document table is enough to start. Keep the number of fields small enough that people will actually fill them in. A useful minimum includes the date, the exact prompt text, the models queried, a short summary of where their answers diverged, the answer your team adopted, and the name of the person responsible for that decision.
Two fields are easy to skip and worth keeping: a note on what information would have resolved the disagreement, and a status marking whether the decision has since been confirmed, reversed, or left open. Over a few months these notes reveal patterns, such as which question types consistently split models or which teammates tend to resolve splits without checking the reasoning.
Assigning ownership before the split happens
The most common failure is not a bad answer but an unowned one. A model returns three plausible options, and everyone assumes someone else will choose. The ledger should name an owner for each category of decision before the disagreements arrive. Ownership can be defined by domain: one person reviews customer-facing wording, another reviews technical configuration, and a third signs off on anything involving commitments to outside parties.
Write the rule down in plain language. For example: if two or more models disagree on a factual claim about our product, the support lead decides and records the source they checked. This turns a vague sense of responsibility into a concrete step that a new team member can follow on their first day.
A practical review checklist
Use this short checklist each time a split appears in your ledger. First, restate the question in one sentence and confirm the prompt actually asked that question. Second, identify the specific claim where the models diverged, rather than the overall tone of their answers. Third, check that claim against a source your team trusts, such as internal documentation, a configuration file, or a person with direct knowledge. Fourth, record which source you used. Fifth, decide whether the open question should change the prompt for next time, for instance by adding context the model lacked.
This review takes minutes when it becomes routine. Its value comes from consistency. A team that performs it every time builds a shared sense of which kinds of questions are safe to automate and which need a person in the loop.
Questions to ask before you scale a decision process
Before you formalize the ledger into a required step, ask a few questions. Does every recorded split have a named owner, or are some still floating? Can a teammate who was not present reconstruct why a decision was made from the entry alone? Are the sources you cite the kind you would accept in a review meeting? Does the ledger show that certain prompts repeatedly produce disagreement, suggesting the prompt itself needs rewriting?
If the answer to any of these is no, the process is not yet ready to scale. Fix the gaps while the volume is small. A ledger that only works when its creator is present will fail the moment that person is unavailable, and the cost of that failure falls on whoever inherits the unresolved entries.