Skip to main content
Give a small decision model the job of deciding whether another revision would improve the work. Use it for a page, a plan or a shortlist when you can’t know how many rounds it will need. The reviewer says what needs fixing. The judge reads each note and the history of revisions, then scores each one from cosmetic to blocking, with a confidence on each call. A note goes back to the writer when its score is impactful or blocking and its confidence clears the bar for its level. A note whose confidence sits under the hand-off floor goes back too when at least one other note is going back, marked unsure with its level, confidence and reason: the round spends no extra build on it, and asks no person for it. When at least one is sent back, the workflow carries those notes to the writer and runs the next round. When none is, the work stands unless the judge’s stop answer ends the run in a fail, or an answer too weak to route it, or none_applies, pauses the run for a person. With no cap, a run ends when the judge stops it or the review passes. You can add a cap as a backstop. Put a person’s decision where the process needs one; this example asks before publication. If a fixed number of rounds is enough, get a second opinion and set the writer’s revision limit.

Shape

The stopping rule

The judge’s seat is jev() from @obversa/engine-jev-api, one line like any other seat. Offline, recordedJudge from @obversa/runtime/testing stands in for it and replays the answers in judge.json:
examples/judge-stops-the-loop.ts (excerpt)
The judge sits on the write stage. The reader reviews the page, and refine names the judge:
examples/judge-stops-the-loop.ts (excerpt)
The brief tells the reader to tag each finding block, should-fix or nice-to-have. A block is a false claim or a sentence a person would not understand. When a judge is set, it answers one question per finding, a block included, with a one-line reason and a confidence. The question is a score over four ordered levels, the same scale in every round: cosmetic (wording, style, naming or polish past the bar), minor (a real fault a reader of this work rarely meets, an edge case outside normal use, or a nice-to-have), impactful (behaviour wrong under normal use, a requirement of the brief unmet, a test that cannot fail, or a risk of data loss, a security hole or a broken rule of the work) and blocking (so wrong the work must not land while it stands). A real finding is not waved through because the rest of the list was trivial.
  • The judge sends at least one finding back. The page goes back: the writer gets only the findings the judge sent back, each with the judge’s reason. A finding sent back unsure also carries the level it scored and the confidence its answer carried.
  • The judge holds every finding back. A confident holds, over_polishing, not_converging or continue stands the page as a pass, with the held findings attached, and an answer too weak to route it, or none_applies, pauses it for a person.
  • The reader’s next round. The reader is told which findings did not go back and why, so it does not raise them again.
A finding the judge holds back never goes to the writer again, whatever the reviewer tagged it: it is recorded hold with the level its score landed on and the confidence the answer carried, and each is collected on minorFindings with its reviewer and the evidence it wrote, where a file and line usually sit: the list a person reads before the work lands. The judge’s own answer routes the finding; the reviewer’s tag only informs what the judge reads. Every route the judge’s answer drives is gated on its confidence, with the bars in JUDGE_CONFIDENCE. A finding scored impactful goes back at or above the send-back bar (0.7); a blocking one clears a lower bar (0.6), because a missed blocker is the worst miss. A stop answer that would end the run needs the stop bar (0.8). A round answer under the hand-off bar (0.6), a none_applies choice, and a reply that cannot be read go to a person. The floor covers a finding’s answer too: under it the call is unsure, never confident enough to route the finding on its own. It never pauses the round for that alone, because it costs no extra build. The finding goes back with the findings the judge acts on, marked unsure with its level, confidence and reason so the writer knows how much weight to give it, and the record says it went back unsure. When nothing is going back, the unsure finding is held like any other, onto minorFindings for the person who reads the run before the work lands. When a pause the round needed anyway, a product decision, a hand-off the round answer raised, or a spent cap asking, carries the judge’s calls on the findings, the person’s answer settles each one. An "act" named under the finding’s id in feedback.findings sends it back to the writer with its evidence. A "hold" keeps it back on minorFindings, even when the judge acted on it. A finding the answer does not name keeps the judge’s call: an acted one goes back with the answer, a held one stays held, unless the answer approves the work (decision: "approved"), which holds the acted ones too. A pending finding a question saved before this routing still carries is held when the answer does not name it. When the call sends no finding back, the run ends there as a pass and buys no extra build. The values are starting bars, tuned against labelled findings with tools/calibrate-judge.mjs. With no cap, the rounds end when the judge stops them or the reader finds nothing that fails. A reader that tags every finding block does not decide alone how long the run goes on, because the judge still weighs each block. A block is sent back without a judge only when there is no judge: a plain refine number on a workflow() stage, or rounds in a dag() node’s review(). When the stage also sets goal, a goal check runs before the reader, every round. A requirement of the brief that it finds unmet goes straight back to the writer. The reader does not run that round, and the judge is not asked, because a requirement in the brief is never skipped as polish. Check the brief was met shows it. The approval is a plain function. It reads the page, puts its sha256 in the question and sends a person’s no back to the writer. With several reviewers, set synthesise on the stage as well. The panel then merges the findings that name the same problem, lets each reviewer vote once on the findings it did not raise, and drops a finding when the reviewers who reject it outnumber those who raised or backed it, unless it is a block. The judge decides each finding on that one list, with who raised it and how the others voted. Ask a panel shows it.

A cap as a backstop

To bound the rounds as well, pass a cap: judge(judgeSeat, { cap: 3 }). A cap of 3 allows three refinements: the first draft and three more, four builds the reviews paid for. It means the same on a workflow() stage’s refine and in a dag() node’s review({ judge }). A failed check or a declared file left missing sends the work back on the check budget, not the cap: up to checkRounds more builds (3 by default, and on a workflow() stage a panel reviews). When the review of the last build does not pass, the judge is asked about it too and told that this is the last round. Its answers decide the outcome:
  • holds or over_polishing, confident. The work stands as a pass with the judge’s reason. The findings still open are on the outcome as openFindings, and in the judge’s entry in the record.
  • product_decision. A person is asked, and a change they ask for buys exactly one more round, past the cap, so the builder can apply it. The record marks the send-back productDecision. An answer that holds every finding, or approves the work and names none to act on, sends nothing back and lets the work stand as a pass.
  • A hand-off. An answer under its bar, none_applies, or a reply the runtime could not read pauses the run for a person, at the cap as before it. A run never ends as a silent fail on an answer it could not trust.
  • Every finding held back. The per-finding answers send nothing back, so the work stands, same as before the cap: a confident holds, over_polishing, not_converging or continue stands the work as a pass, with the held findings attached, and a weak stop answer or none_applies pauses the run for a person. The run fails, with the cap named, when the judge does not decide each finding, with custom questions and no perFinding, or when a review raised no findings to decide, and still asks for another round past the cap.
Findings the judge still calls impactful change the ending. While the count it acts on is smaller than the round before (the runtime counts, not the model), the run is still converging: the spent cap grants one more round, marked converging on the send-back, and the record says why. When the count is not falling, the run pauses and asks a person, the open findings attached, as a resumable product decision. Their call decides each finding the same way as on any pause. The acted findings they do not hold go back with their answer, in one more round past the cap. When they hold every acted finding, or approve the work and name none to act on, nothing goes back and the work stands as a pass, with the held findings on minorFindings. A run never ends as a silent fail while findings the judge called impactful are still open. When the judge decides each finding on the last round, holding every finding back lets the work stand on any confident stop answer, a not_converging included. A block in the last round goes to the judge too, with the same rules.

What the judge sees

The judge reads no file itself. The runtime puts together what it reads, in code, with no extra model call. For the same rounds, a workflow() stage and a dag() give the judge the same input. The judge reads it in four labelled parts, always in this order, and then the questions. The four parts, and cut when the size limit cut something, are the state of the judge’s { state, questions } prompt.
  • why: what the work is for. The use case, and the target’s desc and gate. A workflow() takes the use case from the brief’s Use case: paragraph; a dag() passes its own useCase. A person’s answers to the judge’s earlier product decisions are here too. The whole brief is not sent: the lines of it a finding touches ride on that finding’s own entry.
  • what: the change. In a git workspace, the list of files changed since the run of the workflow or dag began. When the target writes a file, the judge also gets its content: the first file a stage writes, or a dag node’s file. The diffs themselves ride on the findings that cite them.
  • how: this round’s evidence. The latest result of each check step in the graph, a run: stage or a commandJob: its command, its status and, when it did not pass, its output. The verdicts of the target’s own goal check, when it has one. Each finding, with the id its question uses, who raised it, its severity, the lines of the brief it touches, and the hunks of the diff it cites: for each changed file and line a finding cites, such as src/app.ts:42, the parts of that file’s diff within ten lines of that line, and the whole diff when it cites a file with no line. When the panel synthesised, each finding also carries the other reviewers’ votes and their reasons.
  • when: where the rounds stand. The round number, the cap, and whether this is the last round. The numbers the runtime computed, so no question asks the model to count or compare: how many findings are open, how many the judge acted on in this round and in the round before, whether that count is falling, and which findings repeat: a repeat is the same reviewer, the same file and the same claim, normalised so a moved line still counts. Then every earlier round summarised to its counts, how many findings it acted on and its decision, the findings the judge held back with its reasons and levels, and in a git workspace, the files changed since the last round.

The size limit

What the judge reads can be large, so it is kept to 50000 characters, counted as JSON. Set another limit with the judgeContextLimit option of run(), for example run(job, { judgeContextLimit: 20000 }). When the four parts are over the limit, the runtime cuts every long prose part down to one shared length, set so that the whole fits. A short part is cut only when cutting the long ones is not enough. The prose is cut first; only when no length of prose can fit the packet are the round lists cut too. These parts can be cut:
  • Texts. The use case, the desc, the gate, the file’s content, each check’s output, and on each finding the brief lines and the diff hunk attached to it. A text keeps its start.
  • Lists. A person’s earlier answers, the changed files, the goal check’s verdicts, the earlier rounds, the findings held back before and the files changed since the last round. A list keeps whole items. The earlier rounds, the held-back findings and the answers keep the latest. The other lists keep their first items.
The findings to decide are never cut, nor the computed counts and repeats in when. When they, with the names, numbers and commands around them, are over the limit on their own, what the judge reads stays over it, and the last note says so. When the limit cuts something, cut, after the four parts, names each cut, for example the content of page.md: kept the first 20000 of 31250 characters. Each judge entry in the record names the target and the round. It also gives the size of what the judge read, in characters, and the same list of cuts.

The questions

For each finding in the round, the judge gets one question, keyed finding-1, finding-2 and so on in the order of the round’s findings. findingQuestions() from @obversa/runtime builds them. Each is a score over FINDING_LEVELS, cosmetic to blocking, with a one-line reason and a confidence. The judge answers the finding questions in a first call; the runtime reads its scores, computes how many findings it acted on this round against the round before, and asks the questions about the whole round in a second call that carries those numbers. The default set of those is stopQuestions() from @obversa/runtime. It asks two questions, one judgement each:
  • product_decision, a noul. Must a person settle a product decision before this review can continue? Its true and false criteria say what each side means.
  • stop_reason, a choice. If no finding is sent back, how should the run end? Each option has criteria no other option shares: holds, over_polishing, not_converging, continue, and none_applies for the case that fits none of them.
The answers about each finding decide the route, each gated on its confidence by the bars in JUDGE_CONFIDENCE. A finding the judge scores impactful at or over the send-back bar, or blocking at or over its lower one, is recorded act, and another round runs. An answer under the hand-off floor is unsure: it is recorded act too when at least one finding is going back, still carrying its level, confidence and reason, and hold when none is. Every other finding is recorded hold with its level and confidence and joins the collected list. When every finding is held back, the work stands: a confident holds, over_polishing or not_converging ships it as a pass with the collected findings on the outcome as minorFindings, a continue that clears the stop bar with nothing to send back lets the work stand too, and a weaker answer pauses the run for a person like any weak ending. A run does not fail while no finding the judge acts on is open. An acted finding still beats a chosen ending, so a not_converging cannot drop a finding the judge sent back. Two answers still reach a person first: a product_decision the judge is sure of or cannot rule out, and any answer too weak to route on (under the hand-off bar, none_applies, or a stop under the stop bar), which pauses the run with the question attached instead of spending a build. When the judge itself cannot answer, the loop stops the same way. A failed judge call pauses the run with the engine’s own error. A reply that does not parse, or that leaves an asked question without a usable answer, pauses it with a validation error. A resumed run asks the judge again for the same round, over the review that was already done. Nothing builds and nothing reviews again first. Each judge entry in the record carries the model that answered: the model the engine reports, or, when it reports none, the model the request asked for, marked modelSource: 'requested' rather than 'reported'. A dag() node takes the same judge() value in review({ judge }) on its refine, which covers the node’s own reviews and the send-backs other nodes aim at it. The node the work goes back to gets only the findings the judge called impactful, and a stop means the same thing in both places. The judge runs as part of the node that sent the work back, inside that node’s time limit. The unmet requirements a goalCheck() node sends back go to the target without the judge. A stage a person reviews takes the same judge. Each refusal goes to the judge with the person’s note as a finding, as a reviewer’s findings do. Pass your own set as judge(seat, { questions }) when this wording does not fit your use case. With your own set, the judge is not asked about each finding, and the whole round goes back or stands on one answer. When the judge chooses holds or over_polishing, the work stands as a pass carrying the judge’s reason. Every other chosen stop leaves the review’s failure in place. A custom question set that wants a reason of its own to let the work stand names that choice holds or over_polishing. Add perFinding: true to ask about each finding with your own set as well. The judge engine takes { state, questions } as its prompt and returns one answer per question; the Jev API engine page has the shape. The built-in stopQuestions() also offers product_decision: the work needs a person’s judgement before review can continue. Pass an interaction binding to judge() to open your review surface. Its request includes the work and the review history. After the person answers, the work goes back to the writer. The writer gets the person’s structured feedback, their composed prompt and the findings they answered. The review then runs on the new draft, and only after that does the judge decide again. With a cap, the round the answer buys counts against it like any other round, with one exception: a product decision the judge asks after the last review the cap allows buys exactly one more round, past the cap, and the record marks the send-back productDecision. Each product decision a person answers buys its own round that way. Without an interaction handler, answer through the run’s stored callbacks client, or on the run page with a written decision. This gets an answer while the work is being revised. A later human review can still require you to approve the finished work.

What the run did

Run offline, with the judge’s answers replayed from judge.json beside the brief and the person’s answer recorded in approve.json, the example printed:
Output
Three rounds. The first reader pass found two blocks. The judge scored both impactful with the confidence to send them back, so the page went back. The second found a should-fix (the page never says where to run the command) and a nice-to-have. The judge scored the first impactful and the second minor, so the writer got only the first. The third pass found one nice-to-have, the judge scored it minor and chose holds, so the page stood. The held-back findings stayed on the record as minorFindings, for a person to read before the page lands. The record holds each round’s findings and the judge’s answers. Each answer also says whether the page went back or stopped, which answer decided that, and for a stop, whether the step passed or failed. It lists each finding’s id with the judge’s decision and reason, for example:
With JUDGE=jev, and TYPESAFE_ENDPOINT and TYPESAFE_API_KEY in the environment, the same file asks Jev instead of replaying.

Next steps

  • Feedback loops: the check, the target and the stopping rule, and the other shapes a loop takes.
  • Jev API engine: the judge’s question types and how it reports its identity.
  • A person decides: the approval step this loop ends on, bound to the exact bytes.