none_applies, pauses the run for
a person.
With no cap, a run ends when the judge stops it or the review passes. You
can add a cap as a backstop. Put a person’s decision where the process
needs one; this example asks before publication.
If a fixed number of rounds is enough,
get a second opinion
and set the writer’s revision limit.
Shape
The stopping rule
The judge’s seat isjev() from
@obversa/engine-jev-api, one line like any
other seat. Offline, recordedJudge from @obversa/runtime/testing stands
in for it and replays the answers in judge.json:
examples/judge-stops-the-loop.ts (excerpt)
write stage. The reader reviews the page, and
refine names the judge:
examples/judge-stops-the-loop.ts (excerpt)
block, should-fix or
nice-to-have. A block is a false claim or a sentence a person would not
understand.
When a judge is set, it answers one question per finding, a block included,
with a one-line reason and a confidence. The question is a score over four
ordered levels, the same scale in every round: cosmetic (wording, style,
naming or polish past the bar), minor (a real fault a reader of this work
rarely meets, an edge case outside normal use, or a nice-to-have),
impactful (behaviour wrong under normal use, a requirement of the brief
unmet, a test that cannot fail, or a risk of data loss, a security hole or
a broken rule of the work) and blocking (so wrong the work must not land
while it stands). A real finding is not waved
through because the rest of the list was trivial.
- The judge sends at least one finding back. The page goes back: the writer gets only the findings the judge sent back, each with the judge’s reason. A finding sent back unsure also carries the level it scored and the confidence its answer carried.
- The judge holds every finding back. A confident
holds,over_polishing,not_convergingorcontinuestands the page as a pass, with the held findings attached, and an answer too weak to route it, ornone_applies, pauses it for a person. - The reader’s next round. The reader is told which findings did not go back and why, so it does not raise them again.
hold with the level its score
landed on and the confidence the answer carried, and each is collected on
minorFindings
with its reviewer and the evidence it wrote, where a file and line
usually sit: the list a person reads before the work
lands. The judge’s own answer routes the finding; the reviewer’s tag only
informs what the judge reads.
Every route the judge’s answer drives is gated on its confidence, with the
bars in JUDGE_CONFIDENCE. A finding scored impactful goes back at or
above the send-back bar (0.7); a blocking one clears a lower bar (0.6),
because a missed blocker is the worst miss. A stop answer that would end
the run needs the stop bar (0.8). A round answer under the hand-off bar
(0.6), a none_applies choice, and a reply that cannot be read go to a
person.
The floor covers a finding’s answer too: under it the call is unsure,
never confident enough to route the finding on its own. It never pauses
the round for that alone, because it costs no extra build. The finding
goes back with the findings the judge acts on, marked unsure with its
level, confidence and reason so the writer knows how much weight to give
it, and the record says it went back unsure. When nothing is going back,
the unsure finding is held like any other, onto minorFindings for the
person who reads the run before the work lands.
When a pause the round needed anyway, a product decision, a hand-off
the round answer raised, or a spent cap asking, carries the judge’s
calls on the findings, the person’s answer settles each one. An "act"
named under the finding’s id in feedback.findings sends it back to the
writer with its evidence. A "hold" keeps it back on minorFindings,
even when the judge acted on it. A finding the answer does not name
keeps the judge’s call: an acted one goes back with the answer, a held
one stays held, unless the answer approves the work
(decision: "approved"), which holds the acted ones too. A pending
finding a question saved before this routing still carries is held when
the answer does not name it. When the call sends no finding back, the
run ends there as a pass and buys no extra build. The
values are starting bars, tuned against labelled findings with
tools/calibrate-judge.mjs.
With no cap, the rounds end when the judge stops them or the reader finds
nothing that fails. A reader that tags every finding block does not decide
alone how long the run goes on, because the judge still weighs each block.
A block is sent back without a judge only when there is no judge: a plain
refine number on a workflow() stage, or rounds in a dag() node’s
review().
When the stage also sets goal, a goal check runs before the reader,
every round. A requirement of the brief that it finds unmet goes straight
back to the writer. The reader does not run that round, and the judge is
not asked, because a requirement in the brief is never skipped as polish.
Check the brief was met shows it.
The approval is a plain function. It reads the page, puts its sha256 in the
question and sends a person’s no back to the writer.
With several reviewers, set synthesise on the stage as well. The panel
then merges the findings that name the same problem, lets each reviewer
vote once on the findings it did not raise, and drops a finding when the
reviewers who reject it outnumber those who raised or backed it, unless it
is a block. The judge decides each finding on that one
list, with who raised it and how the others voted.
Ask a panel shows it.
A cap as a backstop
To bound the rounds as well, pass a cap:judge(judgeSeat, { cap: 3 }).
A cap of 3 allows three refinements:
the first draft and three more, four builds the reviews paid for. It means
the same on a workflow() stage’s refine and in a dag() node’s
review({ judge }).
A failed check or a declared file left missing sends the work back on the
check budget, not the cap: up to checkRounds more builds (3 by default,
and on a workflow() stage a panel reviews).
When the review of the last build does not pass, the judge is asked about
it too and told that this is the last round. Its answers decide the outcome:
holdsorover_polishing, confident. The work stands as a pass with the judge’s reason. The findings still open are on the outcome asopenFindings, and in the judge’s entry in the record.product_decision. A person is asked, and a change they ask for buys exactly one more round, past the cap, so the builder can apply it. The record marks the send-backproductDecision. An answer that holds every finding, or approves the work and names none to act on, sends nothing back and lets the work stand as a pass.- A hand-off. An answer under its bar,
none_applies, or a reply the runtime could not read pauses the run for a person, at the cap as before it. A run never ends as a silent fail on an answer it could not trust. - Every finding held back. The per-finding answers send nothing
back, so the work stands, same as before the cap: a confident
holds,over_polishing,not_convergingorcontinuestands the work as a pass, with the held findings attached, and a weak stop answer ornone_appliespauses the run for a person. The run fails, with the cap named, when the judge does not decide each finding, with custom questions and noperFinding, or when a review raised no findings to decide, and still asks for another round past the cap.
converging on
the send-back, and the record says why. When the count is not falling,
the run pauses and asks a person, the open findings attached, as a
resumable product decision. Their call decides each finding the same way
as on any pause. The acted findings they do not hold go back with their
answer, in one more round past the cap. When they hold every acted
finding, or approve the work and name none to act on, nothing goes back
and the work stands as a pass, with the held findings on minorFindings.
A run never ends as a silent fail while findings the judge called
impactful are still open.
When the judge decides each finding on the last round, holding every
finding back lets the work stand on any confident stop answer, a
not_converging included. A block in the last round goes to the judge
too, with the same rules.
What the judge sees
The judge reads no file itself. The runtime puts together what it reads, in code, with no extra model call. For the same rounds, aworkflow() stage
and a dag() give the judge the same input. The judge reads it in four
labelled parts, always in this order, and then the questions. The four
parts, and cut when the size limit cut something, are the state of the
judge’s { state, questions } prompt.
why: what the work is for. The use case, and the target’sdescandgate. Aworkflow()takes the use case from the brief’sUse case:paragraph; adag()passes its ownuseCase. A person’s answers to the judge’s earlier product decisions are here too. The whole brief is not sent: the lines of it a finding touches ride on that finding’s own entry.what: the change. In a git workspace, the list of files changed since the run of the workflow or dag began. When the target writes a file, the judge also gets its content: the first file a stagewrites, or a dag node’sfile. The diffs themselves ride on the findings that cite them.how: this round’s evidence. The latest result of each check step in the graph, arun:stage or acommandJob: its command, its status and, when it did not pass, its output. The verdicts of the target’s own goal check, when it has one. Each finding, with the id its question uses, who raised it, its severity, the lines of the brief it touches, and the hunks of the diff it cites: for each changed file and line a finding cites, such assrc/app.ts:42, the parts of that file’s diff within ten lines of that line, and the whole diff when it cites a file with no line. When the panel synthesised, each finding also carries the other reviewers’ votes and their reasons.when: where the rounds stand. The round number, the cap, and whether this is the last round. The numbers the runtime computed, so no question asks the model to count or compare: how many findings are open, how many the judge acted on in this round and in the round before, whether that count is falling, and which findings repeat: a repeat is the same reviewer, the same file and the same claim, normalised so a moved line still counts. Then every earlier round summarised to its counts, how many findings it acted on and its decision, the findings the judge held back with its reasons and levels, and in a git workspace, the files changed since the last round.
The size limit
What the judge reads can be large, so it is kept to 50000 characters, counted as JSON. Set another limit with thejudgeContextLimit option of
run(), for example run(job, { judgeContextLimit: 20000 }).
When the four parts are over the limit, the runtime cuts every long prose
part down to one shared length, set so that the whole fits. A short part
is cut only when cutting the long ones is not enough. The prose is cut
first; only when no length of prose can fit the packet are the round lists
cut too. These parts can be cut:
- Texts. The use case, the
desc, thegate, the file’s content, each check’s output, and on each finding the brief lines and the diff hunk attached to it. A text keeps its start. - Lists. A person’s earlier answers, the changed files, the goal check’s verdicts, the earlier rounds, the findings held back before and the files changed since the last round. A list keeps whole items. The earlier rounds, the held-back findings and the answers keep the latest. The other lists keep their first items.
when. When they, with the names, numbers
and commands around them, are over the limit on their own, what the judge
reads stays over it, and the last note says so.
When the limit cuts something, cut, after the four parts, names each
cut, for example
the content of page.md: kept the first 20000 of 31250 characters.
Each judge entry in the record names the target and the round. It also
gives the size of what the judge read, in characters, and the same list of
cuts.
The questions
For each finding in the round, the judge gets one question, keyedfinding-1, finding-2 and so on in the order of the round’s findings.
findingQuestions() from @obversa/runtime builds
them. Each is a score over FINDING_LEVELS, cosmetic to blocking, with
a one-line reason and a confidence. The judge
answers the finding questions in a first call; the runtime reads its
scores, computes how many findings it acted on this round against the
round before, and asks the questions about the whole round in a second
call that carries those numbers. The
default set of those is stopQuestions() from
@obversa/runtime. It asks two questions, one
judgement each:
product_decision, anoul. Must a person settle a product decision before this review can continue? Its true and false criteria say what each side means.stop_reason, achoice. If no finding is sent back, how should the run end? Each option has criteria no other option shares:holds,over_polishing,not_converging,continue, andnone_appliesfor the case that fits none of them.
JUDGE_CONFIDENCE. A finding the judge scores
impactful at or over the send-back bar, or blocking at or over its lower
one, is recorded act, and another round runs. An answer under the
hand-off floor is unsure: it is recorded act too when at least one
finding is going back, still carrying its level, confidence and reason,
and hold when none is. Every other finding is
recorded hold with its level and confidence and joins the collected
list. When
every finding is held back, the work stands: a confident holds,
over_polishing or not_converging ships it as a pass with the
collected findings on the outcome as minorFindings, a continue that
clears the stop bar with nothing to send back lets the work stand too,
and a weaker answer pauses the run for a person like any weak ending.
A run does not fail while no finding the judge acts on is open.
An acted finding still beats a chosen ending, so a not_converging
cannot drop a finding the judge sent back. Two answers
still reach a person first: a product_decision the judge is sure of or
cannot rule out, and any answer too weak to route on (under the hand-off
bar, none_applies, or a stop under the stop bar), which pauses the run
with the question attached instead of spending a build. When the judge
itself cannot answer, the loop stops the same way. A failed judge call
pauses the run with the engine’s own error. A reply that does not parse,
or that leaves an asked question without a usable answer, pauses it with
a validation error. A resumed run asks the judge again for the same round,
over the review that was already done. Nothing builds and nothing reviews
again first.
Each judge entry in the record carries the model that answered: the model
the engine reports, or, when it reports none, the model the request
asked for, marked modelSource: 'requested' rather than 'reported'.
A dag() node takes the same judge() value in review({ judge }) on its
refine, which covers the node’s own reviews and the send-backs other
nodes aim at it. The node the work goes back to gets only the findings the
judge called impactful, and a stop means the same thing in both places. The judge
runs as part of the node that sent the work back, inside that node’s time
limit. The unmet requirements a goalCheck() node sends back go to the
target without the judge.
A stage a person reviews takes the same judge. Each refusal goes to the
judge with the person’s note as a finding, as a reviewer’s findings do.
Pass your own set as judge(seat, { questions }) when this wording
does not fit your use case. With your own set, the judge is not asked about
each finding, and the whole round goes back or stands on one answer. When
the judge chooses holds or over_polishing, the work stands as a pass
carrying the judge’s reason. Every other chosen stop leaves the review’s
failure in place. A custom question set that wants a reason of its own to
let the work stand names that choice holds or over_polishing. Add
perFinding: true to ask about each finding with your own set as well. The
judge engine takes { state, questions } as its prompt and returns one
answer per question; the Jev API engine page has
the shape.
The built-in stopQuestions() also offers product_decision: the work
needs a person’s judgement before review can continue. Pass an interaction
binding to judge() to open your review surface. Its request includes the
work and the review history. After the person answers, the work goes back
to the writer. The writer gets the person’s structured feedback, their
composed prompt and the findings they answered. The review then runs on
the new draft, and only after that does the judge decide again. With a
cap, the round the answer buys counts against it like any other round,
with one exception: a product decision the judge asks after the last
review the cap allows buys exactly one more round, past the cap, and the
record marks the send-back productDecision. Each product decision a
person answers buys its own round that way. Without an interaction handler,
answer through the run’s stored callbacks client, or on the
run page with a written decision. This gets an answer while
the work is being revised. A later human review can still require you to
approve the finished work.
What the run did
Run offline, with the judge’s answers replayed fromjudge.json beside the
brief and the person’s answer recorded in approve.json, the example
printed:
Output
should-fix (the page never
says where to run the command) and a nice-to-have. The judge scored the
first impactful and the second minor, so the writer got only the first. The third
pass found one nice-to-have, the judge scored it minor and chose holds,
so the page stood.
The held-back findings stayed on the record as minorFindings, for a person
to read before the page lands.
The record holds each round’s findings and the judge’s answers. Each answer
also says whether the page went back or stopped, which answer decided that,
and for a stop, whether the step passed or failed. It lists each finding’s
id with the judge’s decision and reason, for example:
JUDGE=jev, and TYPESAFE_ENDPOINT and TYPESAFE_API_KEY in the
environment, the same file asks Jev instead of replaying.
Next steps
- Feedback loops: the check, the target and the stopping rule, and the other shapes a loop takes.
- Jev API engine: the judge’s question types and how it reports its identity.
- A person decides: the approval step this loop ends on, bound to the exact bytes.