LLM-as-a-Verifier turns uncertainty over score tokens into a continuous reward, then uses that signal to select better trajectories, track progress, and shape learning—without training a separate reward model.
Generate several attempts, compare completed trajectories under explicit criteria, preserve the verifier’s full score distribution, and spend extra compute only where candidate quality is uncertain.
It extracts a continuous score from score-token logprobs and aggregates that score across criteria and repeated evaluations.
The verifier does not make the base generator intrinsically smarter. It improves the deployed system by recovering a better answer already present in the candidate pool.
The framework works only when its upstream dependencies are satisfied. Candidate diversity creates headroom; domain criteria define correctness; a logprob-capable verifier converts both into a reward that the ranker and downstream systems can consume.
| Dependency | Why required | Failure if missing |
|---|---|---|
| Candidate diversity | Creates recoverable Best-of-N headroom | Verifier can only select equally weak attempts |
| Visible evidence | Lets the verifier inspect actual task state | Scores reward narration instead of completion |
| Domain criteria | Decomposes correctness into judgeable factors | Generic preferences replace task validity |
| Score-token logprobs | Expose the distribution needed for expectation | Falls back to coarse discrete judging |
| Verification budget | Funds repetition and pairwise comparisons | Higher variance and weaker selection |
The conceptual system and the repository implementation mirror each other. Switch views to trace the mathematics into concrete modules and APIs.
Map each ordered score token to a scalar and take the probability-weighted expectation instead of the argmax token.
G improves score resolution, K reduces sampling variance, and C decomposes a complex judgment into simpler factors.
Convert two continuous trajectory strengths into a soft probability that candidate A is better than candidate B.
Use a ring pass to reduce A/B position bias, select likely leaders as pivots, then concentrate comparisons around them.
Apply the same reward to successive prefixes to estimate progress, detect stalls, and generate dense learning feedback.
The generator samples solutions; the verifier sees completed alternatives and explicit criteria. Different context activates a different capability.
Computes continuous rewards, normalizes scores, caches comparisons, and records token usage.
llm_verifier/fine_grained_reward.pyRuns ring comparisons, chooses pivots, alternates prompt positions, aggregates win mass, and returns rankings.
llm_verifier/pivot_tournament.pyLoads domain criteria, normalizes user inputs, and builds pairwise verification prompts.
llm_verifier/prompts.py · criteria/*.mdConnect task datasets, candidate trajectories, harness metadata, and reproduction settings.
benchmarks.py · loaders.py · data/Scores a growing trace checkpoint by checkpoint without seeing future actions, enabling live stopping or resampling.
llm_verifier/progress.pySelect chooses the best candidate, compare exposes pairwise rewards, and track returns a progress curve.
Places stable task and trajectory context before changing criteria, warms shared prefixes, and fans out repeated calls.
Uses DeepSeek, Gemini, or an OpenAI-compatible server that exposes scoring-token logprobs and required modalities.
The framework succeeds by matching each source of verification error with a specific system response. Its thought process is less “ask the model again” and more “design a measurement pipeline.”
Argmax discards useful probability mass. Expected value keeps weak preferences that become reliable after aggregation.
A model may fail while acting yet recognize the failure when shown completed alternatives and explicit criteria.
Several simple judgments are more controllable than one vague holistic score.
Repeated evaluations reduce prompt-specific variance when errors are not perfectly correlated.
Pivot selection focuses verification on plausible leaders instead of wasting calls on obviously weak candidates.
Low-cost rollouts and high prefix-cache reuse make inference-time scaling practical for DeepSeek-class models.
Measure Oracle@N first. If multiple samples rarely contain a better answer, verification cannot create one.
Expose terminal state, tests, source provenance, images, or other observable evidence—not the agent’s self-description.
Translate vague quality into independent criteria such as task completion, evidence, robustness, and policy compliance.
Use score margins to decide whether one pass is enough or whether to add repetition, pivots, or human escalation.
Select automatically for reversible tasks; require deterministic controls or human review when consequences are material.
Five trajectories create 96.6% Oracle@5 headroom. Continuous pairwise verification recovers enough of that pool to move from 78.7% Pass@1 to 88.0% ± 0.6.
Think of LLM-as-a-Verifier as a measurement and selection layer around an agent—not a replacement for truth, controls, or model training.
Its deepest principle is architectural: generate diversity, preserve uncertainty, decompose judgment, aggregate independent evidence, and allocate verification compute where it changes the decision.