Check LLM output against a written policy without another LLM as the judge

How to check extracted values against a written policy with no model in the loop — extraction on your model, the check on a solver, and what that buys you.

The common pattern for checking an LLM's work is a second LLM: a judge prompt, a rubric, a score. It inherits every property of the first model — variance, prompt sensitivity, an explanation written after the fact. DSAIL keeps the model where it is good, at reading the document, and puts no model in the loop for the check itself.

The seam

document ──(your model, prompt pack)──► claim dictionary ──(solver)──► per-assertion results

• Left of the seam is yours: your model, your credentials, your document. The service generates the prompt pack — one question per claim, with the answer format, vocabulary and unit — and never sees what you run it on.
• Right of the seam is the service: the compiled ruleset, solved over the claim values by an SMT solver in exact arithmetic. Deterministic. No language model is called, at compile or at check.

What crosses the wire is the claim dictionary — a handful of typed values bounded by a JSON Schema — never the document.

What this buys

• The same answer every time. Same ruleset, same values, same bytes. There is no temperature to set and no judge to re-prompt.
• An explanation that is the reason, not a story. A FALSE carries the rule that decided, with a counterexample: the values under which the assertion fails, found by the solver.
• Honest gaps. A fact your model could not find is submitted as "unknown" and the rule that needed it answers UNKNOWN. Unknown is an answer, not a guess, and a judge model tends to fill it in.
• A smaller evaluation surface. The only fuzzy step left is extraction — did the model read the amount correctly? — and that is a narrow, testable question with a ground truth, unlike "was the judgement right?"
• Nothing to leak. The service holds rules, not documents.

What it does not do

It does not assess tone, relevance, toxicity or grounding; those are content properties, and guardrail frameworks are built for them. It checks whether the facts extracted from a document satisfy a written policy. Many applications need both, on different layers.

Fitting it into a pipeline

1. Compile the policy once; keep the ruleset_hash.
2. For each document: fetch the prompt pack (or cache it per hash), run the prompts on your model, assemble the claim dictionary, check.
3. Read rules[].assertions[].check and act on it in your own code. The service publishes no verdict, because what a FALSE costs is your decision.
4. If validation rejects the dictionary, error.failures names every failing field at once — fix them in one pass and resubmit. The dsail Python client's check_with_repair does this loop for you.

Try it

• Quickstart: REST — the three calls.
• Quickstart: Claude Code — a coding agent wiring the check into a service.
• Turn a policy document into rules a program can check — the language.