DSAIL Quickstarts Guides Compare REST MCP tools Errors

Enforce a written policy in an LLM application and get the same answer every time

Why an LLM-judged policy check drifts, and how to check extracted values against a written policy so the same facts produce the same bytes, every time.

You have a policy in English and an application that reads documents with a model. You need the policy enforced, and you need the same answer every time — not a paraphrase of the policy, applied slightly differently on each call.

Why the obvious approach drifts

Asking the model to judge compliance directly ("does this expense claim comply with the policy below?") produces an answer that depends on the prompt, the model version, the sampling temperature and the order of the evidence. Two runs over the same document can disagree, and neither can say which rule it applied. Nobody can audit a judgement that was never written down.

Split the job at the facts

The seam is the claim dictionary: a small set of typed facts the document is asked for, and nothing else.

  1. Turn a written policy into rules a program can check. The policy becomes a DSAIL ruleset: declared claims (the questions) and assertions (what must hold). A host model drafts it, a person confirms the English summary first, and the service compiles it to a content hash.
  2. Extract the facts on your model, using the prompt pack generated from the ruleset — one question per claim, with the answer format and unit. Your model, your credentials, your document; nothing of it crosses to the service.
  3. Check the claim dictionary against the hash. The service solves the compiled rules over the values and returns each assertion's own result. No model is involved in this step.

What "the same answer every time" means, precisely

Given the same (ruleset_hash, unit_library_hash) and the same claim dictionary, the check response is byte-identical, through the REST door and the MCP door alike. The payload is pure over its inputs: no timestamps, no request id, no duration. A FALSE names the rule that decided, with a counterexample — the values under which the assertion fails — so the answer is not only stable but explainable.

{"name": "receipt_over_75", "check": "FALSE",
 "counterexample": "[amount = 120, has_receipt = False]"}

The formal guarantee ends at the claim dictionary. Whether the extracted values describe the document faithfully is the extraction's responsibility, and it is where your evaluation effort belongs — a narrow, testable surface instead of an open-ended judgement.

What you get back, and what you do not

Every assertion answers TRUE, FALSE, UNKNOWN or AMBIGUOUS. There is no overall verdict: the service does not know what a violated cap costs your business, so it does not rank one failure above another. Fold the results in your own code. A missing fact is UNKNOWN, not FALSE: unknown is an answer, not a guess.

Try it