What it takes to check an AI-assisted assessment

The quotation can be right and the conclusion wrong

Consider a fictional briefing about a transport regulation. A consultation paper says that the ministry is considering an exemption for small operators. The generated briefing says that small operators are exempt, and cites the correct paragraph. Nothing was invented in the quotation. The error is the move from a proposal to an obligation in force. A working link and an authentic document do not resolve it.

This is the problem behind our phrase “AI you can check.” We want a reviewer to see the claim, the passage being used, and the interpretation connecting them. NIST’s work on evaluation probes distinguishes faithfulness, completeness, and sufficiency: support for what was said, coverage of the source’s full message, and the adequacy of the evidence. Those are different questions, and they can produce different answers for the same paragraph. [1]

What the review record needs to say

In the proposed capsule design, an extracted claim and a reviewer decision have separate identifiers. A reviewer might accept the extraction “the ministry proposed an exemption” while rejecting the conclusion “the exemption is in force.” The decision should name its scope, source version, reason, and date. Reusing the accepted extraction would not authorize a later model to make the rejected inference.

W3C PROV provides a vocabulary for relating entities, the activities that produced them, and responsible agents. It describes provenance that people can use to assess trustworthiness; it does not award truth to a record. We use that distinction as a design constraint. An attribution to a reviewer tells us who made a decision, not whether the decision was correct. [2]

A source reference is an engineering problem

A page number is convenient for a human but insufficient for locating text reliably after a document changes. The Web Annotation Data Model describes selectors for identifying a target passage, including quoted text and character positions. Position selectors depend on the representation they address; quote selectors can be ambiguous. A practical reference should preserve the relevant representation, record its identity, and report when the passage can no longer be resolved. [3]

We would keep a digest of the captured source alongside its locator and capture date. If a reviewer cannot open the source because access expired, the interface should show that limitation rather than displaying an intact-looking citation. Retaining a copy also depends on permission and retention rules. Portable context cannot mean that every recipient automatically receives the underlying documents.

These are requirements for DIALECTICA, the knowledge and review engine under development. PRAXIS is the agentic workspace where a person would inspect the proposal and its supporting material. A complete, reliable round trip across ingestion, review, reuse, and export remains work to validate.

Test the reviewer’s task

Our proposed evaluation would compare the same source collection and analytical task across ordinary retrieval with citations and a capsule-assisted workflow. It would seed known errors: an outdated provision, an unsupported inference, a missing qualification, and a citation pointing at the wrong version. Independent reviewers would judge the resulting assessments without knowing which workflow produced them.

We would report error detection, incorrect approvals, important omissions, completion time, and disagreements between reviewers. Faster review would be a poor result if it increased acceptance of unsupported claims. We would also include the effort of preparing and updating the capsule, rather than counting only the final drafting step. NIST’s ARIA planning manual is relevant here because it combines model testing, red teaming, and user testing in a holistic evaluation approach. [4]

No results from this proposed study are claimed here. The useful outcome would be evidence about where structured review earns its cost, where ordinary retrieval is enough, and which mistakes survive both approaches.

Follow the corrected sentence, not a badge

In the fictional reporting brief, “July is adopted” fails against S03 §1. The replacement preserves “considers July” and unresolved adoption. Its dependency record links sentence, claim, exact passage, scoped method and executed check; a local visitor can approve that exact revision with a reservation. Neither an anchor match nor approval establishes real-world truth. [5]