TL;DR
- 01Availability, model quality, integration, and operational reliability are different claims and need different evidence.
- 02A reference artifact or passing local test cannot establish a complete deployed user journey.
- 03Progress should be reported per version and task, including uncertainty, cost, failures, and the limits of human review.
A development label is useful only when it points to a concrete boundary: what a version does, where the evidence comes from, and which claims remain untested.
Revision: Rewritten to replace unsupported quality claims and a fixed pilot-count launch rule with scoped evidence, workflow evaluation, failure reporting, and release criteria. Added the scoped date the claim and state what the check covered example and its evidence boundary.
The label needs a referent
“Experimental” can describe a format, an interface, a model-assisted extraction step, or an entire product. Those are different boundaries. A reader should not have to infer that a downloadable capsule proves a hosted editor exists, or that a working chat demonstrates a reliable delegated research workflow.
For TACITUS, PRAXIS is the analyst’s workspace and DIALECTICA the knowledge engine behind the broader capsule design. Their development status should be stated at the relevant workflow. This note defines the evidence we want release claims to carry; it is not a substitute for the current product capability page or a claim that the criteria below have been met.
Keep four kinds of evidence separate
Model Cards proposed documenting intended use, evaluation procedures, performance characteristics, and relevant limitations alongside a model. A company website describes more than a model, but the reporting principle transfers: make each statement specific enough that a reader can inspect what supports it. [1]
| Claim | Useful evidence | Insufficient on its own |
|---|---|---|
| A format is inspectable. | Versioned files, schema, source references, and documented examples. | A diagram of the intended architecture. |
| A workflow functions. | The deployed version and a complete user journey, including persistence and recovery. | A local demo or a healthy server endpoint. |
| An output is useful. | Representative tasks, supported answers, reviewed errors, and stated limitations. | A plausible answer or selected successful example. |
| A service is dependable. | Observed failure behavior, operational measurements, and recovery procedures. | One successful run or the absence of reported incidents. |
Test the system people actually use
A capsule workflow crosses several boundaries: upload, extraction, review, attachment, retrieval, answer generation, and later reuse. An answer can look correct while its source link is broken, its document was omitted, or its review decision was not saved. An extraction benchmark will not discover every failure in that journey.
NIST’s September 2026 ARIA manual combines model testing, red teaming, and user testing within a defined context of use. Its value here is the insistence on specifying the application, task, and characteristic being measured. We should test a policy-brief workflow with appropriate sources and reviewers, not infer its quality from an unrelated benchmark. [2]
Human review is a hypothesis to test, too
Placing an approval button after a model output does not establish meaningful oversight. Reviewers may lack access to the evidence, misunderstand a status label, or accept a confident answer because checking it takes too long. A realistic evaluation should include planted errors, omitted qualifications, and difficult disagreements, then observe which problems users identify and correct.
Automated checks help with a different part of the problem. NIST’s experimental citation probes assess faithfulness, completeness, and sufficiency against reference material. They provide a useful basis for structured checks, but their verdicts also need evaluation. A second model agreeing with the first is not independent proof that an answer is sound. [3]
Efficiency belongs beside quality
A structured context package may reduce the material sent into repeated queries. The relevant accounting includes the initial extraction, indexing, corrections, model calls, retrieval, retries, and updates. A compact prompt can save tokens at query time while making the overall workflow more expensive or less complete.
Report quality against total budget across a defined workload. Include citation support, important omissions, appropriate uncertainty, reviewer effort, and latency. Compare with a well-configured document workflow. A cost reduction matters only if the comparison preserves the evidence required by the task; no measured TACITUS token-saving claim is made here.
A release decision should name what changed
There is no universal number of pilots that turns an experimental system into a dependable service. The proposed release criteria should be scoped to an intended use, established before evaluation, and revisited when the model, data, or workflow changes. Record known failures, recovery behavior, and which checks can be repeated on the deployed version.
The NIST AI Risk Management Framework is voluntary guidance for managing risk across development and use, not a certification obtained by citing it. Our corresponding commitment is practical: publish the boundary, report evidence honestly, and keep unsupported capabilities labeled as development work. A label should come off only when a narrower, substantiated description can replace it. [4]
Date the claim and state what the check covered
The real PRAXIS tour records three service captures, five chapters, 30 September 2026 and source build 4480650. It is evidence of bounded operations with synthetic material, not a recording, general extraction accuracy or a customer outcome. The local briefing runs separate authored checks. Customer comprehension, independent domain quality, repeat usage and net savings remain unmeasured. An experimental label cannot replace these specific evidence boundaries or conceal a failed engineering gate. [5]
SOURCES
- [1]Mitchell et al. Model Cards for Model Reporting (FAT* 2019).Documentation of intended use, evaluation procedures, and performance characteristics.
- [2]NIST. ARIA Evaluation Planning Manual (18 September 2026).Combining model, red-team, and user testing within an explicit evaluation scope.
- [3]NIST. Building Evaluation Probes into Agentic AI (updated May 2026).Experimental citation probes and structured evaluation records.
- [4]NIST. Artificial Intelligence Risk Management Framework 1.0 (26 January 2023).Voluntary risk-management guidance, not a TACITUS certification.
- [5]Worked example / implementation boundaryAuthored website demonstration or reference explanation. Not independent analytical validation or production integration proof.
