Capturing expert methods without claiming to capture the expert

The difficult part is often the exception

A policy analyst reviewing a fictional implementation brief notices that two apparently independent reports use the same unusual phrase. She pauses before calling the finding corroborated. The phrase may indicate a shared source, but it may also be standard language from a regulation. Her useful contribution is not merely a rule to distrust repeated wording. It is knowing which follow-up question to ask, what alternative explanation to consider, and when the answer matters.

Polanyi’s account of tacit knowing cautions against assuming that everything a person knows can be stated completely. It is a philosophical starting point, not evidence that a particular interview technique will capture expertise. For TACITUS, the narrower engineering question is whether selected methods can be documented well enough to help another person perform a task and recognize the limits of the guidance. [1]

Interview the decision, not the résumé

Applied Cognitive Task Analysis, developed by Militello and Hutton, provides a practical precedent. Its task diagram identifies demanding parts of the job; its knowledge audit probes expertise through examples; its simulation interview examines decisions in a scenario. The authors also describe a cognitive demands table for consolidating the findings. Their evaluation concerned the usefulness of these methods and resulting materials, not the performance of an AI agent using a capsule. [2]

We would adapt that structure for a proposed Tool capsule. First, ask the analyst to outline the task. Next, choose the moment when she hesitated. Reconstruct the information available then, rather than allowing later knowledge to fill the gaps. Ask what a less experienced analyst might miss and what would make her choose a different course. A model could suggest candidate records from the interview, but the practitioner would need to correct them.

A method needs an applicability statement

The proposed capsule would contain the task, required inputs, decision cues, suggested sequence, counterexamples, unresolved disagreements, and references to the cases used to develop it. It should distinguish the practitioner’s own advice from an interviewer’s interpretation. PROV’s separation of entities, activities, and responsible agents offers a vocabulary for those attributions; it does not establish that the resulting guidance is effective. [3]

An important field is the stopping condition. For this example, the method might stop at “possible source dependence; investigate further.” It should not quietly become a classifier that declares manipulation. Another field would identify prerequisites: access to full documents, relevant language expertise, and permission to inspect any restricted material. Missing prerequisites should change what the consumer is allowed to conclude.

Attribution also needs consent. A practitioner may approve an internal teaching example without agreeing to public distribution, automated application, or an endorsement of every later output. We would record those permissions separately from the method’s author and review status. We are not claiming that a metadata field alone creates or enforces the relevant legal rights.

Test whether another person can use it

A convincing evaluation would hold back cases from the elicitation session and ask other practitioners to work through them. Compare the capsule with ordinary written guidance and access to the original example documents. Assess whether users notice relevant cues, distinguish alternative explanations, avoid overapplying the method, and recognize when they need help. A test that merely asks them to repeat the recorded checklist would miss the main question.

An AI-assisted version adds another comparison: does the model apply the method appropriately, or imitate its vocabulary while missing its conditions? NIST’s ARIA approach combines model testing, red teaming, and user testing; that is a useful evaluation structure for separating those questions. It supplies a planning method, not evidence that our proposed transfer mechanism works. [4]

DIALECTICA would organize the candidate method and its review history; PRAXIS would make it available in the context of an analytical task. Both the elicitation workflow and reliable reuse remain under development. The intended result is a small, inspectable piece of guidance that survives a new case, rather than a claim that an expert has been converted into software.

Elicit a decision cue that can change the next draft

Ask a practitioner what task they were doing, for whom, what a newcomer would miss, what they checked next, who had authority, and what would change their view. Ask when the rule fails, request acceptable and unacceptable examples, and separate the parts that cannot be reliably automated. Also ask who owns the material and what may be shared. No interview or expert endorsement was performed for this website fixture.

The authored legislative method says: preserve proposal modality, identify authority and ask for the adoption instrument. Its counterexample is an adopted July date with a mandatory funded pilot. The browser checker uses those specific conditions; it is not an exhaustive transfer of expertise or a universal procedural model. [5]