Where language-model analysis needs external structure

Start with a question that spans documents

“Which commitments remained in force when the talks resumed, and which sources disagree?” is not a request for a single summary. It requires finding relevant statements, resolving references, distinguishing proposals from accepted obligations, applying dates and conditions, and preserving disagreement. A fluent answer can conceal a mistake at any one of those steps.

There is no need to claim that a transformer cannot represent time, cite a source, or reason about causes. Those are empirical questions about particular systems and tasks. For the engineering problem here, a narrower observation is enough: an answer should leave the evidence and intermediate interpretations accessible, so a reviewer can identify where it went wrong.

Retrieval is a real intervention

Lewis and colleagues’ original RAG work combined a language generator with a retrieved external memory and reported improvements on knowledge-intensive tasks. It is direct evidence against the assertion that retrieval merely rearranges the problem without helping. Its results concern the models, corpora, and evaluation tasks in that study, not every retrieval system built since. [1]

Retrieval still introduces its own failure modes. A relevant amendment may not be retrieved; two retrieved passages may use the same name for different entities; a high-ranked summary may omit a condition. Before adding a graph, test simpler changes such as document version filters, better chunk boundaries, hybrid search, and explicit source passages in the answer context.

A larger context window is capacity, not a result

Lost in the Middle examined multi-document question answering and key-value retrieval. The tested models’ performance varied with the position of relevant information, often declining when it appeared in the middle of a long context. That supports testing placement and distractors; it is not a theorem that current or future long-context models must fail. [2]

For a one-off question over a modest file, giving a capable model the complete documents may be the clearest solution. For repeated questions over changing files, selecting a compact evidence set may be more practical. The selection must retain exceptions, dates, and contrary accounts. A shorter context that removes the clause which changes the answer is cheaper only in a misleading sense.

What structure can make explicit

A graph is one way to represent the relationships; SQL tables and explicit joins are another. Neither makes an extracted causal link true. If the evidence says an event preceded a decision, the record should not silently promote that order into a claim that it caused the decision. Preserve the proposed mechanism and its supporting evidence as an assessment to review.

A citation is an evaluation target

ALCE evaluates systems that retrieve evidence and generate cited answers, separating fluency, correctness, and citation quality. Its experiments showed that coherent answers could still have incomplete citation support. This is an observed weakness of the evaluated systems, alongside evidence that models can generate citations—not an architectural prohibition on doing so. [3]

For TACITUS, the intended boundary is a reviewable proposal: DIALECTICA organizes source and knowledge records; PRAXIS provides the workspace where a person examines the output. That architecture remains under development. Its value depends on whether it helps the reviewer detect unsupported inferences, not on whether its graph looks detailed.

Compare complete workflows, not favorable screenshots

A useful study would compare full-document prompting, a strong retrieval baseline, structured tables, and graph-assisted context on the same held-out questions. Fix or report the model, source access, output requirements, and budgets. Measure supported claims, missed qualifications, temporal mistakes, appropriate abstention, and human correction time.

Report both per-query tokens and the cost of extraction, indexing, updates, retries, and evaluation. Plot quality against total budget rather than assuming compression is beneficial. NIST’s ARIA manual adds the necessary human dimension: model testing, adversarial testing, and user testing answer different questions about an application. These are proposed comparisons; we do not yet claim a measured TACITUS advantage. [4]

Test a specific error against a fair baseline

For the reporting case, count wrong actor, date or modality; missing optionality and funding caveats; source/citation entailment; and reviewer time. Compare equivalent sources with direct context, retrieval and structured packets. Include setup, licensing, upkeep and a second analyst’s next assignment. This synthetic teaching fixture demonstrates a distinction, not categorical model inferiority or a measured TACITUS advantage. [5]