TCGC: test the structure, then test what it helps people do.

The repository defines fourteen task categories. They are a useful starting point for evaluating a structured representation, not evidence that every category already has enough independently annotated examples for a reliable estimate.

Schema-extension induction, invariant checking, and cross-domain transfer are additional research directions. Before scoring them, we need published task definitions, annotation guidance, split construction, and enough examples to expose where a shared vocabulary fails.

The proposed experiment uses the same case snapshots, questions, model versions, answer instructions, and evidence requirements across conditions. It compares systems that attempt the same task; it does not compare a deliberately vague paragraph with a hand-authored gold graph.

1. Freeze a corpus snapshot and an annotation guide. Split by case or source family so near-duplicates do not cross into the test set.
2. Keep development questions separate from held-out evaluation. Tune budgets and retrieval settings on development data only.
3. Run each condition on identical questions. Save the assembled context and raw outputs, including failures and retries.
4. Score extraction, evidence selection, and final answers separately. Have blinded reviewers assess interpretive claims and citation support.
5. Report paired differences with uncertainty estimates grouped by case. Record repeated-run variation and avoid treating many correlated questions as independent evidence.
6. Ablate typed labels, time fields, graph links, compression, and human review to identify what changes the result.

The 2024 RAG/long-context comparison motivates a serious direct-reading baseline. GraphRAG's documented query modes show why “graph retrieval” also needs a precise configuration rather than a single generic label.

Use the scorer implementation at the pinned repository commit for exact formulas. Do not infer calibration from a score between zero and one. An LLM judge adjusted against a small anchor set still requires independent evaluation on unseen cases.

Report abstention and missing outputs explicitly. A system that avoids hard questions can appear precise while being unhelpful; a system that answers every question can conceal unsupported guesses. Performance should be broken down by task, domain, evidence availability, and question difficulty.

A graph or capsule may reduce repeated reading across a sequence of questions. It also incurs extraction, indexing, storage, maintenance, and review costs. A fair evaluation should show both the cost of a first question and the cost after repeated use of the same case.

Run a quality-versus-budget curve, not only one compression setting. In one proposed sweep, compare four input budgets at 2,000, 4,000, 8,000, and 16,000 tokens where the chosen models support them. Use the same tokenizer and reserve answer space consistently. These are experimental settings, not measured capsule sizes or claimed savings.

SARA (2026) and the Graph-RAG reasoning preprint (2026) study different compression mechanisms. Include their relevant training, preprocessing, and routing costs if reproducing them. If a compressed condition loses the decisive exception, score that as a quality failure even if it is cheaper.

These are proposed synthetic test designs, not completed benchmark runs. Real cases should add domain review, source permissions, ambiguous examples, and alternative defensible interpretations.

The repository includes manifests for public-domain works such as the Federalist Papers, the Melian Dialogue, Leviathan, The Prince, The Art of War, and Caesar's Gallic War. They offer passages that readers can inspect and discuss. Editions and translations still matter, and public-domain status depends on jurisdiction and the particular edition.

Well-known material may occur in model training data. Familiarity does not disappear because the task asks for a graph. Results need contamination checks and follow-up evaluation on held-out contemporary material, with lawful access and independent annotations.

The benchmark's own ontology can also bias its evaluation. Include multiple defensible annotations, document disagreements, and test downstream questions that do not reward merely reproducing our preferred labels. Broader policy or commercial usefulness must be evaluated with the people who do that work.

The public repository is the source of truth for executable commands and available fixtures. Before a run, record its commit, install from its instructions, validate the supplied sample items, and inspect the exact scoring functions. The experiment protocol above is a proposed extension of that starting point.

A useful release includes the corpus and question versions, split manifest, model settings, prompts, index configuration, raw predictions, assembled context, scorer version, annotation rubric, costs, and failure analysis. Where source material is private, explain the reproducibility limit and publish an authorized test fixture.

We welcome critical reproduction and domain expertise. A result showing that a simpler baseline is equally good, cheaper, or easier to review would be valuable evidence about where this approach belongs.

Primary sources and the TCGC README were reviewed September 29, 2026. No results from these studies are presented as TACITUS performance.

1. GraphRAG query engine

   Microsoft · official documentation · Reviewed September 29, 2026

   Distinguishes local, global, DRIFT, and basic search; local search combines graph records and original text. Implementation documentation is not an independent performance comparison.

2. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

   Li et al. · EMNLP Industry Track · 2024

   Finds stronger average long-context performance when sufficiently resourced in its tested setup, alongside cost advantages for retrieval and a hybrid routing approach. The tested models and workloads do not determine the best choice for current deployments.

3. SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression

   Jin et al. · ACL long paper · July 2026

   Combines retained natural-language passages with learned compressed representations and evidence reranking under token budgets. The approach includes model training and mixed representations; it is not a drop-in JSON compression result.

4. The Reasoning Bottleneck in Graph-RAG: Structured Prompting and Context Compression for Multi-Hop QA

   Zarrinkia et al. · preprint · March 18, 2026 · version 2

   Studies structured prompting and graph-walk compression on KET-RAG and LightRAG across three multi-hop QA benchmarks. Compression helps some configurations and hurts others. The results are not independently reproduced by TACITUS.