LLM-Compiled Wikis for Cross-Institutional Policy Retrieval: A Benchmark on Chinese University Grant Policies
DOI:
https://doi.org/10.62411/faith.3048-3719-377Keywords:
BM25, Chinese NLP, Citation Faithfulness, Cross-Document Comparison, Knowledge Compilation, Large Language Models, Policy Documents, Retrieval-Augmented GenerationAbstract
We investigate whether pre-compiling a document corpus into an LLM-compiled wiki can improve cross-institutional policy-conflict queries that naturally exhibit a tabular comparison structure. We define a task (T3a) in which a system must identify, for a focal institution and policy dimension, both the focal rule and the institutions that disagree with it, supported by article-level citations. We construct a 28-query benchmark from 22 Chinese university grant policy documents across 19 institutions. Gold citations are extracted automatically using an LLM-based pipeline with mechanically auditable post-checks and validated through a human audit of six representative queries (172 gold citations audited, with 93.0% manually verified as correct or partially supporting; see Section 4.2.1). We compare three systems that share the same language-model generator: no-retrieval, BM25 retrieval-augmented generation, and a wiki-based retrieval system whose pages are compiled offline from the same corpus. The wiki system achieves a citation recall of 0.242 ± 0.21, compared with 0.095 ± 0.12 for BM25 and 0.023 ± 0.03 for no-retrieval; all pairwise differences are statistically significant (p < 10⁻³, paired Wilcoxon test). Crucially, the wiki system achieves this performance while consuming only approximately 29% of BM25's input-context tokens, indicating that the observed improvement is not driven by larger context volume. A failure-mode analysis shows that hallucinated citations decrease monotonically across the three systems (75% → 54% → 39%). We further identify two LLM-induced failure modes in the compiled wiki that are not detected by citation-level validation. We frame these findings as a design insight into task-representation alignment for cross-document comparison queries—a proof-of-concept for compile-then-query arrangements within this task family rather than a general claim of superiority over modern retrieval systems
Downloads
References
P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks,” arXiv. Apr. 12, 2021. [Online]. Available: http://arxiv.org/abs/2005.11401
Y. Gao et al., “Retrieval-Augmented Generation for Large Language Models: A Survey,” arXiv. Mar. 27, 2024. doi: 10.48550/arXiv.2312.10997.
P. Sharma and S. Bhattarai, “A Review on Retrieval-Augmented Generation: Architectures, Research Challenges, and Emerging Frontiers,” J. Futur. Artif. Intell. Technol., vol. 2, no. 4, pp. 616–628, Jan. 2026, doi: 10.62411/faith.3048-3719-297.
A. Karpathy, “LLM Wiki,” GitHub Gist. Apr. 2026. [Online]. Available: https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” arXiv. Dec. 24, 2023. [Online]. Available: http://arxiv.org/abs/2306.05685
A. Vaswani et al., “Attention Is All You Need,” in 31st Conference on Neural Information Processing Systems (NIPS 2017), 2017. [Online]. Available: http://arxiv.org/abs/1706.03762
J. Devlin, M.-W. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North, Oct. 2019, pp. 4171–4186. doi: 10.18653/v1/N19-1423.
C. Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” arXiv, Sep. 2023, [Online]. Available: http://arxiv.org/abs/1910.10683
T. B. Brown et al., “Language Models are Few-Shot Learners,” in arXiv, Jul. 2020. [Online]. Available: http://arxiv.org/abs/2005.14165
OpenAI et al., “GPT-4 Technical Report,” ArXiv. Mar. 04, 2024. [Online]. Available: http://arxiv.org/abs/2303.08774
H. Touvron et al., “Llama 2: Open Foundation and Fine-Tuned Chat Models,” arXiv. Jul. 19, 2023. [Online]. Available: http://arxiv.org/abs/2307.09288
S. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Found. Trends® Inf. Retr., vol. 4, no. 1–2, pp. 1–174, Sep. 2009, doi: 10.1561/1500000019.
J. Menick et al., “Teaching language models to support answers with verified quotes,” arXiv. Mar. 21, 2022. [Online]. Available: http://arxiv.org/abs/2203.11147
N. F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts,” Trans. Assoc. Comput. Linguist., vol. 12, pp. 157–173, Feb. 2024, doi: 10.1162/tacl_a_00638.
T. Gao, H. Yen, J. Yu, and D. Chen, “Enabling Large Language Models to Generate Text with Citations,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 6465–6488. doi: 10.18653/v1/2023.emnlp-main.398.
A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection,” in arXiv, Oct. 2023. [Online]. Available: http://arxiv.org/abs/2310.11511
O. Ram et al., “In-Context Retrieval-Augmented Language Models,” Trans. Assoc. Comput. Linguist., vol. 11, pp. 1316–1331, Nov. 2023, doi: 10.1162/tacl_a_00605.
D. Edge et al., “From Local to Global: A Graph RAG Approach to Query-Focused Summarization,” arXiv. Feb. 19, 2025. [Online]. Available: http://arxiv.org/abs/2404.16130
I. Chalkidis, M. Fergadiotis, P. Malakasiotis, N. Aletras, and I. Androutsopoulos, “LEGAL-BERT: The Muppets straight out of Law School,” in Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 2898–2904. doi: 10.18653/v1/2020.findings-emnlp.261.
N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych, “BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models,” arXiv. Oct. 21, 2021. [Online]. Available: http://arxiv.org/abs/2104.08663
Y. Cui, W. Che, T. Liu, B. Qin, and Z. Yang, “Pre-Training With Whole Word Masking for Chinese BERT,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 29, pp. 3504–3514, 2021, doi: 10.1109/TASLP.2021.3124365.
V. Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 6769–6781. doi: 10.18653/v1/2020.emnlp-main.550.
O. Khattab and M. Zaharia, “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT,” in Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul. 2020, pp. 39–48. doi: 10.1145/3397271.3401075.
J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu, “M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation,” in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 2318–2335. doi: 10.18653/v1/2024.findings-acl.137.
K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang, “REALM: Retrieval-Augmented Language Model Pre-Training,” in ICML’20: Proceedings of the 37th International Conference on Machine Learning, Feb. 2020. [Online]. Available: http://arxiv.org/abs/2002.08909
Z. Shao, Y. Gong, Y. Shen, M. Huang, N. Duan, and W. Chen, “Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 9248–9274. doi: 10.18653/v1/2023.findings-emnlp.620.
Y. Zhu et al., “Large Language Models for Information Retrieval: A Survey,” arXiv. Sep. 17, 2025. doi: 10.1145/3748304.
X. Ma, L. Wang, N. Yang, F. Wei, and J. Lin, “Fine-Tuning LLaMA for Multi-Stage Text Retrieval,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Jul. 2024, pp. 2421–2425. doi: 10.1145/3626772.3657951.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Bailing Zhang
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
