Enhancing Trustworthiness of Retrieval-Augmented Large Language Models via Confidence Calibration and Selective Rejection
Article Sidebar
Main Article Content
Abstract
This study proposes a robust monitoring method based on structure-aware feature learning to address the challenges of complex multidimensional feature coupling, dynamically changing dependencies, and diverse anomaly patterns in backend systems. The method builds upon multi-source monitoring data, integrating temporal dynamic features and topological dependency structures to achieve unified system state representation across different time scales and structural hierarchies. First, a feature encoder is employed to extract temporal features and construct a dynamic dependency graph that captures latent structural relationships among service nodes. Second, a structure- aware mechanism and graph propagation layer are introduced to perform cross-node feature fusion and apply dependency consistency constraints, enhancing the model's stability and generalization capability in complex topological environments. Finally, variational regularization and a robust optimization objective are applied to further improve anomaly detection reliability under high noise and non-stationary distributions. Experimental results show that the proposed method outperforms existing models across multiple key metrics, including F1-Score, AUROC, and Robust Recall, while maintaining stable monitoring performance under structural perturbations, workload fluctuations, and feature heterogeneity. These results validate the effectiveness of structure-aware feature learning in complex system monitoring tasks, demonstrating its ability to accurately model anomaly propagation paths and achieve system-level semantic recognition in the feature space, thereby providing strong support for building highly reliable and interpretable backend monitoring frameworks.
Article Details
Issue
Section
References
[1]Asai A, Wu Z, Wang Y, et al. Self-rag: Learning to retrieve, generate, and critique through self-reflection[C]//The Twelfth International Conference on Learning Representations. 2023.
[2]Yoran O, Wolfson T, Ram O, et al. Making retrieval-augmented language models robust to irrelevant context[J]. arXiv preprint arXiv:2310.01558, 2023.
[3]Joren H, Zhang J, Ferng C S, et al. Sufficient context: A new lens on retrieval augmented generation systems[J]. arXiv preprint arXiv:2411.06037, 2024.
[4]Lee M, Kim K, Kim T, et al. Selective generation for controllable language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 50494-50527.
[5]Zablotskaia P, Phan D, Maynez J, et al. On uncertainty calibration and selective generation in probabilistic neural summarization: A benchmark study[C]//Findings of the Association for Computational Linguistics: EMNLP 2023. 2023: 2980-2992.
[6]P. Rajpurkar, R. Jia and P. Liang, "Know What You Don’t Know: Unanswerable Questions for SQuAD," Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
[7]Liu L, Liu X, Wong D F, et al. Selectit: Selective instruction tuning for llms via uncertainty-aware self-reflection[J]. Advances in Neural Information Processing Systems, 2024, 37: 97800-97825.
[8]Y. Li, "Task-Aware Differential Privacy and Modular Structural Perturbation for Secure Fine-Tuning of Large Language Models," 2024.
[9]P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal and D. Kiela, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," Advances in Neural Information Processing Systems, vol. 33, pp. 9459-9474, 2020.
[10]G. Izacard and E. Grave, "Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering," Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 874-880, 2021.
[11]V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov and W. T. Yih, "Dense Passage Retrieval for Open-Domain Question Answering," Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
[12]Q. Gan, "Large Language Model Framework for Multi-Document Financial Anomaly Detection in Intelligent Auditing via Semantic Mapping and Risk Reasoning," 2024.
[13]C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, "On Calibration of Modern Neural Networks," Proceedings of the International Conference on Machine Learning, pp. 1321-1330, 2017.
[14]S. Desai and G. Durrett, "Calibration of Pre-Trained Transformers," Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 295-302, 2020.
[15]Y. Deng, "Transfer Methods for Large Language Models in Low-Resource Text Generation Tasks," 2024.
[16]H. Liu, "Structural Regularization and Bias Mitigation in Low-Rank Fine-Tuning of LLMs," 2023.
[17]H. Jiang, B. Kim, M. Guan and M. Gupta, "To Trust or Not to Trust a Classifier," Advances in Neural Information Processing Systems, vol. 31, 2018.
[18]Y. Geifman and R. El-Yaniv, "Selective Classification for Deep Neural Networks," Advances in Neural Information Processing Systems, vol. 30, 2017.
[19]A. Kamath, R. Jia and P. Liang, "Selective Question Answering under Domain Shift," Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696, 2020.
[20]J. Li, "LocateNet: Large Multimodal Models for Text-Guided Object Localization," 2024.
[21]Y. Hu, "Autonomous Agent Architecture for Complex Tasks via Hierarchical Planning and Language Model Reasoning," 2024.
[22]C. Hua, "A Semantic-Prior-Guided AI Framework for Collaborative Environment Understanding and Robust Agent Decision Making," 2024.
[23]Chen L, Zhang R, Guo J, et al. Controlling risk of retrieval-augmented generation: a counterfactual prompting framework[C]//Findings of the Association for Computational Linguistics: EMNLP 2024. 2024: 2380-2393.
[24]Abdumalikov R, Minervini P, Kementchedjhieva Y. Answerability in Retrieval-Augmented Open-Domain Question Answering[J]. arXiv preprint arXiv:2403.01461, 2024.
[25]J. Zhang, Y. Zhao, M. Saleh and P. Liu, "PEGASUS: Pre-Training with Extracted Gap-Sentences for Abstractive Summarization," Proceedings of the International Conference on Machine Learning, pp. 11328-11339, 2020.
[26]Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov and C. D. Manning, "HotpotQA: A Dataset for Diverse, Explainable Multi-Hop Question Answering," Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369-2380, 2018.
[27]J. Thorne, A. Vlachos, C. Christodoulopoulos and A. Mittal, "FEVER: A Large-Scale Dataset for Fact Extraction and VERification," Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809-819, 2018.
[28]Jang C, Cho D, Lee S, et al. Reliable Decision Making via Calibration Oriented Retrieval Augmented Generation[J]. arXiv preprint arXiv:2411.08891, 2024.
[29]R. Xiong, Y. Chen, L. Pang, X. Cheng, Z. M. Ma and Y. Lan, "Uncertainty Calibration for Ensemble-Based Debiasing Methods," Advances in Neural Information Processing Systems, vol. 34, pp. 13657-13669, 2021.