A Diagnosing Untruthfulness: A G-Eval and Bootstrap Analysis of LLM Failure Modes on TruthfulQA

Abstract

While large language models (LLMs) often fail on adversarial questions, the specific nature of these failures is not well understood, hindering efforts to improve their reliability. This study moves beyond simple accuracy metrics to systematically diagnose the error patterns of GPT-3.5-turbo and GPT-4 on the challenging TruthfulQA benchmark. To do this, it created an "error corpus" by collecting incorrect responses from both models. A more advanced model, gpt-4o, was then employed as a judge within a G-Eval framework to classify each error into a fine-grained taxonomy of predefined categories. The statistical robustness of the resulting error distributions was confirmed via a 5,000-iteration bootstrap simulation. The analysis reveals that both models exhibit stable and dominant failure modes, with the repetition of common human misconceptions being the primary cause of untruthfulness. These findings provide a granular, validated "pathology report" of LLM errors, offering crucial insights for future model alignment and targeted safety interventions.