Measuring and improving chain-of-thought faithfulness in language models

Language models increasingly use chain-of-thought (CoT) reasoning to explain their outputs, but these explanations may not faithfully represent the model's actual reasoning process. A model might provide plausible-sounding reasoning that doesn't reflect the true computational path to its answer, creating risks for AI safety: we might trust model explanations that are misleading, fail to detect problematic reasoning, or incorrectly assume we understand model behavior.

This problem asks researchers to investigate when and why CoT explanations are faithful or unfaithful, and to develop better methods for measuring and improving faithfulness. Recent work has shown that models can produce post-hoc rationalizations rather than genuine reasoning traces (Lanham et al., 2023), and that CoT faithfulness varies significantly across contexts (Mills et al., 2024).

Key research questions include:

Task-dependent faithfulness: Does CoT faithfulness vary systematically across different domains? For instance, are mathematical reasoning chains more faithful than those for social reasoning or moral judgment? If a model performs significantly better with CoT than without, does this performance boost indicate the CoT is actually being used (and thus faithful), or could the model be leveraging CoT in ways that don't match the stated reasoning?

Situational awareness effects: Can models' knowledge about their evaluation context affect CoT faithfulness? If a model knows its reasoning is being monitored for faithfulness, does it produce more faithful explanations, less faithful ones (perhaps trying to game the evaluation), or explanations that superficially appear more faithful without actually being so?

Detection methods: Existing automatic methods for detecting unfaithful CoT (Lanham et al., 2023; Mills et al., 2024; Chen et al., 2024) often have limited recall or require significant computational resources. Can we develop more efficient detection methods with higher recall? What are the fundamental limits of automated faithfulness detection?

Ensuring faithfulness: Beyond detection, can we develop training methods or architectural changes that make models produce more faithful CoT by default? Work by Radhakrishnan et al. (2023), Chen et al. (2024), and Chua et al. (2024) has explored this direction, but much remains unknown about what training objectives or model designs best promote faithfulness.

Successful solutions could involve empirical studies characterizing faithfulness across different settings, new benchmarks for measuring faithfulness, improved automated detection methods, training techniques that increase faithfulness, or theoretical frameworks for understanding when and why models produce faithful versus unfaithful reasoning chains. This work is critical for interpretability, model evaluation, and ensuring we can trust model explanations in high-stakes applications.