Investigate Informative Self-Reports

Background: In user studies, self-reported subjective ratings are often collected alongside objective performance measures to assess human understanding derived from AI explanations.

Question / Future Work: Systematically investigate the conditions under which self-reported explanation quality ratings reliably track demonstrated user understanding (e.g., forward simulation accuracy), especially concerning how explanation correctness moderates this link, given that self-reports only correlated with performance when explanations were fully correct and the participant had learned the pattern. This is necessary to determine if subjective measures can be generally informative for evaluating XAI systems.

Why It Matters: Understanding when subjective self-reports align with objective performance is a major unresolved issue in human-centered XAI evaluation, as current findings suggest they are only reliable under specific, favorable conditions.

Evidence: A related question is whether explanation correctness moderates the link between self-reported and demonstrated understanding, as self-reports tracked performance only when explanations were fully correct and the participant had learned the decision pattern. If this pattern replicates beyond our specific task, the disconnect may be partly context-dependent rather than a fixed limitation of subjective measures, and the conditions under which self-reports are informative deserve systematic investigation.