Explaining Agent Decisions: A Study on Natural Language Explanations

This article presents a novel approach to explain complex policy decisions made by artificial intelligence (AI) systems using natural language explanations. The proposed method, called BR (Path), leverages decision trees and large language models (LLMs) to generate easily understandable explanations for laypeople. BR (Path) first distills the black-box policy into a decision tree, then extracts a decision path from the tree for a given state, which contains a set of decision rules used to derive the associated action. Finally, an LLM is utilized to generate a natural language explanation based on the decision path.
The authors conduct participant studies to evaluate the effectiveness of BR (Path) and compare it to other approaches. The results show that participants significantly prefer explanations generated by BR (Path) over those produced by either human domain experts or Template, which generates explanations based on states. The study demonstrates that using natural language explanations can increase trust under uncertainty and help laypeople understand complex policy decisions made by AI systems.
The authors highlight the importance of avoiding hallucinated facts in explanations, as laypeople may struggle to identify them. They also emphasize that BR (Path) provides an upper-bound on explanation quality, while human explanations serve as a lower-bound. The proposed method has significant implications for improving transparency and trustworthiness in AI systems, especially in high-stakes decision-making scenarios.
In summary, the article presents a novel approach to generating natural language explanations for complex policy decisions made by AI systems. The proposed method leverages decision trees and LLMs to generate easily understandable explanations that are both accurate and informative. The study demonstrates the effectiveness of BR (Path) in increasing trust under uncertainty and improving comprehension of complex policy decisions made by AI systems.