The Explainability principle states that medical AI tools should provide clinically meaningful information about the logic behind the AI decisions. While medicine is a high-stake discipline that requires transparency, reliability and accountability, machine learning techniques often produce complex models which are black box in nature. Explainability is considered desirable from a technological, medical, ethical, legal as well as patient perspective. Explainability is a complex task which has challenges that need to be carefully addressed during AI development and evaluation to ensure that AI explanations are clinically meaningful and beneficial to the end-users.
To this end, two recommendations for Explainability are defined in the FUTURE-AI framework. At the design phase, it should be first established with end-users and domain experts whether explainable AI is needed for the medical AI tool in questions. In this case, the specific goal and approaches for explainability should be defined (Explainability 1). After their implementation, the selected approaches for explainability should be evaluated, both quantitatively using in silico methods, as well qualitatively with end-users to assess their impact on the user’s satisfaction and performance (Explainability 2).
| Recommendation | Description |
| Explainability 1
Define explainability needs |
At the design phase, it should be established if explainability is required for the AI tool. In this case, the specific requirements for explainability should be defined with representative experts and end-users, including (i) the goal of the explanations (e.g. global description of the model’s behaviour vs. local explanation of each AI decision), (ii) the most suitable approach for AI explainability, and (iii) the potential limitations to anticipate and monitor (e.g. over-reliance of the end-users on the AI decision). |
| Explainability 2
Evaluate explainability |
The explainable AI methods should be evaluated, first quantitatively by using in silico methods to assess the correctness of the explanations, then qualitatively with end-users to assess their impact on user satisfaction, confidence and clinical performance. The evaluations should also identify any limitations of the AI explanations (e.g. they are clinically incoherent or sensitive to noise or adversarial attacks, they unreasonably increase the confidence in the AI-generated results). |