Authors - Zubaria Inayat, Hanin Esawi, Maya Daneva, Marten van Sinderen, Giancarlo Guizzardi, Luiz Bonino da Silva Santos Abstract - Explainable artificial intelligence (XAI) is becoming an important component of healthcare systems, supporting transparent and trustworthy AI-assisted decision-making. However, existing explainable AI approaches are mainly designed around healthcare professionals, while patient needs and expectations regarding AI-generated explanations remain insufficiently explored. This study investigates how patients perceive the quality of explanations provided by XAI healthcare systems and identifies the challenges that influence their understanding, trust, and engagement. A mixed-method research approach was adopted, combining (i) a rapid literature review, (ii) an online survey with 33 participants, and (iii) expert consultation for validation. The literature review identified nine quality dimensions for patient-oriented XAI explanations. The survey findings revealed eight challenges experienced by patients when interacting with explainable AI systems. Based on the synthesis of these findings, a patient-centred evaluation matrix was developed, linking explanation quality dimensions with patient-related challenges. The proposed matrix was validated through expert feedback. The results highlight the importance of moving beyond developer and clinician-centric XAI design towards patient-centred explainable healthcare systems. This work contributes to trustworthy artificial intelligence in healthcare by providing guidance for evaluating explanation quality, improving usability, supporting patient trust, and enabling informed shared decision-making.
