In oncology AI, explainability is often discussed as if a highlighted region or feature attribution were sufficient. It is not. A useful explanation must help the intended user understand what the system was designed to do, what information it used, how the output should be interpreted, and where the evidence does not support a conclusion.
Define the audience and decision
A pathologist, laboratory manager, patient, regulator, and researcher need different information. Start by identifying the decision the output may inform and the person responsible for that decision. Explainability should support professional review and accountability, not encourage a user to accept an output without examining the case.
Separate logic from evidence
A heatmap, saliency map, counterfactual, or feature list is a representation of model behavior. It is not automatically proof that the highlighted morphology caused the output, nor proof that the output is clinically correct. Pair any explanation with performance evidence, known failure modes, the evaluation population, and the conditions under which the output should not be used.
Test explanations with intended users
Evaluate whether intended users can understand the explanation, identify uncertainty, detect an out-of-scope case, and make an appropriate next step. Test comprehension and behavior rather than only visual appeal. A technically detailed explanation can still be unsafe if it creates misplaced confidence or distracts from relevant tissue findings.
Report uncertainty and limitations
Evidence should include uncertainty where it is meaningful, subgroup coverage, calibration or threshold behavior, missing data, and known failure modes. Explain how users can question an output and what happens when the system cannot produce a result. Clear limitations are part of transparency, not an admission that evaluation has failed.
Explainability can make evidence easier to inspect, but it does not turn decision support into autonomous diagnosis. In oncology, a qualified professional must interpret the complete case within the approved or intended workflow, and public claims should describe evidence and limitations without implying a clinical result that has not been established.
Sources
- Transparency for Machine Learning-Enabled Medical Devices: Guiding Principles (U.S. Food and Drug Administration)
- Artificial Intelligence Risk Management Framework (National Institute of Standards and Technology)
- Counterfactual Diffusion Models for Interpretable Morphology-based Explanations of Artificial Intelligence Models in Pathology (Cancer Research)
Written by
Digital Pathology Solutions Editorial Team
Medical AI and digital pathology
