Research on Explainability of Deep Neural Networks and Its Applications
DOI:
https://doi.org/10.54097/ekzm5z29Keywords:
Neural Networks; Interpretability; Explainable AI; Classification.Abstract
In recent years, Artificial Intelligence (AI) models have surged in popularity and success, garnering much attention across many disciplines and applications. In particular, recent developments in Deep Neural Networks (DNN) have allowed them to match and even surpass human capabilities in many tasks. However, the complexity of deep neural networks causes them function like black boxes that are very difficult for humans to understand, thus making them untrustworthy and limiting their usefulness in high-risk applications. Explainable Artificial Intelligence aims to alleviate this issue by generating explanations for DNNs to allow humans to understand their reasoning and increase their transparency. This paper reviews research on neural network explainability, detailing its definition, necessity, classification, and evaluation. It categorizes current explanation methods for deep neural networks according to their number of inputs. Additionally, it introduces the principles and evaluation methods for common interpretability algorithms. Future research directions and applications of interpretable neural networks are summarized. The challenges facing interpretable neural networks are outlined, along with potential solutions to these challenges.
Downloads
References
[1] Joshi, Gargi, et al. “A Review on Explainability in Multimodal Deep Neural Nets.” IEEE Access, vol. 9, 2021, pp. 59800–21. [doi: 10.1109/ACCESS.2021.3070212.]
[2] Bodria, Francesco, et al. “Benchmarking and Survey of Explanation Methods for Black Box Models.” Data Mining and Knowledge Discovery, vol. 37, no. 5, Sept. 2023, pp. 1719–78. [doi: 0.1007/s10618-023-00933-9.]
[3] Rangapur, Aryan, and Aman Rangapur. “The Battle of LLMs: A Comparative Study in Conversational QA Tasks”. [arXiv:2405.18344]
[4] Kong XW, Tang XZ, Wang ZM. “A survey of explainable artificial intelligence decision.” Systems Engineering —Theory & Practice, 2021, 41(2): 524–536 [doi: 10.12011/SETP2020-1536]
[5] Chamola, V., Hassija, V., Sulthana, A. R., Ghosh, D., Dhingra, D., & Sikdar, B. (2023). “A Review of Trustworthy and Explainable Artificial Intelligence (XAI).” IEEE [doi: 10.1109/ACCESS.2023.3294569]
[6] Doshi-Velez, Finale, and Been Kim. “Towards A Rigorous Science of Interpretable Machine Learning” 2 Mar. 2017. [arXiv: 1702.08608].
[7] Hassija, Vikas, et al. “Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence.” Cognitive Computation, vol. 16, no. 1, Jan. 2024, pp. 45–74. [doi: 10.1007/s12559-023-10179-8.]
[8] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2020). “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization.” International Journal of Computer Vision, 128(2), 336–359. [doi: 10.1007/s11263-019-01228-7]
[9] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier [arXiv: 1602.04938]
[10] Lundberg, Scott, and Su-In Lee. “A Unified Approach to Interpreting Model Predictions.” 24 Nov. 2017. [arXiv: 1705.07874].
[11] Goyal, Yash, et al. “Counterfactual Visual Explanations.” 11 June 2019. [arXiv: 1904.07451].
[12] Dhurandhar, Amit, et al. “Explanations Based on the Missing: Towards Contrastive Explanations with Pertinent Negatives.” 29 Oct. 2018. [arXiv: 1802.07623].
[13] Ribeiro, Marco Tulio, et al. "Anchors: High-Precision Model-Agnostic Explanations.” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, Apr. 2018.: [doi: 10.1609/aaai. v32i1.11491.]
[14] Rodis, Nikolaos, et al. “Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions.” 30 June 2024. [arXiv: 2306.05731].
[15] Si NW, Zhang WL, Qu D, Luo XY, Chang HY, Niu T. “Representation visualization of convolutional neural networks: A survey.” Acta Automatica Sinica, 2022, 48(8): 1890–1920 (in Chinese with English abstract). [doi: 10.16383/j.aas.c200554]
[16] Joshi, Gargi, et al. “A Review on Explainability in Multimodal Deep Neural Nets.” IEEE Access, vol. 9, 2021, pp. 59800–21. [doi: 10.1109/ACCESS.2021.3070212.]
[17] Adadi, Amina & Berrada, Mohammed. (2018). “Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)”. [doi:10.1109/ACCESS.2018.2870052.]
[18] Park, Dong Huk, et al. “Multimodal Explanations: Justifying Decisions and Pointing to the Evidence”. [arXiv :1802.08129]
[19] Smilkov, Daniel, et al. SmoothGrad: “Removing Noise by Adding Noise.” 12 June 2017. [arXiv: 1706.03825].
[20] Sundararajan, Mukund, et al. Axiomatic Attribution for Deep Networks. 12 June 2017. [arXiv: 1703.01365].
[21] Singla S, Wallace E, Feng S, Feizi S. “Understanding impacts of high-order loss approximations and features in deep learning interpretation.” In: Proc. of the 36th Int’l Conf. on Machine Learning. Long Beach: PMLR, 2019. 5848–5856.
[22] Lapuschkin, Sebastian, et al. “Unmasking Clever Hans Predictors and Assessing What Machines Really Learn.” Nature Communications, vol. 10, no. 1, Mar. 2019, p. 1096. [doi: 10.1038/s41467-019-08987-4.]
[23] Ramamurthy, Karthikeyan Natesan, et al. “Model Agnostic Multilevel Explanations.” 12 Mar. 2020. [arXiv: 2003.06005].
[24] Yang, Chengliang, et al. “Global Model Interpretation via Recursive Partitioning.” 23 May 2018. [arXiv: 1802.04253].
[25] Coroama, Loredana, and Adrian Groza. “Evaluation Metrics in Explainable Artificial Intelligence (XAI).” Advanced Research in Technologies, Information, Innovation and Sustainability, edited by Teresa Guarda et al., vol. 1675, Springer Nature Switzerland, 2022, pp. 401–13. [doi: 10.1007/978-3-031-20319-0_30.]
[26] Hoffman, Robert R., et al. “Metrics for Explainable AI: Challenges and Prospects”. [arXiv:1806.08049]
[27] Alvarez-Melis, David, and Tommi S. Jaakkola. “On the Robustness of Interpretability Methods”. [arXiv:1806.08049]
[28] Miró-Nicolau, Miquel, et al. “A Comprehensive Study on Fidelity Metrics for XAI”. [arXiv:2401.10640]
[29] Phillips, P. Jonathon, et al. “Four Principles of Explainable Artificial Intelligence.” NIST IR 8312, National Institute of Standards and Technology (U.S.), 29 Sept. 2021, p. NIST IR 8312. [doi: 10.6028/NIST.IR.8312.]
[30] Bhatt, Umang, et al. “Explainable Machine Learning in Deployment”. [arXiv:1909.06342]
[31] Singh, Amitojdeep, et al. Explainable Deep Learning Models in Medical Image Analysis. 28 May 2020. [doi: 10.48550/arXiv.2005.13799.]
[32] Pahud de Mortanges, Aurélie, et al. “Orchestrating Explainable Artificial Intelligence for Multimodal and Longitudinal Data in Medical Imaging.” Npj Digital Medicine, vol. 7, no. 1, July 2024, pp. 1–10. www.nature.com, [doi: 10.1038/s41746-024-01190-w.]
[33] Soares E, Angelov P. “Fair-by-design explainable models for prediction of recidivism.” [arXiv: 1910.02043, 2019]
[34] Soares E, Angelov P, Costa B, Castro M. “Actively semi-supervised deep rule-based classifier applied to adverse driving scenarios”. In: Proc. of the 2019 Int’l Joint Conf. on Neural Networks. Budapest: IEEE, 2019. 1–8. [doi: 10.1109/IJCNN.2019.8851842]
[35] Soares E, AngelovPP, Costa B, Castro MPG,Nageshrao S, Filev D. “Explaining deep learning models through rule-based approximation and visualization.” IEEE Trans. on Fuzzy Systems, 2021, 29(8): 2399–2407. [doi: 10.1109/TFUZZ.2020.2999776]
[36] Gao JY, Wang XT, Wang YS, Xie X. “Explainable recommendation through attentive multi-view learning.” In: Proc. of the 33rd AAAIConf. on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conf. and 9th AAAI Symp. on Educational Advances in Artificial Intelligence. Honolulu: AAAI Press, 2019. 445. [doi: 10.1609/aaai.v33i01.33013622]
[37] Sovrano, Francesco, and Fabio Vitali. “An Objective Metric for Explainable AI: How and Why to Estimate the Degree of Explainability.” Knowledge-Based Systems, vol. 278, Oct. 2023, p. 110866. arXiv.org, [doi: 10.1016/j.knosys.2023.110866]
[38] Chou, Y.-L., Moreira, C., Bruza, P., Ouyang, C., & Jorge, J. (2022). “Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms and applications”. Information Fusion, 81, 59–83.
[39] Miró-Nicolau, M., et al. “Assessing Fidelity in XAI Post-hoc Techniques: A Comparative Study with Ground Truth Explanations Datasets”. [arXiv:2311.01961]
[40] Singh, Chandan, et al. “Rethinking Interpretability in the Era of Large Language Models”. [arXiv:2402.01761]
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Highlights in Science, Engineering and Technology
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
