Research on Reinforcement Learning Explainable Strategies Based on Advantage Saliency

Dandan Yan

doi:10.54097/fcis.v3i1.6348

Authors

Dandan Yan

DOI:

https://doi.org/10.54097/fcis.v3i1.6348

Keywords:

Advantage Function, Explain ability, Perturbation-based Saliency

Abstract

Deep reinforcement learning is increasingly being used in difficult environments with sparse rewards and high-dimensional inputs, and it performs well, but its decision-making processes are largely unclear and difficult to explain to end users. Saliency map methods explain an agent's behavior by highlighting state features relevant for the agent to take an action. In this paper, we use the perturbation-based saliency map method, propose the use of advantage function to replace the existing method of calculating state saliency, realize the combination of advantage function and perturbation-based saliency map. A saliency map is generated by noting the saliency of the dependent elements of the agent's chosen action in the Atari game environment. Experimental comparisons show that our method generates more accurate explanatory saliency maps.

Downloads

Download data is not yet available.

References

Puiutta, Erika, and Eric MSP Veith. "Explainable reinforcement learning: A survey." Machine Learning and Knowledge Extraction: 4th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2020, Dublin, Ireland, August 25–28, 2020, Proceedings 4. Springer International Publishing, 2020.

Sutton, Richard S., and Andrew G. Barto. Introduction to reinforcement learning. Vol. 135. Cambridge: MIT press, 1998.

Affonso, Carlos, et al. "Deep learning for biological image classification." Expert systems with applications 85 (2017): 114-122.

Singh, Shashi Pal, et al. "Machine translation using deep learning: An overview." 2017 international conference on computer, communications and electronics (comptelix). IEEE, 2017.

You, Quanzeng, et al. "Image captioning with semantic attention." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

Atrey, Akanksha, Kaleigh Clary, and David Jensen. "Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning." arXiv preprint arXiv:1912.05743 (2019).

Wang, Ziyu, et al. "Dueling network architectures for deep reinforcement learning." International conference on machine learning. PMLR, 2016.

Greydanus, Samuel, et al. "Visualizing and understanding atari agents." International conference on machine learning. PMLR, 2018.

Gupta, Piyush, et al. "Explain your move: Understanding agent actions using focused feature saliency." arXiv preprint arXiv:1912.12191 (2019).

Iyer, Rahul, et al. "Transparency and explanation in deep reinforcement learning neural networks." Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 2018.

Mott, Alexander, et al. "Towards interpretable reinforcement learning using attention augmented agents." Advances in neural information processing systems 32 (2019).

Ivanovs, Maksims, Roberts Kadikis, and Kaspars Ozols. "Perturbation-based methods for explaining deep neural networks: A survey." Pattern Recognition Letters 150 (2021): 228-234.

Khakzar, Ashkan, et al. "Improving feature attribution through input-specific network pruning." arXiv preprint arXiv:1911.11081 (2019).

Zeiler, Matthew D., and Rob Fergus. "Visualizing and understanding convolutional networks." Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. Springer International Publishing, 2014.

Petsiuk, Vitali, Abir Das, and Kate Saenko. "Rise: Randomized input sampling for explanation of black-box models." arXiv preprint arXiv:1806.07421 (2018).

Yang, Qing, et al. "Mfpp: Morphological fragmental perturbation pyramid for black-box model explanations." 2020 25th International conference on pattern recognition (ICPR). IEEE, 2021.

Qi, Chen, et al. "Deep reinforcement learning with discrete normalized advantage functions for resource management in network slicing." IEEE Communications Letters 23.8 (2019): 1337-1341.

Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International conference on machine learning. PMLR, 2016.

Zahavy, Tom, Nir Ben-Zrihem, and Shie Mannor. "Graying the black box: Understanding dqns." International conference on machine learning. PMLR, 2016.