The Compression Techniques Applied on Deep Learning Model
DOI:
https://doi.org/10.54097/hset.v4i.920Keywords:
Quantization, Knowledge Distillation, Low-Rank Matrix Factorization, Pruning.Abstract
In recent years, the penetration rate of smartphones has gradually completed, artificial intelligence is the cutting-edge technology that can trigger disruptive changes. Deep learning neural networks are also starting to appear on mobile devices. In order to obtain better performance, more complex networks need to be designed, and the corresponding models, computation and storage space are increasing, however, the challenges of resource allocation and energy consumption still exist in mobile. The techniques for compressing deep learning models are quite important, and this paper studies a series of related literatures. This paper reviews deep learning-based deep neural network compression techniques and introduces the key operational points of knowledge extraction and network model on the learning performance of Resolution-Aware Knowledge Distillation. In this paper, a low-rank decomposition algorithm is evaluated based on sparse parameters and rank using the extended BIC for tuning parameter selection. This paper discusses the reduction of redundancy in the fully connected and constitutive layers of the training network model by pruning strategies.Moreover, this paper presents the quantization techniques and a neural network that quantifies weights and activations by applying differentiable nonlinear functions.
Downloads
References
G. Hinton, O. Vinyals, J. Dean. Distilling the Knowledge in a Neural Network[J]. Computer Science, 2015
X. Chen, Z. Q. Xing and Y. Y. Cheng, "Introduction to Model Compression Knowledge Distillation," 2021 6th International Conference on Intelligent Computing and Signal Processing (ICSP), 2021, pp. 1464-1467, doi: 10.1109/ICSP51882.2021.9408881.
I. -H. Shin, Y. -H. Moon and Y. -J. Lee, "Towards Understanding Architectural Effects on Knowledge Distillation," 2020 International Conference on Information and Communication Technology Convergence (ICTC), 2020, pp. 1144-1146, doi: 10.1109/ICTC49870.2020.9289630.
H. Ni, J. Shen and C. Yuan, "Enhanced Knowledge Distillation for Face Recognition," 2019 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), 2019, pp. 1441-1444, doi: 10.1109/ISPA-BDCloud-SustainCom-SocialCom48970.2019.00207.
Z. Feng, J. Lai and X. Xie, "Resolution-Aware Knowledge Distillation for Efficient Inference," in IEEE Transactions on Image Processing, vol. 30, pp. 6985-6996, 2021, doi: 10.1109/TIP.2021.3101158.
J. Sigurdsson, M. O. Ulfarsson and J. R. Sveinsson, "Sparse and low rank hyperspectral unmixing," 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 2017, pp. 229-232, doi: 10.1109/IGARSS.2017.8126936.
F. Ong and M. Lustig, "Beyond Low Rank + Sparse: Multiscale Low Rank Matrix Decomposition," in IEEE Journal of Selected Topics in Signal Processing, vol. 10, no. 4, pp. 672-687, June 2016, doi: 10.1109/JSTSP.2016.2545518.
M. O. Ulfarsson, V. Solo and G. Marjanovic, "Sparse and low rank decomposition using l0 penalty," 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 3312-3316, doi: 10.1109/ICASSP.2015.7178584.
H. Cao, X. Shang, C. Yu, M. Song and C. -I. Chang, "Hyperspectral Classification Using Low Rank and Sparsity Matrices Decomposition," IGARSS 2020 - 2020 IEEE International Geoscience and Remote Sensing Symposium, 2020, pp. 477-480, doi: 10.1109/IGARSS39084.2020.9324009.
M. F. Kaloorazi and R. C. de Lamare, "Low-rank and sparse matrix recovery based on a randomized rank-revealing decomposition," 2017 22nd International Conference on Digital Signal Processing (DSP), 2017, pp. 1-5, doi: 10.1109/ICDSP.2017.8096137.
L.H.Guo,D. Chen,K. Jia.Knowledge transferred adaptive filter pruning for CNN compression and acceleration[J/OL].ScienceChina(InformationSciences):1-2[2022-04-06].
Y.Fang,C.Li,P.C.Wang,C.Q.Han,R.Huang,X. Huang. EasiEdge: A Novel Global Deep Neural Networks Pruning Method for Efficient Edge Computing[J]. IEEE INTERNET OF THINGS JOURNAL,2021,8(3).
Y. S. Ki, S. Philipp, L. Sebastian, B. Alexander,W. Simon, M. K. Robert, S. Wojciech. Pruning by explaining: A novel criterion for deep neural network pruning[J]. Pattern Recognition,2021,115(prepublish).
K. KAMMA, Y. ISODA, S. INOUE, T. WADA. Neural Behavior-Based Approach for Neural Network Pruning[J]. IEICE Transactions on Information and Systems,2020,E103.D(5).
G.Li,F.Liu,Y.P.Xia.Overview of Deep Convolutional Neural Network Pruning[J]. 2020 INTERNATIONAL CONFERENCE ON IMAGE, VIDEO PROCESSING AND ARTIFICIAL INTELLIGENCE,2020,11584.
J.Yang,X.Shen,J.Xing, et al. Quantization networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 7308-7316.
C.Zhu, S.Han, H.Mao, et al. Trained ternary quantization[J]. arXiv preprint arXiv:1612.01064, 2016.
V.Moura, V.Almeida, D.B.S.Santos, et al. Mobile Device ECG Classification using quantized Neural Networks[J]. 2020.
F.Liu, W.Zhao, Z.He, et al. Improving neural network efficiency via post-training quantization with adaptive floating-point[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 5281-5290.
H.Pouransari, Z.Tu, O.Tuzel. Least squares binary quantization of neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2020: 698-699.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
