Research Advanced in Image Inpainting based on Deep Learning
DOI:
https://doi.org/10.54097/8rn86v59Keywords:
Image inpainting, deep learning, autoencoders, generative adversarial networks.Abstract
Image inpainting has always been a hotspot in the computer vision community, which aims to restore damaged or missing parts of an image, ensuring that the restored image is physically reasonable, visually pleasing, and consistent in texture and structure with the original image. Early inpainting methods primarily depend on diverse image processing technologies, such as sample replication, partial differential equation solving, etc. As artificial intelligence rapidly progresses, image inpainting methods based on deep learning exploit robust feature extraction capabilities to extract useful information from vast amounts of data, contributing to more robust and natural inpainting effects, especially when dealing with complex and high-dimensional scenes. This paper provides a review of deep learning-based image inpainting technologies, categorized into four main types: autoencoders, generative adversarial networks (GANs), recurrent neural networks (RNNs), and convolutional neural networks (CNNs). It summarizes and analyzes the basic principles, advantages, disadvantages, and performance of these methods in image inpainting, as well as introduces commonly used datasets and quantitative evaluation metrics. Additionally, the paper discusses the principal challenges currently faced in this domain and provides insights into future research directions for deep learning-based image inpainting.
Downloads
References
[1] D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell and A. A. Efros, "Context Encoders: Feature Learning by Inpainting," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 2536-2544, doi: 10.1109/CVPR.2016.278.
[2] Iizuka S, Simo-Serra E, Ishikawa H. Globally and locally consistent image completion. ACM Transactions on Graphics, 2017, 36 (4): 107. DOI:10.1145/3072959.3073659.
[3] Liu G, Reda F A , Shih K J ,et al.Image Inpainting for Irregular Holes Using Partial Convolutions [J].Springer, Cham, 2018.DOI:10.1007/978-3-030-01252-6_6.
[4] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu and T. Huang, "Free-Form Image Inpainting With Gated Convolution," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 4470-4479, doi: 10.1109/ICCV.2019.00457.
[5] Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning. arXiv:1901.00212, 2019.
[6] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu and T. S. Huang, "Generative Image Inpainting with Contextual Attention," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 5505-5514, doi: 10.1109/CVPR.2018.00577.
[7] ZHANG X, WANG X, SHI C, et al. De-gan: domain embedded gan for high quality face image inpainting [J]. Pattern Recognition, 2022, 124: 108415.
[8] TIAN H, ZHANG L, LI S, et al. Pyramid-vae-gan: transferring hierarchical latent variables for image inpainting [J]. Computational Visual Media, 2023, 9 (4): 827- 841.
[9] LI H, LI G, LIN L, et al. Context-aware semantic inpainting [J]. IEEE transactions on cybernetics, 2018, 49 (12): 4398-4411.
[10] ZENG Y, FU J, CHAO H, et al. Aggregated contextual transformations for high-resolution image inpainting [J]. IEEE Transactions on Visualization and Computer Graphics, 2022.
[11] CHEN Y, XIA R, YANG K, et al. Dnnam: image inpainting algorithm via deep neural networks and attention mechanism [J]. Applied Soft Computing, 2024, 154: 111392.
[12] Xie et al. (2019) introduced RNNs into image inpainting and proposed a model that integrates Long Short-Term Memory (LSTM) networks, aiming to capture the relationships between the missing regions of an image and their surrounding known areas, resulting in better inpainting outputs with continuity .
[13] Liu et al. (2020) proposed an image inpainting method based on Gated Recurrent Units (GRU). This method is noted for its simplicity and effectiveness in handling long-term dependencies, demonstrating good performance in image inpainting tasks involving large missing regions .
[14] Zhang et al. (2021) expanded the application of RNNs in image inpainting by proposing a model based on bidirectional RNNs. This approach allows for a more comprehensive capture of contextual relationships by integrating information from both forward and backward information flows, enhancing the quality and consistency of the inpainting results .
[15] Dong C, Loy C C, He K, et al. Image super-resolution using deepconvolutional networks [J]. IEEE transactions on pattern analysis and machineintelligence, 2015, 38 (2): 295-307.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Highlights in Science, Engineering and Technology
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
How to Cite
