A Survey on Improved GAN based Image Inpainting for Different Aims
DOI:
https://doi.org/10.54097/hset.v39i.6552Keywords:
Image Inpainting; GAN; Deep Learning; Computer Vision.Abstract
Computer Vision (CV) is an important field of Artificial Intelligence (AI) that focuses on processing and analyzing the visual data input like images or videos. And image inpainting technique is a task of reconstructing image with missing parts by using the information of the existing part, which becomes a crucial part in CV. It has various image processing applications such as digital cultural heritage preservation, old photograph restoration and object removal. The researchers found that using generative adversarial network can extract the feature from image without huge amount of computation which are two main challenges of image inpainting, therefore GAN-based inpainting method has recently been a hub for research. In this paper, we review most of the latest improved image inpainting methods and divide them into three categories according to their different aims: for semantic image inpainting, for generating higher diversity and for irregular or free-form image input. Firstly, we review the methods from structure and algorithm. Then we analyze the advantages and disadvantages of each method. Finally, we conclude the challenges or limitations raised from our findings and some potential development trend.
Downloads
References
Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
Pathak D, Krahenbuhl P, Donahue J, et al. Context encoders: Feature learning by inpainting[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 2536-2544.
Iizuka S, Simo-Serra E, Ishikawa H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics (ToG), 2017, 36(4): 1-14.
Yu J, Lin Z, Yang J, et al. Generative image inpainting with contextual attention[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 5505-5514.
Zeng Y, Lin Z, Lu H, et al. Cr-fill: Generative image inpainting with auxiliary contextual reconstruction [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 14164 -14173.
Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C]//International conference on machine learning. PMLR, 2017: 214-223.
Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of wasserstein gans[J]. Advances in neural information processing systems, 2017, 30.
Vitoria P, Sintes J, Ballester C. Semantic image inpainting through improved wasserstein generative adversarial networks[J]. arXiv preprint arXiv:1812.01071, 2018.
Zeng Y, Fu J, Chao H, et al. Learning pyramid-context encoder network for high-quality image inpainting [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1486-1494.
Liu H, Wan Z, Huang W, et al. Pd-gan: Probabilistic diverse gan for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 9371-9381.
Liu G, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions [C]// Proceedings of the European conference on computer vision (ECCV). 2018: 85-100.
Yu J, Lin Z, Yang J, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 4471-4480.
Jo Y, Park J. Sc-fegan: Face editing generative adversarial network with user's sketch and color [C]// Proceedings of the IEEE/CVF international conference on computer vision. 2019: 1745-1753.
Liu Z, Luo P, Wang X, et al. Deep learning face attributes in the wild[C]//Proceedings of the IEEE international conference on computer vision. 2015: 3730-3738.
Karras T, Aila T, Laine S, et al. Progressive growing of gans for improved quality, stability, and variation[J]. arXiv preprint arXiv:1710.10196, 2017.
Russakovsky O, Deng J, Su H, et al. Imagenet large scale visual recognition challenge[J]. International journal of computer vision, 2015, 115(3): 211-252.
Doersch C, Singh S, Gupta A, et al. What makes paris look like paris? [J]. ACM Transactions on Graphics, 2012, 31(4).
Zhou B, Lapedriza A, Khosla A, et al. Places: A 10 million image database for scene recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(6): 1452-1464.
Cimpoi M, Maji S, Kokkinos I, et al. Describing textures in the wild[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 3606-3613.
Everingham M, Eslami S M, Van Gool L, et al. The pascal visual object classes challenge: A retrospective [J]. International journal of computer vision, 2015, 111(1): 98-136.
Tyleček R, Šára R. Spatial pattern templates for recognition of objects with regular structure[C]//German conference on pattern recognition. Springer, Berlin, Heidelberg, 2013: 364-374.
Netzer Y, Wang T, Coates A, et al. Reading digits in natural images with unsupervised feature learning[J]. 2011.
Yu F, Seff A, Zhang Y, et al. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop[J]. arXiv preprint arXiv:1506.03365, 2015.
Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images[J]. 2009.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
