A Survey on CLIP-Guided Vision-Language Tasks
DOI:
https://doi.org/10.54097/hset.v12i.1418Keywords:
Contrastive Language Image Pre-training, Vision-Language tasks, Transfer learning, Retrieval task.Abstract
Multimodal learning refers to the representation of different modalities using a unified model. Modalities include images, text, audio, etc. In this article, we will first introduce the basic approach of CLIP which is a vision language model with the power of connecting different modalities, and then present different models inspired by CLIP on various downstream tasks. In the end, we conclude with a summary of the prospects and limitations of multimodal learning.
Downloads
References
Feilong Chen, Duzhen Zhang, Minglun Han, Xiuyi Chen, Jing Shi, Shuang Xu, and Bo Xu. Vlp: A survey on vision-language pre-training, 2022.
RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision [J/OL]. CoRR, 2021, abs/2103.00020. https://arxiv.org/abs/2103.00020.
SCHWEMMER C, KNIGHT C, BELLO-PARDO E D, et al. Diagnosing gender bias in image recognition systems [J/OL]. Socius, 2020, 6: 2378023120967171.https://doi.org/10.1177/2378023120967171.
BIRHANE A, PRABHU V U, KAHEMBWE E. Multimodal datasets: misogyny, pornography, and malignant stereotypes [J/OL]. CoRR, 2021, abs/2110.01963.https://arxiv.org/abs/2110.01963.
JIA J, LIU Y, GONG N Z. Badencoder: Backdoor attacks to pre-trained encoders in self-supervised learning [J/OL]. CoRR, 2021, abs/2108.00352. https://arxiv.or g/abs/2108.00352.
Ron Mokady, Amir Hertz, and Amit H. Bermano. Clipcap: CLIP prefix for image captioning. CoRR, abs/2111.09734, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 (8): 9, 2019. 2, 3.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CoRR, abs/1812.04948, 2018.
PATASHNIK O, WU Z, SHECHTMAN E, et al. Styleclip: Text-driven manipulation of stylegan imagery [J/OL]. CoRR, 2021, abs/2103.17249. https: //arxiv.org/abs/2103.17249.
VINKER Y, PAJOUHESHGAR E, BO J Y, et al. Clipasso: Semantically-aware object sketching [A]. 2022. arXiv: 2202.05822.
HONG F, ZHANG M, PAN L, et al. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars [J]. ACM Transactions on Graphics (TOG), 2022, 41 (4): 1-19.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval. CoRR, abs/2104.0886.
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for videoaction recognition. CoRR, abs/2109.08472, 2021.
Andrey Guzhov, Federico Raue, J ̈orn Hees, and Andreas Dengel. Audioclip: Extending CLIP to image, text, and audio. CoRR, abs/2106.13043.
ZHOU K, YANG J, LOY C C, et al. Learning to prompt for vision-language models [J/OL]. CoRR, 2021, abs/2109.01134.https://arxiv.org/abs/2109.01134.
ZHOU K, YANG J, LOY C C, et al. Conditional prompt learning for vision- language models [M/OL]. arXiv, 2022. https://arxiv.org/abs/2203.05557. DOI: 10.48550/ARXIV.2203.05557.
Marcos V. Conde and Kerem Turgutlu. CLIP-art: Contrastive pre-training for fine-grained art classification. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, jun 2021.
Piotr Mirowski, Dylan Banarse, Mateusz Malinowski, Simon Osindero, and Chrisantha Fernando. Clip-clop: Clip-guided collage and photomontage, 2022.
Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot open set detection by extending CLIP. CoRR, abs/2109.0274,2021.
Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image CLIP. CoRR, abs/2106.11097, 2021.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. CoRR, abs/2104.08718,2021.
Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. CoRR, abs/2109.14084,2021.
Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, and Juan Pablo Bello. Wav2clip: Learning robust audio representations from CLIP. CoRR, abs/2110.11499, 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. CoRR, abs/2109.12098, 2021.
Kevin Frans, Lisa B. Soros, and Olaf Witkowski. Clipdraw: Exploring text-to-drawing synthesis through language-image encoders. CoRR, abs/2106.14843,2021.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
