GSAIC: GeoScience Articles Illustration and Caption Dataset
DOI:
https://doi.org/10.54097/hset.v9i.1858Keywords:
Enter key words or phrases in alphabetical order, separated by commas.Abstract
The scientific investigation of geoscience includes data collection, sample classification and semantic, consisting of a large number of images. An image-text search model that can well assist the research work of geoscience. However, the existing image-text datasets are mainly in the field of daily life and lack academic image-text datasets. In order to help geoscience researchers to investigate through the image and text, and to provide a new benchmark for researchers in the fields of data mining and information retrieval, this paper proposes a novel parallel material of geoscience academic illiustrateion and caption (GSAIC) based on GAKG, which contains over 900,000 illustrations of earth science papers and the corresponding captions. GSAIC filters out high-quality illustrations and captions through a classifier, and with the support of experts annotations. The GSAIC will support several tasks Including text search for images, retrieving corresponding images or papers based on academic image descriptions and academic illustration classification tasks, for geoscience scenarios Finally, both the GSAIC benchmark and classifier are publicly accessible.
Downloads
References
Devlin, Jacob et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” ArXiv abs/1810.04805 (2019): n. pag.
Vaswani, Ashish et al. “Attention is All you Need.” ArXiv abs/1706.03762 (2017): n. pag.
LeCun, Yann et al. “Gradient-based learning applied to document recognition.” Proc. IEEE 86 (1998): 2278-2324.
Simonyan, Karen and Andrew Zisserman. “Very Deep Convolutional Networks for Large-Scale Image Recognition.” CoRR abs/1409.1556 (2015): n. pag.
He, Kaiming et al. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): 770-778.
Lin, Tsung-Yi et al. “Microsoft COCO: Common Objects in Context.” ECCV (2014).
Salvador, Amaia et al. “Learning Cross-Modal Embeddings for Cooking Recipes and Food Images.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 3068-3076.
Wu, Jiahong et al. “AI Challenger : A Large-scale Dataset for Going Deeper in Image Understanding.” ArXiv abs/1711.06475 (2017): n. pag.
Isola, Phillip et al. “Discovering states and transformations in image collections.” 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015): 1383-1391.
Sharma, Piyush et al. “Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.” ACL (2018).
Wah, Catherine et al. “The Caltech-UCSD Birds-200-2011 Dataset.” (2011).
Nilsback, Maria-Elena and Andrew Zisserman. “Automated Flower Classification over a Large Number of Classes.” 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing (2008): 722-729.
Song, Hyun Oh et al. “Deep Metric Learning via Lifted Structured Feature Embedding.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016): 4004-4012.
Srinivasan, Krishna et al. “WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning.” Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (2021): n. pag.
Kim, Jin-Hwa et al. “CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication.” ACL (2019).
Zhang, Han et al. “StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks.” 2017 IEEE International Conference on Computer Vision (ICCV) (2017): 5908-5916.
Ramesh, Aditya et al. “Zero-Shot Text-to-Image Generation.” ArXiv abs/2102.12092 (2021): n. pag.
Xu, Tao et al. “AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018): 1316-1324.
Yuan, Li et al. “Central Similarity Quantization for Efficient Image and Video Retrieval.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 3080-3089.
Messina, Nicola et al. “Transformer Reasoning Network for Image- Text Matching and Retrieval.” 2020 25th International Conference on Pattern Recognition (ICPR) (2021): 5222-5229.
Xie, Zhongwei et al. “Cross-Modal Retrieval between Event-Dense Text and Image.” Proceedings of the 2022 International Conference on Multimedia Retrieval (2022): n. pag.
Fain, Mikhail et al. “Dividing and Conquering Cross-Modal Recipe Retrieval: from Nearest Neighbours Baselines to SoTA.” ArXiv abs/1911.12763 (2019): n. pag.
Fontanellaz, Matthias et al. “Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries.” Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management (2019): n. pag.
Wang, Hao et al. “Structure-Aware Generation Network for Recipe Generation from Images.” ECCV (2020).
Lu, Huimin et al. “Chinese Image Captioning via Fuzzy Attention-based DenseNet-BiLSTM.” ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 17 (2021): 1 - 18.
Song, Yuqing et al. “Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards.” Proceedings of the 27th ACM International Conference on Multimedia (2019): n. pag.
Li, Xirong et al. “COCO-CN for Cross-Lingual Image Tagging, Captioning, and Retrieval.” IEEE Transactions on Multimedia 21 (2019): 2347-2360.
Anwaar, Muhammad Umer et al. “Compositional Learning of Image-Text Query for Image Retrieval.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2021): 1139-1148.
Zhang, Feifei et al. “Joint Attribute Manipulation and Modality Alignment Learning for Composing Text and Image to Image Retrieval.” Proceedings of the 28th ACM International Conference on Multimedia (2020): n. pag.
Vo, Nam S. et al. “Composing Text and Image for Image Retrieval - an Empirical Odyssey.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 6432-6441.
Mokady, Ron et al. “ClipCap: CLIP Prefix for Image Captioning.” ArXiv abs/2111.09734 (2021): n. pag.
Li, Zhuowan et al. “Context-Aware Group Captioning via Self-Attention and Contrastive Features.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 3437-3447.
Guo, Dan et al. “Recurrent Relational Memory Network for Unsupervised Image Captioning.” IJCAI (2020).
Vedantam, Ramakrishna et al. “Context-Aware Captions from Context-Agnostic Supervision.” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017): 1070-1079.
Chen, Tseng-Hung et al. “Show, Adapt and Tell: Adversarial Training of Cross-Domain Image Captioner.” 2017 IEEE International Conference on Computer Vision (ICCV) (2017): 521-530.
Wei, Xiu-Shen et al. “Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval.” IEEE Transactions on Image Processing 26 (2017): 2868-2881.
Zhe, Xuefei et al. “Directional Statistics-based Deep Metric Learning for Image Classification and Retrieval.” ArXiv abs/1802.09662 (2019): n. pag.
Yuan, Mingkuan and Yuxin Peng. “CKD: Cross-Task Knowledge Distillation for Text-to-Image Synthesis.” IEEE Transactions on Multimedia 22 (2020): 1955-1968.
Zhao, Wei et al. “Dual Learning for Cross-domain Image Captioning.” Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (2017): n. pag.
Lv, Xiaoming and Fa-jie Duan. “Metric learning via feature weighting for scalable image retrieval.” Pattern Recognit. Lett. 109 (2018): 97-102.
Brown, A. et al. “Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval.” ArXiv abs/2007.12163 (2020): n. pag.
Chen, Binghui and Weihong Deng. “Hybrid-Attention Based Decoupled Metric Learning for Zero-Shot Image Retrieval.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019): 2745-2754.
Chen, Binghui and Weihong Deng. “Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering.” ArXiv abs/1901.07169 (2019): n. pag.
Jain, Aashi et al. “MURAL: Multimodal, Multitask Retrieval Across Languages.” ArXiv abs/2109.05125 (2021): n. pag.
Messina, Nicola et al. “Transformer-Based Multi-modal Proposal and Re-Rank for Wikipedia Image-Caption Matching.” ArXiv abs/2206.10436 (2022): n. pag.
El-Nouby, Alaaeldin et al. “Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV) (2019): 10303-10311.
Liu, Zhenhuan et al. “IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoning.” Proceedings of the 28th ACM International Conference on Multimedia (2020): n. pag.
Matsumori, Shoya et al. “LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation.” IEEE Access 9 (2021): 160521-160532.
Deng, Cheng et al. “GAKG: A Multimodal Geoscience Academic Knowledge Graph.” Proceedings of the 30th ACM International Conference on Information & Knowledge Management (2021): n. pag.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
