Research on Multimodal Summarization Based on Sequence to Sequence Model
DOI:
https://doi.org/10.54097/fcis.v1i1.1076Keywords:
Multimodal summarization, Sequence to Sequence model, Attention mechanismAbstract
With the rapid development of information technology in the 21st century, the data on the Internet is growing exponentially, which makes it difficult for users to receive useful information from an endless stream of information sources. Therefore, as a kind of data compression task, automatic summarization technology has gradually become a research hotspot in academia and industry. This paper mainly introduces the multimodal summarization method from sequence to sequence model, then introduces the current multimodal summarization evaluation methods and finally gives some conclusions and thoughts.
Downloads
References
Jangra, A., et al., Text-Image-Video Summary Generation Using Joint Integer Linear Programming, in Advances in Information Retrieval. 2020. p. 190-198.
Li, H., et al. Multi-modal summarization for asynchronous collection of text, image, audio and video. in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017.
[Cho, K., et al., Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078, 2014.
Sutskever, I., O. Vinyals, and Q.V. Le. Sequence to sequence learning with neural networks. in Advances in neural information processing systems. 2014.
Bahdanau, D., K. Cho, and Y. Bengio, Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
Rush, A.M., S. Chopra, and J. Weston, A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685, 2015.
Shi, T., et al., Neural abstractive text summarization with sequence-to-sequence models. ACM Transactions on Data Science, 2021. 2(1): p. 1-37 .
Hochreiter, S. and J. Schmidhuber, Long short-term memory. Neural computation, 1997. 9(8): p. 1735-1780.
Zhu, J., et al. MSMO: Multimodal summarization with multimodal output. in Proceedings of the 2018 conference on empirical methods in natural language processing. 2018.
Zhu, J., et al., Multimodal Summarization with Guidance of Multimodal Reference. Proceedings of the AAAI Conference on Artificial Intelligence, 2020. 34(05): p. 9749-9756.
Khullar, A. and U. Arora, MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention. arXiv preprint arXiv:2010.08021, 2020.
Liu, N., et al. Multistage Fusion with Forget Gate for Multimodal Summarization in Open-Domain Videos. in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020.
He, K., et al. Deep residual learning for image recognition. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Simonyan, K. and A. Zisserman, Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
Szegedy, C., et al. Going deeper with convolutions. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
Deng, J., et al. Imagenet: A large-scale hierarchical image database. in 2009 IEEE conference on computer vision and pattern recognition. 2009. IEEE.
Karpathy, A., A. Joulin, and L. Fei-Fei, Deep fragment embeddings for bidirectional image sentence mapping. arXiv preprint arXiv:1406.5679, 2014.
Wang, L., Y. Li, and S. Lazebnik. Learning deep structure-preserving image-text embeddings. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Rashtchian, C., et al. Collecting image annotations using amazon’s mechanical turk. in Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. 2010.
Hodosh, M., P. Young, and J. Hockenmaier, Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 2013. 47: p. 853-899.
Young, P., et al., From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014. 2: p. 67-78.
Wang, L., Y. Li, and S. Lazebnik. Learning deep structure-preserving image-text embeddings. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
Vaswani, A., et al. Attention is all you need. in Advances in neural information processing systems. 2017.
Li, H., et al. Multi-modal Sentence Summarization with Modality Attention and Image Filtering. in IJCAI. 2018.
Fu, X., J. Wang, and Z. Yang, Multi-modal Summarization for Video-containing Documents. arXiv preprint arXiv:2009.08018, 2020.
Kim, J.-H., et al., Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325, 2016.
Li, M., et al., VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles. arXiv preprint arXiv:2010.05406, 2020.
