Research on Application of Financial Large Language Models
DOI:
https://doi.org/10.54097/1z673097Keywords:
Large Language Models, BloombergGPT, PIXIU, FinBERT, natural language processing.Abstract
With the increasing use of large language models such as chatgpt, it is not difficult to apply their capabilities to the research of natural language processing in the financial field, including but not limited to text extraction, sentiment analysis, etc. This paper analyzes the construction ideas and applications of three financial big language models, including BloombergGPT, PIXIU and FinBERT, and concludes that the current application of big language models in the financial field is possible, multi-faceted and suitable, but there are still shortcomings in ethics, data processing and other aspects. The application of large language models in the field of finance is still something to look forward to. Through this study and the comparative exploration of various models, we hope to provide valuable modeling experience for practitioners in the field of finance or computer. At the same time, it is hoped that each researcher can follow the ideas of these model-making teams to make up for the shortcomings in their own models and make their own financial big language models better.
Downloads
References
[1] T. Brown, B. Mann, N. Ryder, et al. Language models are few-shot learners. NeurIPS, 33: 1877 – 1901. (2020).
[2] T. Almutiri, F. Nadeem, Markov models applications in natural language processing: a survey. Int. J. Inf. Technol. Comput. Sci 2, 1 – 16 (2022).
[3] Tom B. Brown et al. Language Models are Few-Shot Learners, arXiv:2005.14165 (2020).
[4] J. Devlin, M. Chang, K. Lee, et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv: 1810.04805 (2019).
[5] T. Scao, A. Fan, C. Akiki, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv: 2211.05100. (2022).
[6] S. Wu, O. Irsoy, S. Lu, et al. BloombergGPT: A Large Language Model for Finance, arXiv: 2303.17564v3 (2023).
[7] J. Hoffmann, S. Borgeaud, A. Mensch et al. (2022). An empirical analysis of compute-optimal large language model training. Adv. neural inf. process. syst, 35, 30016 - 30030 (2022).
[8] M. Maia, S. Handschuh, A. e Freitas, et al. financial opinion mining and question answering. The Web Conference 2018, pages 1941 – 1942 (2018).
[9] Q. Xie, W. Han, X. Zhang, et al. PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance, arXiv: 2306.05443v1 (2023).
[10] OpenAI. GPT-4 Technical Report. arXiv: 2303.08774 (2023).
[11] T. Le Scao, A. Fan, C. Akiki, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv: 2211.05100 (2022).
[12] S. Zhang, S. Roller, N. Goyal, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: 2205.01068 (2022).
[13] Z. Zhang, H. Zhang, K. Chen, et al. Mengzi: Towards lightweight yet ingenious pre-trained models for chinese. arXiv preprint arXiv: 2110.06696 (2021).
[14] M. E Peters, M. Neumann, M. Iyyer, et al. Deep contextualized word representations. https: //doi.org/10.18653/v1/N18- 1202 arXiv: 1802.05365 (2018).
[15] J. Howard, S. Ruder. Universal Language Model Fine- tuning for Text Classification. arXiv: 1801.06146 http://arxiv.org/abs/ 1801.06146 (2018).
[16] P. Malo, A. Sinha, P. Korhonen, et al. good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol. 65, 4, 782 – 796 (2014).
[17] D. Tan Araci, FinBERT: Financial Sentiment Analysis with Pre-trained Language Models. arXiv preprint arXiv: 1908.10063 (2019).
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
