Transformer Applying in the Time Series Prediction
DOI:
https://doi.org/10.54097/82pw7746Keywords:
Time Series, Transformer, Paradigms.Abstract
Time series prediction is of great significance in various fields such as finance, transportation, and biomedical science. The Transformer architecture and its derivative models, based on their reliance on long-distance relationships and their ability to extract and model complex information, have been widely adopted and deeply studied in time series prediction tasks in recent years, and have achieved considerable results. However, regarding the rapid development and innovation of Transformer models in the field of time series, there has not yet been a systematic classification and comprehensive summary of the research conducted in recent years. This paper reviews the time series prediction methods based on Transformer, and classifies the existing methods into three main paradigms: the widely studied and referenced classic paradigms (such as Informer, PatchTST, Cross former), the innovative Transformer paradigms (integrating signal theory, block theory, and architectural improvements), and the combination paradigms of time series and large language models. The article systematically analyzes the core innovation points, unique mechanisms, and performance levels of various models and directions, and also discusses the limitations of current methods and future research directions. This paper is helpful for researchers and practitioners to gain a systematic reference for understanding and applying time series prediction models based on the Transformer.
References
[1] Bartholomew D J. Time series analysis forecasting and control. 1971.
[2] Taylor S J, Letham B. Forecasting at scale. The American Statistician, 2018, 72 (1): 37 – 45.
[3] Williams R J, Zipser D. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1989, 1 (2): 270 – 280.
[4] Hochreiter S, Schmid Huber J. Long short-term memory. Neural Computation, 1997, 9 (8): 1735 – 1780.
[5] Cho K, Van Merriënboer B, Bahdanau D, Bengio Y. On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint, 2014. arXiv: 1409.1259.
[6] Sutskever I, Vinyals O, Le Q V. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 2014, 27.
[7] Lai G, Chang W C, Yang Y, Liu H. Modeling long-and short-term temporal patterns with deep neural networks. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, 2018: 95 – 104.
[8] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A N, et al. Attention is all you need. Advances in Neural Information Processing Systems, 2017, 30.
[9] Nie Y, Nguyen N H, Sinthong P, Kalagnanam J. A time series is worth 64 words: long-term forecasting with transformers. arXiv preprint, 2022. arXiv: 2211.14730.
[10] Zhang Y, Yan J. Crossformer: transformer utilizing cross-dimension dependency for multivariate time series forecasting. In: The Eleventh International Conference on Learning Representations, 2023.
[11] Kojima T, Gu S S, Reid M, Matsuo Y, Iwasawa Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, 2022, 35: 22199 – 22213.
[12] Geva M, Schuster R, Berant J, Levy O. Transformer feed-forward layers are key-value memories. arXiv preprint, 2020. arXiv: 2012.14913.
[13] Zhou H, Zhang S, Peng J, Zhang S, Li J, Xiong H, Zhang W. Informer: beyond efficient transformer for long sequence time-series forecasting. In: Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35 (12): 11106 – 11115.
[14] Zhou T, Ma Z, Wen Q, Wang X, Sun L, Jin R. Fed former: frequency enhanced decomposed transformer for long-term series forecasting. In: International Conference on Machine Learning, 2022: 27268 – 27286. PMLR.
[15] Wu H, Xu J, Wang J, Long M. Autoformer: decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 2021, 34: 22419 – 22430.
[16] Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Entertainer T, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint, 2020. arXiv: 2010.11929.
[17] Gong Z, Tang Y, Liang J. Patchmixer: a patch-mixing architecture for long-term time series forecasting. arXiv preprint, 2023. arXiv: 2310.00655.
[18] Chen P, Zhang Y, Cheng Y, Shu Y, Wang Y, Wen Q, et al. Pathformer: multi-scale transformers with adaptive pathways for time series forecasting. arXiv preprint, 2024. arXiv: 2402.05956.
[19] Liu Y, Hu T, Zhang H, Wu H, Wang S, Ma L, Long M. itransformer: inverted transformers are effective for time series forecasting. arXiv preprint, 2023. arXiv: 2310.06625.
[20] Zeng A, Chen M, Zhang L, Xu Q. Are transformers effective for time series forecasting? In: Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37 (9): 11121 – 11128.
[21] Tashiro Y, Song J, Song Y, Ermon S. CSDI: conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 2021, 34: 24804 – 24816.
[22] Bian Y, Ju X, Li J, Xu Z, Cheng D, Xu Q. Multi-patch prediction: adapting language models for time series representation learning. In: Forty-first International Conference on Machine Learning, 2024.
[23] Liu Y, Qin G, Huang X, Wang J, Long M. AutoTimes: autoregressive time series forecasters via large language models. arXiv preprint, 2024. arXiv: 2402.02370.
[24] Ansari A F, Stella L, Turkmen C, Zhang X, Mercado P, Shen H, Wang Y, et al. Chronos: learning the language of time series. arXiv preprint, 2024. arXiv: 2403.07815.
[25] Tan M, Merrill M, Gupta V, Althoff T, Hartvigsen T. Are language models actually useful for time series forecasting? Advances in Neural Information Processing Systems, 2024, 37: 60162 – 60191.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
