Multivariate Stock Time-Series Forecasting Using an Improved Transformer
DOI:
https://doi.org/10.54097/9cvcyp93Keywords:
Financial time series; stock-price forecasting; Transformer; feature gating; multivariate forecasting.Abstract
As financial markets become increasingly digital and intelligent, stock-price forecasting has emerged as an important topic in financial engineering, artificial intelligence, and data mining. Stock prices are shaped by firm fundamentals, macroeconomic conditions, investor sentiment, and market policies, while also exhibiting strong nonlinearity, nonstationarity, and noise. Extracting useful information from historical financial data and developing accurate and stable forecasting models therefore remain challenging. This study proposes an improved Transformer for multivariate stock time-series forecasting that combines feature-gated embedding with a residual prediction head. Daily data for Apple (AAPL), Microsoft (MSFT), and Google (GOOG) were obtained from the public YFinance interface for the period from January 1, 2000, to January 1, 2020. The model uses eight features—Open, High, Low, Close, Volume, RSI, ATR, and MACD—from the previous 30 trading days to predict the closing price five trading days ahead. A strictly chronological split, training-set-only normalization, and separate sliding-window construction are adopted to prevent look-ahead leakage. The proposed model is compared with Persistence, residual XGBoost, LSTM, a vanilla Transformer, and a standard feature-gated Transformer through the main experiment, cross-stock generalization tests, an ablation study, and a supplementary return-prediction experiment. On AAPL, the model achieves an RMSE of 1.2712, an MAE of 0.9242, and a MAPE of 2.5268%, slightly outperforming Persistence and residual XGBoost. It performs nearly identically to Persistence on GOOG and is slightly weaker than Persistence and residual XGBoost on MSFT, while consistently outperforming the LSTM and Transformer-based deep-learning baselines. These results demonstrate the accuracy and stability of the proposed model.
References
[1] Box G E P, Jenkins G M, Reinsel G C, Ljung G M. Time Series Analysis: Forecasting and Control[M]. 5th ed. Hoboken: Wiley, 2015.
[2] Bollerslev T. Generalized autoregressive conditional heteroskedasticity[J]. Journal of Econometrics, 1986, 31(3): 307-327.
[3] Chen T, Guestrin C. XGBoost: A scalable tree boosting system[C] //Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016: 785-794.
[4] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[5] Fischer T, Krauss C. Deep learning with long short-term memory networks for financial market predictions[J]. European Journal of Operational Research, 2018, 270(2): 654-669.
[6] Gu S, Kelly B, Xiu D. Empirical asset pricing via machine learning[J]. The Review of Financial Studies, 2020, 33(5): 2223-2273.
[7] Zhang Z, Zohren S, Roberts S. DeepLOB: Deep convolutional neural networks for limit order books[J]. IEEE Transactions on Signal Processing, 2019, 67(11): 3001-3012.
[8] Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] //Advances in Neural Information Processing Systems. 2017, 30: 5998-6008.
[9] Qin Y, Song D, Chen H, et al. A dual-stage attention-based recurrent neural network for time series prediction[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence. 2017: 2627-2633.
[10] Lim B, Arik S O, Loeff N, Pfister T. Temporal Fusion Transformers for interpretable multi-horizon time series forecasting[J]. International Journal of Forecasting, 2021, 37(4): 1748-1764.
[11] Zhou H, Zhang S, Peng J, et al. Informer: Beyond efficient Transformer for long sequence time-series forecasting[C] //Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(12): 11106-11115.
[12] Wu H, Xu J, Wang J, Long M. Autoformer: Decomposition Transformers with auto-correlation for long-term series forecasting[C]//Advances in Neural Information Processing Systems. 2021, 34: 22419-22430.
[13] Zhou T, Ma Z, Wen Q, et al. FEDformer: Frequency enhanced decomposed Transformer for long-term series forecasting[C] //Proceedings of the 39th International Conference on Machine Learning. 2022: 27268-27286.
[14] Zeng A, Chen M, Zhang L, Xu Q. Are Transformers effective for time series forecasting?[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(9): 11121-11128.
[15] [15] Wen Q, Zhou T, Zhang C, et al. Transformers in time series: A survey[C]//Proceedings of the 32nd International Joint Conference on Artificial Intelligence. 2023: 6778-6786.
[16] Nie Y, Nguyen N H, Sinthong P, Kalagnanam J. A time series is worth 64 words: Long-term forecasting with Transformers[C]//International Conference on Learning Representations. 2023.
[17] Wu H, Hu T, Liu Y, et al. TimesNet: Temporal 2D-variation modeling for general time series analysis[C] //International Conference on Learning Representations. 2023.
[18] Liu Y, Hu T, Zhang H, et al. iTransformer: Inverted Transformers are effective for time series forecasting[C] //International Conference on Learning Representations. 2024.
[19] Wang C, Chen Y, Zhang S, Zhang Q. Stock market index prediction using deep Transformer model[J]. Expert Systems with Applications, 2022, 208: 118128.
[20] Zhang Q, Qin C, Zhang Y, et al. Transformer-based attention network for stock movement prediction[J]. Expert Systems with Applications, 2022, 202: 117239.
[21] Peng Z Y, Guo P C. A data organization method for LSTM and Transformer when predicting Chinese banking stock prices[J]. Discrete Dynamics in Nature and Society, 2022: 7119678.
[22] Ma T, Wang W, Chen Y. Attention is all you need: An interpretable Transformer-based asset allocation approach[J]. International Review of Financial Analysis, 2023, 90: 102876.
[23] Thwal C M, Tun Y L, Kim K, Park S B, Hong C S. Transformers with attentive federated aggregation for time series stock forecasting[C] //37th International Conference on Information Networking. IEEE, 2023: 499-504.
[24] Hu J, Shen L, Sun G. Squeeze-and-excitation networks[C] //Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 7132-7141.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
