Design of Real-time Pedestrian Trajectory Prediction System based on Jetson Xavier

Quankai Liu; Haifeng Sang

doi:10.54097/fcis.v4i3.11156

Authors

Quankai Liu
Haifeng Sang

DOI:

https://doi.org/10.54097/fcis.v4i3.11156

Keywords:

Pedestrian Trajectory Prediction, Attention Mechanism, Embedded Deployment, Automatic Driving

Abstract

This paper presents a vehicle-mounted real-time pedestrian trajectory prediction system based on the embedded device Jetson Xavier. It achieves low-cost real-time pedestrian trajectory prediction using only the front camera of the vehicle. Firstly, pedestrian detection and tracking are implemented based on YoLoV7, while also estimating pedestrian poses and optical flow to provide multiple information sequences for the trajectory prediction network. Secondly, the pedestrian trajectory algorithm from a driver's perspective is studied, and a trajectory prediction algorithm that considers pedestrian pose, optical flow, and trajectory information is proposed. A novel multi-information fusion network is designed to better integrate multiple features. The algorithm is tested on the JAAD and PIE datasets, and the displacement errors are reduced by 6.35% and 3.28%, respectively, compared to BiTraP. Finally, the algorithm is ported to the embedded device Xavier and installed on a simulated vehicle for testing. By predicting pedestrian future trajectories based on pedestrian detection, collisions can be avoided in advance, improving the safety of autonomous driving. The proposed system has significant practical value.

Downloads

Download data is not yet available.

References

Rudenko A, Palmieri L, Herman M, et al. Human motion trajectory prediction: A survey[J]. The International Journal of Robotics Research, 2020, 39(8): 895-935.

Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics[J]. Physical review E, 1995,51(5):4282-4286.

Ramin Mehran, Alexis Oyama, and Mubarak Shah. Abnormal crowd behavior detection using social force model[C]. IEEE Conference on Computer Vision and Pattern Recognition,2009: 935–942.

Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You’ll never walk alone: Modeling social behavior for multi-target tracking[C]. IEEE International Conference on Computer Vision and Pattern Recognition, 2009:261–268.

Kota Yamaguchi, Alexander C Berg, Luis E Ortiz, and Tamara L Berg. Who are you with and where are you going? [C], IEEE International Conference on Computer Vision and Pattern Recognition, 2011:1345–1352.

Gianluca Antonini, Michel Bierlaire, and Mats Weber. Discrete choice models of pedestrian walking behavior[J]. Transportation Research Part B: Methodological, 2006, 40(8): 667–687.

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks[C]. IEEE Conference on Computer Vision and Pattern Recognition,2018:2255–2264.

Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds[C]. IEEE International Conference on Robotics and Automation, 2018:1–7.

Tharindu Fernando, Simon Denman, Sridha Sridharan, and Clinton Fookes. Soft+ hardwired attention: An LSTM framework for human trajectory prediction and abnormal event detection[J]. Neural networks, 2018,108:466–478.

Mohamed A, Qian K, Elhoseiny M, et al. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction[C]. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 1442.

Wang C, Cai S, Tan G. Graphtcn: Spatio-temporal interaction modeling for human trajectory prediction[C]. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021: 3450-3459.

Yagi T, Mangalam K, Yonetani R, Sato Y. Future Person Localization in First-Person Videos[C]. IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake: IEEE, 2018: 7593-7602.ong Beach: IEEE. 2019: 2960-2963.

Wang C Y, Bochkovskiy A, Liao H Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[J]. arXiv preprint arXiv:2207.02696, 2022.

Cao Z, Simon T, Wei SE, Sheikh Y. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields[C]. IEEE/CVF Conference on Computer Vision and Pattern Recognition. Hawaii: IEEE, 2017:1302-1310.

Xu H, Zhang J, Cai J, Rezatofighi H, Tao D. GMFlow:Learning Optical Flow via Global Matching[C]. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE. 2022: 8111-8120.

Vaswani A, Shazeer N, Parmar N, et al. Attention Is All You Need[C]. Conference and Workshop on Neural Information Processing Systems. California: MIT Press,2017: 5998-6008.

Yao Y, Atkins E, Johnson-Roberson M, et al. BiTraP:Bi-Directional Pedestrian Trajectory Prediction With Multi Modal Goal Estimation[J]. IEEE Robotics and Automation Letters, 2021,6(2): 1463-1470.

K. Ramanathan-V. Robicquet A. Li FF. Savarese S. Alahi, A. Goel. Social lstm: Human trajectory prediction in crowded spaces. In IEEE Conference on Computer Vision and Pattern Recognition, pages 961-971, 2016.

MZ. Wang YC. Crandall DJ. Atkins EM. Yao, Y. Xu. Unsupervised traffic accident detection in first-person videos. In IEEE International Conference on Intelligent Robots and Systems., pages 273–280, 2019.

I. Kunic-T. Tsotsos J. Rasouli, A. Kotseruba. Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV)., pages 6261–6270, 2019.

O Bideau. Halawa, M. Hellwich. Action-based contrastive learning for trajectory prediction. In European Conference on Computer Vision, pages 143–159, 2022.