Multimodal Violin Technique Recognition with Audio and Skeleton Features
DOI:
https://doi.org/10.54097/2y6nmp35Keywords:
Violin Technique Recognition, Multimodal Learning, Audio-visual Analysis, Skeleton Features, Music Education TechnologyAbstract
Automatic analysis of violin performance can support music education by helping students and teachers identify playing techniques from ordinary practice videos. This paper focuses on violin technique recognition rather than full automated grading. A simple multimodal framework is used to extract audio descriptors from the soundtrack and skeleton-motion descriptors from video frames, then combine the two modalities for single-label technique classification. The current study considers six dominant technique categories: pizzicato, stopped bowing, plain playing, vibrato, double stops, and spiccato. Experiments on 147 labeled clips show that multimodal fusion achieves 96.60% accuracy and 96.43% macro-F1 under stratified clip-level three-fold cross-validation. A source-video-grouped audit gives 95.92% accuracy and 96.08% macro-F1. The results provide preliminary evidence that audio and skeleton cues are complementary for violin technique recognition.
Downloads
References
[1] Zhang, H., Cheung, V. K. M., Nishioka, H., Dixon, S., & Furuya, S. (2025). LLaQo: Towards a query-based coach in expressive music performance assessment. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). DOI: https://doi.org/10.1109/ICASSP49660.2025.10890522
[2] Schedl, M., Gomez, E., & Urbano, J. (2014). Music information retrieval: Recent developments and applications. Foundations and Trends in Information Retrieval, 8(2–3), 127–261. DOI: https://doi.org/10.1561/1500000042
[3] Tzanetakis, G., & Cook, P. (2002). Musical genre classification of audio signals. IEEE Transactions on Speech and Audio Processing, 10(5), 293–302. DOI: https://doi.org/10.1109/TSA.2002.800560
[4] Benetos, E., Dixon, S., Giannoulis, D., Kirchhoff, H., & Klapuri, A. (2013). Automatic music transcription: Challenges and future directions. Journal of Intelligent Information Systems, 41(3), 407–434. DOI: https://doi.org/10.1007/s10844-013-0258-3
[5] Lugaresi, C., et al. (2019). MediaPipe: A framework for building perception pipelines. CoRR, abs/1906.08172.
[6] Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.-L., & Grundmann, M. (2020). MediaPipe Hands: On-device real-time hand tracking. CoRR, abs/2006.10214.
[7] Bazarevsky, V., Grishchenko, I., Raveendran, K., Zhu, T., Zhang, F., & Grundmann, M. (2020). BlazePose: On-device real-time body pose tracking. CoRR, abs/2006.10204.
[8] Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017). Realtime multi-person 2D pose estimation using part affinity fields. In Proceedings of the CVPR (pp. 7291–7299). DOI: https://doi.org/10.1109/CVPR.2017.143
[9] Yoo, S. J., Shrestha, S., Muresanu, I., & Fermuller, C. (2025). VioPose: Violin performance 4D pose estimation by hierarchical audiovisual inference. In Proceedings of the WACV. DOI: https://doi.org/10.1109/WACV61041.2025.00482
[10] Lucena, R., & Ramirez, R. (2024). A machine learning approach to gesture detection in violin performance. In Proceedings of Audio Mostly. DOI: https://doi.org/10.1145/3678299.3678313
[11] Dalmazzo, D., & Ramirez, R. (2019). Bowing gestures classification in violin performance: A machine learning approach. Frontiers in Psychology, 10, 344. DOI: https://doi.org/10.3389/fpsyg.2019.00344
[12] Wilson, W., Granieri, N., & Ali-MacLachlan, I. (2023). Ubiquitous multimodality as a tool in violin performance classification. In Proceedings of the International Symposium on the Internet of Sounds. DOI: https://doi.org/10.1109/IEEECONF59510.2023.10335435
[13] Muhammed, Z., et al. (2024). Ensemble of multimodal deep learning models for violin bowing techniques classification. Journal of Advances in Information Technology, 15(1), 40–48. DOI: https://doi.org/10.12720/jait.15.1.40-48
[14] Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32. DOI: https://doi.org/10.1023/A:1010933404324
[15] Pedregosa, F., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Frontiers in Computing and Intelligent Systems
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
