Quality Control and Interannotator Agreement in Manual Annotation of Damage in Chinese Paintings and Calligraphy
DOI:
https://doi.org/10.54097/rneeea82Keywords:
Chinese Paintings and Calligraphy, Damage Annotation, Quality Control, Independent Double Annotation, LabelmeAbstract
Manual annotation of damage in images of Chinese paintings and calligraphy involves locating damaged areas, assigning damage categories, and delineating their boundaries. Annotators may disagree over the category or extent of the same feature, affecting both AI training data and documentation made before conservation. Drawing on standards for paper-based cultural heritage and image-annotation practice, this methods paper proposes a workflow comprising common annotation rules, independent annotation by two people, discrepancy detection, expert adjudication, and retention of revision records. Using labelme as an example, it describes how to retain two independent annotation files, check for omissions and incorrect categories, and assess agreement with category agreement, intersection over union (IoU), and the Dice coefficient. Five working labels, four operational tables, and three illustrative figures support implementation. The figures and numerical examples are explanatory and are not measured outcomes from collection objects.
Downloads
References
[1] National Cultural Heritage Administration. (2010). WW/T 0026—2010, Classification and Illustrations of Deterioration in Museum Collections of Paper-Based Cultural Heritage. (In Chinese).
[2] State Administration for Market Regulation, & Standardization Administration of China. (2024). GB/T 44208—2024, Basic Elements for Description and Illustration of Deterioration in Museum Collections. (In Chinese).
[3] Deng, X., Li, Z., Zhang, Y., et al. (2025). Application of AI-based object detection to damage surveys of painting and calligraphy artifacts. Journal of Northwest University (Natural Science Edition), 55(1), 98–105. https://doi.org/10.16152/j.cnki.xdxbzr.2025-01-008. (In Chinese).
[4] National Cultural Heritage Administration. (2010). WW/T 0027—2010, Specifications for Conservation and Restoration Records of Paper-Based Cultural Heritage in Museum Collections. (In Chinese).
[5] State Administration for Market Regulation, & Standardization Administration of China. (2023). GB/T 42755—2023, Artificial Intelligence: Data Annotation Procedures for Machine Learning. (In Chinese).
[6] Le, P., Liu, R., Shangguan, B., et al. (2023). GeoAI samples: Models, quality, and services. Geomatics and Information Science of Wuhan University, 48(10), 1616–1631. https://doi.org/10.13203/j.whugis20230125. (In Chinese).
[7] Zhang, N., Zhang, Q., Feng, W., et al. (2017). A damage identification system for ancient murals and its application at the Mogao Grottoes. Dunhuang Research, (2), 135–140. (In Chinese).
[8] Deng, X., Yu, Y., Yang, L., et al. (2023). Automatic delineation of damaged areas in Ming-dynasty murals at Zhilin Temple in Jianshui. Journal of Computer-Aided Design and Computer Graphics, 35(10), 1554–1565. https://doi.org/10.3724/SP.J.1089.2023.19738. (In Chinese). DOI: https://doi.org/10.3724/SP.J.1089.2023.19738
[9] Zhao, H., Yu, Y., Chen, A., et al. (2025). Detection of damage in ancient murals using a residual dual-channel attention U-Net. Journal of Computer-Aided Design and Computer Graphics, 37(6), 1040–1052. https://doi.org/10.3724/SP.J.1089.2023-00430. (In Chinese). DOI: https://doi.org/10.3724/SP.J.1089.2023-00430
[10] Zhang, Z., Shui, W., Zhou, M., et al. (2021). Damage extraction and virtual restoration algorithms for digital murals. Application Research of Computers, 38(8), 2495–2498, 2504. (In Chinese).
[11] Wada, K. (n.d.). labelme: Image annotation with Python. Retrieved September 25, 2026. https://github.com/wkentaro/labelme.
[12] Russell, B. C., Torralba, A., Murphy, K. P., et al. (2008). LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173. https://doi.org/10.1007/s11263-007-0090-8. DOI: https://doi.org/10.1007/s11263-007-0090-8
[13] Lin, T. Y., Maire, M., Belongie, S., et al. (2014). Microsoft COCO: Common objects in context. In Computer Vision—ECCV 2014 (pp. 740–755). Springer. https://doi.org/10.1007/978-3-319-10602-1_48. DOI: https://doi.org/10.1007/978-3-319-10602-1_48
[14] Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104. DOI: https://doi.org/10.1177/001316446002000104
[15] Jaccard, P. (1912). The distribution of the flora in the alpine zone. New Phytologist, 11(2), 37–50. https://doi.org/10.1111/j.1469-8137.1912.tb05611.x. DOI: https://doi.org/10.1111/j.1469-8137.1912.tb05611.x
[16] Dice, L. R. (1945). Measures of the amount of ecologic association between species. Ecology, 26(3), 297–302. https://doi.org/10.2307/1932409. DOI: https://doi.org/10.2307/1932409
[17] Taha, A. A., & Hanbury, A. (2015). Metrics for evaluating 3D medical image segmentation: Analysis, selection, and tool. BMC Medical Imaging, 15, 29. https://doi.org/10.1186/s12880-015-0068-x. DOI: https://doi.org/10.1186/s12880-015-0068-x
[18] Maier-Hein, L., Reinke, A., Godau, P., et al. (2024). Metrics reloaded: Recommendations for image analysis validation. Nature Methods, 21(2), 195–212. https://doi.org/10.1038/s41592-023-02151-z. DOI: https://doi.org/10.1038/s41592-023-02151-z
[19] Sambasivan, N., Kapania, S., Highfill, H., et al. (2021). “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes AI. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3411764.3445518. DOI: https://doi.org/10.1145/3411764.3445518
[20] Northcutt, C. G., Athalye, A., & Mueller, J. (2021). Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Frontiers in Computing and Intelligent Systems
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
