Evaluation Pitfalls in Post-Training Quantization of Diffusion Transformers: An Empirical Audit
DOI:
https://doi.org/10.54097/v6881m05Keywords:
Post-training Quantization, Diffusion Transformer, Evaluation Audit, Fréchet Inception Distance, CalibrationAbstract
Diffusion transformers are moving toward practical deployment, and post-training quantization (PTQ) has emerged as a standard, retraining-free compression route. The conventions used to evaluate PTQ for diffusion transformers—the Fréchet Inception Distance (FID) of images produced by a fixed sampling pipeline, with a cheap calibration-error statistic as a configuration-search proxy—are typically inherited from prior codebases rather than audited. Auditing this pipeline through a DiT-XL/2 (ImageNet 256×256) case study of 209 controlled experimental records, this paper identifies three systematic pitfalls. First, sampling-pipeline configuration: two latent defects in the legacy pipeline manufactured spurious method-level differences, and controlled A/B reproduction (deviation below 1%) shows that correcting them reverses earlier conclusions on channel equalization. Second, proxy-metric failure: 32 records sharing a bit-identical calibration error span an FID range of 98.74, and its rank correlation with FID is statistically indistinguishable from zero in every experiment family tested. Third, calibration-protocol sensitivity: under the corrected protocol, an error-aware dynamic-programming grouping strategy is statistically indistinguishable from a fixed-interval baseline (273.62±1.78 vs. 274.32±1.80), while group-wise equalization turns from apparently beneficial to harmful (273.62→313.23). The only method-level conclusion that survives the protocol correction—under three seeds, a single model, and the 8-bit weight-and-activation (W8A8) configuration studied—is that grouped quantization outperforms no grouping (388.96→274.32). This work closes with a corrected evaluation-protocol checklist.
Downloads
References
[1] Peebles, W., & Xie, S. (2023). Scalable Diffusion Models with Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV51070.2023.00449. DOI: https://doi.org/10.1109/ICCV51070.2023.00387
[2] Wu, J., Wang, H., Shang, Y., Shah, M., & Yan, Y. (2024). PTQ4DiT: Post-training Quantization for Diffusion Transformers. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). DOI: https://doi.org/10.52202/079017-2006
[3] Chen, L., Meng, Y., Tang, C., Ma, X., Jiang, J., Wang, X., Wang, Z., & Zhu, W. (2025). Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). DOI: https://doi.org/10.1109/CVPR52734.2025.02636
[4] Zhao, T., Fang, T., Liu, E., Rui, W., Soedarmadji, W., Li, S., Lin, Z., Dai, G., Yan, S., Yang, H., Ning, X., & Wang, Y. (2024). ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generation. arXiv preprint arXiv:2406.02540.
[5] Ding, N., Han, J., Tian, Y., Xu, C., Han, K., & Tang, Y. (2025). Post-Training Quantization for Diffusion Transformer via Hierarchical Timestep Grouping. arXiv preprint arXiv:2503.06930.
[6] Hwang, Y., Lee, H., & Kang, J. (2025). TQ-DiT: Efficient Time-Aware Quantization for Diffusion Transformers. arXiv preprint arXiv:2502.04056. DOI: https://doi.org/10.1109/IJCNN64981.2025.11228481
[7] Liu, W., & Zhang, S. Q. (2024). HQ-DiT: Efficient Diffusion Transformer with FP4 Hybrid Quantization. arXiv preprint arXiv:2405.19751.
[8] Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. In Proceedings of the International Conference on Machine Learning (ICML). https://doi.org/10.48550/arXiv.2211.10438.
[9] Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., & Keutzer, K. (2023). Q-Diffusion: Quantizing Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). https://doi.org/10.1109/ICCV51070.2023.00244. DOI: https://doi.org/10.1109/ICCV51070.2023.01608
[10] Parmar, G., Zhang, R., & Zhu, J.-Y. (2022). On Aliased Resizing and Surprising Subtleties in GAN Evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/CVPR52688.2022.01137. DOI: https://doi.org/10.1109/CVPR52688.2022.01112
[11] Stein, G., Cresswell, J. C., Hosseinzadeh, R., Sui, Y., Ross, B. L., Villecroze, V., Liu, Z., Caterini, A. L., Taylor, J. E. T., & Loaiza-Ganem, G. (2023). Exposing Flaws of Generative Model Evaluation Metrics and their Unfair Treatment of Diffusion Models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). DOI: https://doi.org/10.52202/075280-0165
[12] Obukhov, A., Seitzer, M., Lin, E. Y.-J., Kyl, J., Zhydenko, S., & willylulu. (2021). torch-fidelity (Version 0.3.0) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.4957738.
[13] Lee, D., Hur, J., Shon, H., Lee, J. Y., & Kim, J. (2025). DMQ: Dissecting Outliers of Diffusion Models for Post-Training Quantization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). DOI: https://doi.org/10.1109/ICCV51701.2025.01720
[14] Feng, W., Qin, H., Yang, C., Li, X., Yang, H., Li, Y., An, Z., Huang, L., Magno, M., & Xu, Y. (2025). S²Q-VDiT: Accurate Quantized Video Diffusion Transformer with Salient Data and Sparse Token Distillation. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). DOI: https://doi.org/10.52202/085713-1172
[15] Ryu, S., Salzmann, M., & Javed, S. (2026). Q-Drift: Quantization-Aware Drift Correction for Diffusion Model Sampling. arXiv preprint arXiv:2603.18095.
[16] Sørensen, M., Holm, F., Kristensen, A., Jensen, S., Madsen, L., Rasmussen, E., & Aline, C. (2025). Quantization as a Foundation for Deployable High Performance Diffusion Models within the Landscape of Large Scale Generative AI. TechRxiv preprint. https://doi.org/10.36227/techrxiv.175624440.05705133. DOI: https://doi.org/10.36227/techrxiv.175624440.05705133/v1
[17] Yang, K., Yang, K., Wu, B., Zhang, X., Yang, Q., Qin, H., Zhang, H., & Zhang, Y. (2025). TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search. arXiv preprint arXiv:2512.06353.
[18] Sun, R., Yang, T., Niu, W., & Sun, J. (2025). RSTR: Reducing SpatioTemporal Redundancy in Diffusion Transformers. arXiv preprint arXiv:2512.14096.
[19] Dufour, N., Efros, A. A., & Pérez, P. (2026). The FID Lottery: Quantifying Hidden Randomness in Generative Model Evaluation. arXiv preprint arXiv:2606.20536.
[20] Yang, L., Lin, H., Zhao, T., et al. (2025). LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation. arXiv preprint arXiv:2508.03485. DOI: https://doi.org/10.1109/TCSVT.2026.3723090
[21] Kim, D., Hwang, J., Oh, C., & Park, J. (2025). MixDiT: Accelerating Image Diffusion Transformer Inference with Mixed-Precision MX Quantization. arXiv preprint arXiv:2504.08398. DOI: https://doi.org/10.1109/LCA.2025.3560786
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Frontiers in Computing and Intelligent Systems
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
