Improvement and Application of Fusion Scheme in Automatic Medical Image Analysis

Subtitle Is Not Required, Please Write It Here If Your Article Has One


  • Minjun Liang
  • Mingyang Wei
  • Yanan Li
  • He Tian
  • Yabei Li



Image Fusion, Medical Image Analysis, Deep Learning, Multimodal image fusion.


The research in this paper provides generalization and new ideas for research topics in computer-assisted medicine. The main improvement efforts in deep learning-based multimodal fusion schemes, which provide alternative directions and robust feature fitting performance for fusion schemes, are building complex structures, migrating knowledge or experience, processing and enhancing data, and targeting features for semantic correction based on contextual features. At the application level, the brain, liver, and lungs are the main targets of scientific research, so this paper surveys related work and analyzes the reasons for performance gains. Taken together, deep learning-based image fusion schemes can assist physicians in understanding information about lesion sites, lesion types, and sizes, providing an important basis for developing personalized treatment plans, which is important for improving diagnosis and specifying precise treatment plans. Therefore, the investigation of medical image fusion schemes is promising and beneficial.


Download data is not yet available.
<br data-mce-bogus="1"> <br data-mce-bogus="1">


S.-Q. Deng, L.-J. Deng, X. Wu, R. Ran, D. Hong, and G. Vivone, "PSRT: Pyramid Shuffle-and-Reshuffle Transformer for Multispectral and Hyperspectral Image Fusion," IEEE Transactions on Geoscience and Remote Sensing, vol. 61, pp. 1-15, 2023.

L. Tang, J. Yuan, H. Zhang, X. Jiang, and J. Ma, "PIAFusion: A progressive infrared and visible image fusion network based on illumination aware," Information Fusion, vol. 83, pp. 79-92, 2022.

B. Meher, S. Agrawal, R. Panda, and A. Abraham, "A survey on region based image fusion methods," Information Fusion, vol. 48, pp. 119-132, 2019.

A. P. James and B. V. Dasarathy, "Medical image fusion: A survey of the state of the art," Information fusion, vol. 19, pp. 4-19, 2014.

R. Singh and A. Khare, "Fusion of multimodal medical images using Daubechies complex wavelet transform–A multiresolution approach," Information fusion, vol. 19, pp. 49-60, 2014.

H. Yin, Y. Li, Y. Chai, Z. Liu, and Z. Zhu, "A novel sparse-representation-based multi-focus image fusion approach," Neurocomputing, vol. 216, pp. 216-229, 2016.

Y. Liu, X. Chen, Z. Wang, Z. J. Wang, R. K. Ward, and X. Wang, "Deep learning for pixel-level image fusion: Recent advances and future prospects," Information Fusion, vol. 42, pp. 158-173, 2018.

M. D. Zeiler, D. Krishnan, G. W. Taylor, and R. Fergus, "Deconvolutional networks," in 2010 IEEE Computer Society Conference on computer vision and pattern recognition, 2010: IEEE, pp. 2528-2535.

B. Wohlberg, "Efficient algorithms for convolutional sparse representations," IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 301-315, 2015.

K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.

S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, "Aggregated residual transformations for deep neural networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1492-1500.

S. Zagoruyko and N. Komodakis, "Wide residual networks," arXiv preprint arXiv:1605.07146, 2016.

T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 2117-2125.

F. Milletari, N. Navab, and S.-A. Ahmadi, "V-net: Fully convolutional neural networks for volumetric medical image segmentation," in 2016 fourth international conference on 3D vision (3DV), 2016: Ieee, pp. 565-571.

X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, "H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes," IEEE transactions on medical imaging, vol. 37, no. 12, pp. 2663-2674, 2018.

G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.

H. Li and X.-J. Wu, "DenseFuse: A fusion approach to infrared and visible images," IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614-2623, 2018.

X. Song, X.-J. Wu, and H. Li, "MSDNet for medical image fusion," in International conference on image and graphics, 2019: Springer, pp. 278-288.

C. Zhao, T. Wang, and B. Lei, "Medical image fusion method based on dense block and deep convolutional generative adversarial network," Neural Computing and Applications, vol. 33, no. 12, pp. 6595-6610, 2021.

K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, "Domain separation networks," Advances in neural information processing systems, vol. 29, 2016.

F. Lahoud and S. Süsstrunk, "Zero-learning fast medical image fusion," in 2019 22th International Conference on Information Fusion (FUSION), 2019: IEEE, pp. 1-8.

H. Hermessi, O. Mourali, and E. Zagrouba, "Convolutional neural network-based multimodal image fusion via similarity learning in the shearlet domain," Neural Computing and Applications, vol. 30, no. 7, pp. 2029-2045, 2018.

Y. Wang, C. Wu, L. Herranz, J. van de Weijer, A. Gonzalez-Garcia, and B. Raducanu, "Transferring gans: generating images from limited data," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 218-234.

L. Perez and J. Wang, "The effectiveness of data augmentation in image classification using deep learning," arXiv preprint arXiv:1712.04621, 2017.

Y. Guo et al., "Segmenting hippocampus from infant brains by sparse patch matching with deep-learned features," in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2014: Springer, pp. 308-315.

Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, "Ccnet: Criss-cross attention for semantic segmentation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 603-612.

H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, "mixup: Beyond empirical risk minimization," arXiv preprint arXiv:1710.09412, 2017.

X. Liang, P. Hu, L. Zhang, J. Sun, and G. Yin, "MCFNet: Multi-layer concatenation fusion network for medical images fusion," IEEE Sensors Journal, vol. 19, no. 16, pp. 7107-7119, 2019.

Z. Liu et al., "Automatic diagnosis of fungal keratitis using data augmentation and image fusion with deep convolutional neural network," Computer Methods and Programs in Biomedicine, vol. 187, p. 105019, 2020.

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1125-1134.

A. Vaswani et al., "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.

Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, "Transformer-xl: Attentive language models beyond a fixed-length context," arXiv preprint arXiv:1901.02860, 2019.

V. VS, J. M. J. Valanarasu, P. Oza, and V. M. Patel, "Image Fusion Transformer," arXiv preprint arXiv:2107.09011, 2021.

L. Qu, S. Liu, M. Wang, and Z. Song, "TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework using Self-Supervised Multi-Task Learning," arXiv preprint arXiv:2112.01030, 2021.

L. Qu et al., "TransFuse: A Unified Transformer-based Image Fusion Framework using Self-supervised Learning," arXiv preprint arXiv:2201.07451, 2022.

A. L. Da Cunha, J. Zhou, and M. N. Do, "The nonsubsampled contourlet transform: theory, design, and applications," IEEE transactions on image processing, vol. 15, no. 10, pp. 3089-3101, 2006.

Z. Wang, X. Li, H. Duan, Y. Su, X. Zhang, and X. Guan, "Medical image fusion based on convolutional neural networks and non-subsampled contourlet transform," Expert Systems with Applications, vol. 171, p. 114574, 2021.

S. Goyal, V. Singh, A. Rani, and N. Yadav, "Multimodal image fusion and denoising in NSCT domain using CNN and FOTGV," Biomedical Signal Processing and Control, vol. 71, p. 103214, 2022.

D. S. Shibu and S. S. Priyadharsini, "Multi scale decomposition based medical image fusion using convolutional neural network and sparse representation," Biomedical Signal Processing and Control, vol. 69, p. 102789, 2021.

A. İ. Abas, H. E. Koçer, and N. A. Baykan, "Medical image fusion with convolutional neural network in multiscale transform domain," TURKISH JOURNAL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCES, 2021.

B. Lei, S. Chen, D. Ni, and T. Wang, "Discriminative learning for Alzheimer's disease diagnosis via canonical correlation analysis and multimodal fusion," Frontiers in aging neuroscience, vol. 8, p. 77, 2016.

O. B. Ahmed, J. Benois-Pineau, M. Allard, G. Catheline, C. B. Amar, and A. s. D. N. Initiative, "Recognition of Alzheimer's disease and Mild Cognitive Impairment with multimodal image-derived biomarkers and Multiple Kernel Learning," Neurocomputing, vol. 220, pp. 98-110, 2017.

A. D. Algarni, "Automated medical diagnosis system based on multi-modality image fusion and deep learning," Wireless Personal Communications, vol. 111, no. 2, pp. 1033-1058, 2020.

T. D. Vu, H.-J. Yang, V. Q. Nguyen, A.-R. Oh, and M.-S. Kim, "Multimodal learning using convolution neural network and Sparse Autoencoder," in 2017 IEEE International Conference on Big Data and Smart Computing (BigComp), 2017: IEEE, pp. 309-312.

J. Ma, H. Xu, J. Jiang, X. Mei, and X.-P. Zhang, "DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion," IEEE Transactions on Image Processing, vol. 29, pp. 4980-4995, 2020.

Z. Shi, C. Zhang, D. Ye, P. Qin, R. Zhou, and L. Lei, "MMI-Fuse: Multimodal Brain Image Fusion With Multiattention Module," IEEE Access, vol. 10, pp. 37200-37214, 2022.

R. Nandhini Abirami, P. Durai Raj Vincent, K. Srinivasan, K. S. Manic, and C.-Y. Chang, "Multimodal Medical Image Fusion of Positron Emission Tomography and Magnetic Resonance Imaging Using Generative Adversarial Networks," Behavioural Neurology, vol. 2022, 2022.

M. G. Reddy, P. V. N. Reddy, and P. R. Reddy, "Segmentation of fused MR and CT images using DL-CNN with PGK and NLEM filtered AACGK-FCM," Biomedical Signal Processing and Control, vol. 68, p. 102618, 2021.

S. Li, Y. Xie, G. Wang, L. Zhang, and W. Zhou, "Attention guided discriminative feature learning and adaptive fusion for grading hepatocellular carcinoma with Contrast-enhanced MR," Computerized Medical Imaging and Graphics, vol. 97, p. 102050, 2022.

Y. Fu et al., "Fusion of 3D lung CT and serum biomarkers for diagnosis of multiple pathological types on pulmonary nodules," Computer Methods and Programs in Biomedicine, vol. 210, p. 106381, 2021.

Y. Zhang, J. Zhao, Y. Qiang, X. Yang, W. Wu, and L. Jia, "Improved heterogeneous data fusion and multi‐scale feature selection method for lung cancer subtype classification," Concurrency and Computation: Practice and Experience, vol. 34, no. 1, p. e6535, 2022.




How to Cite

Liang, M., Wei, M., Li, Y., Tian, H., & Li, Y. (2023). Improvement and Application of Fusion Scheme in Automatic Medical Image Analysis: Subtitle Is Not Required, Please Write It Here If Your Article Has One. Academic Journal of Science and Technology, 5(3), 225–230.