An unsupervised three-dimensional medical image registration method based on shifted window Transformer and convolutional neural network_Journal of Biomedical Engineering

Authors：

 LI Yang , RUAN Chenmiao , RUAN Dongsheng

School of Computer Science and Technology (School of Artificial Intelligence), Zhejiang Sci-Tech University, Hangzhou 310018, P. R. China;

Corresponding?author：

LI Yang, Email: yangli@zstu.edu.cn

Keywords：

Medical image registration; Deep learning; Convolutional neural network; Shifted window Transformer

DOI：

10.7507/1001-5515.202502039

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Three-dimensional (3D) deformable image registration plays a critical role in 3D medical image processing. This technique aligns images from different time points, modalities, or individuals in 3D space, enabling the comparison and fusion of anatomical or functional information. To simultaneously capture the local details of anatomical structures and the long-range dependencies in 3D medical images, while reducing the high costs of manual annotations, this paper proposes an unsupervised 3D medical image registration method based on shifted window Transformer and convolutional neural network (CNN), termed Swin Transformer-CNN-hybrid network (STCHnet). In the encoder part, STCHnet uses Swin Transformer and CNN to extract global and local features from 3D images, respectively, and optimizes feature representation through feature fusion. In the decoder part, STCHnet utilizes Swin Transformer to integrate information globally, and CNN to refine local details, reducing the complexity of the deformation field while maintaining registration accuracy. Experiments on the information extraction from images (IXI) and open access series of imaging studies (OASIS) datasets, along with qualitative and quantitative comparisons with existing registration methods, demonstrate that the proposed STCHnet outperforms baseline methods in terms of Dice similarity coefficient (DSC) and standard deviation of the log-Jacobian determinant (SDlogJ), achieving improved 3D medical image registration performance under unsupervised conditions.

1.	Wei D, Ahmad S, Huo J, et al. SLIR: synthesis, localization, inpainting, and registration for image-guided thermal ablation of liver tumors. Med Image Anal, 2020, 65: 101763.
2.	Saad F, Frysch R, Saalfeld S, et al. Deformable 3D/3D CT-to-digital-tomosynthesis image registration in image-guided bronchoscopy interventions. Comput Biol Med, 2024, 171: 108199.
3.	Qiao M, Wang Y, Guo Y, et al. Temporally coherent cardiac motion tracking from cine MRI: traditional registration method and modern CNN method. Med Phys, 2020, 47(9): 4189-4198.
4.	Lu J, Jin R, Wang M, et al. A bidirectional registration neural network for cardiac motion tracking using cine MRI images. Comput Biol Med, 2023, 160: 107001.
5.	Bitarafan A, Nikdan M, Baghshah M S. 3D image segmentation with sparse annotation by self-training and internal registration. IEEE J Biomed Health Inform, 2020, 25(7): 2665-2672.
6.	Beekman C, van Beek S, Stam J, et al. Improving predictive CTV segmentation on CT and CBCT for cervical cancer by diffeomorphic registration of a prior. Med Phys, 2022, 49(3): 1701-1711.
7.	Jiang J, Hong J, Tringale K, et al. Progressively refined deep joint registration segmentation (ProRSeg) of gastrointestinal organs at risk: application to MRI and cone‐beam CT. Med Phys, 2023, 50(8): 4758-4774.
8.	Nenoff L, Ribeiro C O, Matter M, et al. Deformable image registration uncertainty for inter-fractional dose accumulation of lung cancer proton therapy. Radiother Oncol, 2020, 147: 178-185.
9.	Wang B, Wang D Q, Lin M S, et al. Accumulation of the delivered dose based on cone-beam CT and deformable image registration for non-small cell lung cancer treated with hypofractionated radiotherapy. BMC Cancer, 2020, 20: 1112.
10.	Bosma L S, Zachiu C, Denis de Senneville B, et al. Technical note: intensity-based quality assurance criteria for deformable image registration in image-guided radiotherapy. Med Phys, 2023, 50(9): 5715-5722.
11.	Chen X, Ravikumar N, Xia Y, et al. Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds. Med Image Anal, 2021, 74: 102228.
12.	Casamitjana A, Lorenzi M, Ferraris S, et al. Robust joint registration of multiple stains and MRI for multimodal 3D histology reconstruction: application to the Allen human brain atlas. Med Image Anal, 2022, 75: 102265.
13.	Balakrishnan G, Zhao A, Sabuncu M R, et al. Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging, 2019, 38(8): 1788-1800.
14.	Chen J, He Y, Frey E C, et al. Vit-V-Net: vision transformer for unsupervised volumetric medical image registration. arXiv preprint, 2021, arXiv: 2104.06468.
15.	Chen J, Frey E C, He Y, et al. Transmorph: transformer for unsupervised medical image registration. Med Image Anal, 2022, 82: 102615.
16.	Chen J, Liu Y, Wei S, et al. A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond. Med Image Anal, 2024: 103385.
17.	Luo W, Li Y, Urtasun R, et al. Understanding the effective receptive field in deep convolutional neural networks//Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), Red Hook: Curran Associates Inc, 2016: 4905-4913.
18.	Zeiler M D, Fergus R. Visualizing and understanding convolutional networks//Computer Vision-ECCV 2014: 13th European Conference, Zurich: Springer International Publishing, 2014: 818-833.
19.	Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv preprint, 2015, arXiv: 1511.07122.
20.	Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu: IEEE, 2017: 2881-2890.
21.	Richter M L, Pal C. Receptive field refinement for convolutional neural networks reliably improves predictive performance. arXiv preprint, 2022, arXiv: 2211.14487.
22.	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), Red Hook: Curran Associates Inc, 2017: 6000-6010.
23.	Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint, 2020, arXiv: 2010.11929.
24.	Li S, Sui X, Luo X, et al. Medical image segmentation using squeeze-and-expansion transformers. arXiv preprint, 2021, arXiv: 2105.09511.
25.	Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows//Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 10012-10022.
26.	Darzi F, Bocklitz T. A review of medical image registration for different modalities. Bioengineering, 2024, 11(8): 786.
27.	Avants B B, Epstein C L, Grossman M, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal, 2008, 12(1): 26-41.
28.	University College London. Information extraction from images dataset. (2022-01-29) [2023-10-25]. https://brain-development.org/ixi-dataset.
29.	Marcus D S, Wang T H, Parker J, et al. Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J Cogn Neurosci, 2007, 19(9): 1498-1507.
30.	Hering A, Hansen L, Mok T C W, et al. Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Trans Med Imaging, 2023, 42(3): 697-712.
31.	Avants B B, Tustison N, Song G. Advanced normalization tools (ANTS). Insight J, 2009, 2(365): 1-35.
32.	Dice L R. Measures of the amount of ecologic association between species. Ecology, 1945, 26(3): 297-302.
33.	Beg M F, Miller M I, Trouvé A, et al. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int J Comput Vis, 2005, 61: 139-157.
34.	Wang W, Xie E, Li X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions//Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 568-578.
35.	Chen Z, Zheng Y, Gee J C. TransMatch: a Transformer-based multilevel dual-stream feature matching network for unsupervised deformable image registration. IEEE Trans Med Imaging, 2024, 43(1): 15-27.
36.	Li Z, Li X, Fan J, et al. Non-iterative pyramid network for unsupervised deformable medical image registration//Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul: IEEE, 2024: 1956-1960.

1. Wei D, Ahmad S, Huo J, et al. SLIR: synthesis, localization, inpainting, and registration for image-guided thermal ablation of liver tumors. Med Image Anal, 2020, 65: 101763.
2. Saad F, Frysch R, Saalfeld S, et al. Deformable 3D/3D CT-to-digital-tomosynthesis image registration in image-guided bronchoscopy interventions. Comput Biol Med, 2024, 171: 108199.
3. Qiao M, Wang Y, Guo Y, et al. Temporally coherent cardiac motion tracking from cine MRI: traditional registration method and modern CNN method. Med Phys, 2020, 47(9): 4189-4198.
4. Lu J, Jin R, Wang M, et al. A bidirectional registration neural network for cardiac motion tracking using cine MRI images. Comput Biol Med, 2023, 160: 107001.
5. Bitarafan A, Nikdan M, Baghshah M S. 3D image segmentation with sparse annotation by self-training and internal registration. IEEE J Biomed Health Inform, 2020, 25(7): 2665-2672.
6. Beekman C, van Beek S, Stam J, et al. Improving predictive CTV segmentation on CT and CBCT for cervical cancer by diffeomorphic registration of a prior. Med Phys, 2022, 49(3): 1701-1711.
7. Jiang J, Hong J, Tringale K, et al. Progressively refined deep joint registration segmentation (ProRSeg) of gastrointestinal organs at risk: application to MRI and cone‐beam CT. Med Phys, 2023, 50(8): 4758-4774.
8. Nenoff L, Ribeiro C O, Matter M, et al. Deformable image registration uncertainty for inter-fractional dose accumulation of lung cancer proton therapy. Radiother Oncol, 2020, 147: 178-185.
9. Wang B, Wang D Q, Lin M S, et al. Accumulation of the delivered dose based on cone-beam CT and deformable image registration for non-small cell lung cancer treated with hypofractionated radiotherapy. BMC Cancer, 2020, 20: 1112.
10. Bosma L S, Zachiu C, Denis de Senneville B, et al. Technical note: intensity-based quality assurance criteria for deformable image registration in image-guided radiotherapy. Med Phys, 2023, 50(9): 5715-5722.
11. Chen X, Ravikumar N, Xia Y, et al. Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds. Med Image Anal, 2021, 74: 102228.
12. Casamitjana A, Lorenzi M, Ferraris S, et al. Robust joint registration of multiple stains and MRI for multimodal 3D histology reconstruction: application to the Allen human brain atlas. Med Image Anal, 2022, 75: 102265.
13. Balakrishnan G, Zhao A, Sabuncu M R, et al. Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging, 2019, 38(8): 1788-1800.
14. Chen J, He Y, Frey E C, et al. Vit-V-Net: vision transformer for unsupervised volumetric medical image registration. arXiv preprint, 2021, arXiv: 2104.06468.
15. Chen J, Frey E C, He Y, et al. Transmorph: transformer for unsupervised medical image registration. Med Image Anal, 2022, 82: 102615.
16. Chen J, Liu Y, Wei S, et al. A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyond. Med Image Anal, 2024: 103385.
17. Luo W, Li Y, Urtasun R, et al. Understanding the effective receptive field in deep convolutional neural networks//Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), Red Hook: Curran Associates Inc, 2016: 4905-4913.
18. Zeiler M D, Fergus R. Visualizing and understanding convolutional networks//Computer Vision-ECCV 2014: 13th European Conference, Zurich: Springer International Publishing, 2014: 818-833.
19. Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv preprint, 2015, arXiv: 1511.07122.
20. Zhao H, Shi J, Qi X, et al. Pyramid scene parsing network//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu: IEEE, 2017: 2881-2890.
21. Richter M L, Pal C. Receptive field refinement for convolutional neural networks reliably improves predictive performance. arXiv preprint, 2022, arXiv: 2211.14487.
22. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need//Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS), Red Hook: Curran Associates Inc, 2017: 6000-6010.
23. Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint, 2020, arXiv: 2010.11929.
24. Li S, Sui X, Luo X, et al. Medical image segmentation using squeeze-and-expansion transformers. arXiv preprint, 2021, arXiv: 2105.09511.
25. Liu Z, Lin Y, Cao Y, et al. Swin transformer: hierarchical vision transformer using shifted windows//Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 10012-10022.
26. Darzi F, Bocklitz T. A review of medical image registration for different modalities. Bioengineering, 2024, 11(8): 786.
27. Avants B B, Epstein C L, Grossman M, et al. Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal, 2008, 12(1): 26-41.
28. University College London. Information extraction from images dataset. (2022-01-29) [2023-10-25]. https://brain-development.org/ixi-dataset.
29. Marcus D S, Wang T H, Parker J, et al. Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J Cogn Neurosci, 2007, 19(9): 1498-1507.
30. Hering A, Hansen L, Mok T C W, et al. Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Trans Med Imaging, 2023, 42(3): 697-712.
31. Avants B B, Tustison N, Song G. Advanced normalization tools (ANTS). Insight J, 2009, 2(365): 1-35.
32. Dice L R. Measures of the amount of ecologic association between species. Ecology, 1945, 26(3): 297-302.
33. Beg M F, Miller M I, Trouvé A, et al. Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int J Comput Vis, 2005, 61: 139-157.
34. Wang W, Xie E, Li X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions//Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal: IEEE, 2021: 568-578.
35. Chen Z, Zheng Y, Gee J C. TransMatch: a Transformer-based multilevel dual-stream feature matching network for unsupervised deformable image registration. IEEE Trans Med Imaging, 2024, 43(1): 15-27.
36. Li Z, Li X, Fan J, et al. Non-iterative pyramid network for unsupervised deformable medical image registration//Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Seoul: IEEE, 2024: 1956-1960.

Journal of Biomedical Engineering

Latest ArticlesAn unsupervised three-dimensional medical image registration method based on shifted window Transformer and convolutional neural network

Abstract Full text Figures/Tables Video References Cited by

Format

Content