Multimodality Medical Image Fusion: Applications in Congenital Cardiology

KEYWORDS :

• congenital cardiology
• image fusion
• image registration
• medical image computing

Image fusion: definition

Image fusion consists of registering and visualizing different sets of images of a given object or scene. In medical disorders, pathology affects different tissues in a variable manner. These are best imaged in different modalities using diverse technologies. Image fusion helps to integrate all of the information and aids in accurate diagnosis and treatment. The implicit image fusion composed by the treating physician can be subjective and prone to error, which image fusion systems can help to mitigate. Medical image fusion is a very broad field covering aspects of computer vision, pattern recognition, machine learning and artificial intelligence in addition to image processing.

Brief history

The earliest mention of the word ‘image fusion’ in the clinical context was in 1992 by Swayne, who reported on computer-assisted fusion of single-photon emission computed tomographic (SPECT) and computed tomographic (CT) images in localizing inflammatory processes in 10 patients [1]. A combination of standard scaling, rotation, translation and stretching transformations were used to fuse images with anatomic and external landmarks. Stereotactic neurosurgery [2] was the next major application for image fusion, closely followed by oncology [3]. Rapid improvements in computers and their processing power led to increasing development and use of image fusion, particularly in image-guided surgery in a variety of medical specialties. The problems with software-based posthoc image fusion have led to the development of combined scanners, which acquire both anatomical and functional images during a single imaging session. This is achieved by fusing of technologies such as PET/CT scanner and PET/MRI scanner [4]. This also stimulated the development of co-location of complementary imaging modalities such as MRI and fluoroscopy. Integrated visualization systems for surgical planning and guidance using image fusion of structural and functional images (T1- and T2-weighted MRI, MR angiography and functional MRI) have been developed for neurosurgical procedures [5]. Preoperative data augment interventional imaging to expedite tissue characterization and precise localization and targeting. As surgery progresses and anatomical changes subsequently reduce the relevance of preoperative data, interventional data are refreshed for software navigation in true real time.

Image fusion in cardiology

One of the earliest application of image fusion in cardiology was in cardiovascular MRI. Short-axis cardiac MR images were fused with long-axis images into an isotropic volume. These were used for the measurement of ventricular volumes and function [6]. Other modalities were then included and integration of multimodal biomedical images of the heart (PET, SPECT, MRI and echocardiography) were attempted with clinically acceptable results.

• Image fusion in coronary imaging

Imaging of the coronary vasculature is of great importance in cardiology and fusion of images of the coronary anatomy has been performed extensively. Fusion of images obtained by coronary angiography with myocardial perfusion SPECT images has helped to understand the relationship between myocardial ischemia and corresponding vessels. This kind of ‘cardiac-hybrid imaging’ has been shown by a number of studies to increase the diagnostic accuracy and to be of synergistic clinical value.

3D reconstruction of coronary arteries has been performed by fusing images obtained by intravascular ultrasound with biplane angiography in the catheterization laboratory to allow accurate reconstruction of vessel anatomy [7].

3D computerized tomographic angiogram (3D-CTA) has been fused with color 3D functional surface map of 3D systolic wall thickening (3D-SWT) data, obtained in the same CT study. This was thought to offer better understanding of the coronary artery anatomy and regional function [8].

Image fusion of coronary CT angiography with cardiac perfusion MRI facilitated comprehensive assessment of functionally relevant coronary artery disease by the exact allocation of culprit coronary arterial stenosis to the corresponding myocardial perfusion defects, at a low radiation dose [9].

Another exciting area of research is the targeted transendocardial therapeutic delivery of stem cells. This is guided by MRI/x-ray image fusion. 3D visualization of the infarct and other cardiac surfaces is essential for this task. Registering and overlaying MRI roadmaps onto live x-ray fluoroscopy, to guide targeted transendocardial delivery, has been performed on a swine model of myocardial infarction without complications [10].

• Image fusion in cardiac electrophysiology

Another important application of image fusion in cardiology is to assist electrophysiological interventions. Integration of electroanatomic mapping with 3D MR/CT images to guide atrial fibrillation makes the procedure faster, with reduced complications and reduces the use of fluoroscopy. It has been shown that mapping of body surface ECG mapping, in which a multielectrode vest is used to collect body surface potentials, can be fused with images from CT to compute a large array of epicardial unipolar electrocardiograms. Delayed contrast-enhanced MRI can be used to assess focal myocardial fibrosis, which is the substrate involved in most cardiac arrhythmias. This can also be combined with body surface ECG mapping resulting in a patient-specific multimodal 3D model to assist in the diagnosis, prognosis and intracardiac catheter navigation for ablation targeting [11]. Multimodality imaging, combining cardiac MRI/CT with real time x-rays during cardiac resynchronization therapy implantation serves as an invaluable adjunct to physicians to accurately deliver pacemaker leads into optimal position to deliver cardiac resynchronization therapy [12].

• Image fusion in congenital cardiac catheterization

The use of image fusion in congenital heart disease catheterization is relatively recent. Fluoroscopy is the main imaging modality used to guide these procedures, despite its limitations of poor soft tissue, lack of 3D data and above all use of ionizing radiation.

CT/MR image overlay

Overlaying 3D MRI or CT data in real-time on fluoroscopic images adds 3D perspective to the high spatial and temporal resolution obtained. In 2010, angiographic cardiac tomography was used during 41 cardiac catheterization procedures in patients who had a mean age of 5.1 years, and good quality images were obtained in 71% of cases [13]. 3D rotational angiography, multiplanar reformatting and 3D reconstruction was used in 87 patients and was found to be of diagnostic quality or better in 97.4% of studies and provided additional clinically relevant data, when compared with fixed plane angiography [14]. In 2011, x-ray magnetic image fusion (XMRF) with cardiac MRI was performed on 23 patients with congenital heart disease during catheterization to demonstrate the feasibility of reducing radiation exposure and improving catheterization outcome [15]. This was performed with the ingenious use of a parameter file generated during a blank spin (spin without radiation) of the A plane of the fluoroscopic suite. This file contained the bed and gantry co-ordinates and the dimensions of a theoretical x-ray space. This was used to perform 3D–3D registration using an automatic linear transformation algorithm to translate the MRI co-ordinate system into x-ray space and to scale the MRI dataset. Finally, 2D–3D registration was performed with live fluoroscopic 2D image into the 3D dataset. The registration was then fine-tuned manually by using translation along the three spatial co-ordinates of the internal marker of choice as seen on both the MRI and x-ray images. Two orthogonal maximal field-of-view fluoroscopy images, one in AP projection (0°) and with the AP camera in the lateral projection (90°), were used for performing the registration. The images were then displayed during the catheterization procedure. The use of internal markers for registration with minimal additional radiation and without the need of contrast, with clinically acceptable degree of accuracy, using commercially available software, was another innovation. This was found to be useful in road mapping in 74% of cases and camera angle selection in 48% of cases. A variety of internal markers, such as bones, airways, artefact, calcified conduit and heart and blood vessel borders, were used as substrate for registration, and this could be performed in a few minutes with minimal additional radiation. Registration error in patients was low with a median of 2.15 mm (interquartile range 1.65–2.56 mm). Although rigid-body registration algorithms were used and no motion compensation algorithms were incorporated, periodic motion such as cardiac and respiratory motion did not affect the error tolerance for most interventions, but it was noted that nonperiodic errors, especially those caused by distortion of the cardiac structures due to stiff guidewires and catheters were a significant limiting factor during certain procedures.

Biplane XMRF registration was made possible with further development of the prototype, and the need for C-arm dry spin was obviated. 3D/3D registration could be performed with both contour and surface carving were supported, allowing clear presentation of complex 3D structures [16]. This was performed in 20 patients, using surface models generated by threshold-based segmentation from high resolution magnetic resonance angiogram of structures of interest. A high level of accuracy with registration errors of 2.2 ± 1.1 mm (AP) and 1.36 ± 0.7 mm (lateral) were achieved and it was found that solid rendering of surface models, in combination with carving techniques, were most useful for visualization.

XMRF was tested with 44 diagnostic group and weight-matched congenital cardiac patients undergoing diagnostic cardiac catheterization and was found to significantly reduce the amount of radiation and contrast used. The procedural time was shorter, but anesthetic time tended to be longer, partly due to the time needed for performing the MRI study itself [17]. Nevertheless, a few patients already had prior MRI studies for clinical reasons, irrespective of the need for cardiac catheterization.

There are, however, no reported studies to determine if XMRF confers similar benefits to congenital cardiac interventional procedures. This is partly due to the difficulty in ensuring homogeneity between the study groups, as a number of aspects of interventional procedures tend to be operator-, team- and patient-specific.

Transesophageal echocardiography overlay

Co-registering both transesophageal echocardiographic with the fluoroscopic image space and placing them in the same co-ordinate system, with localization and tracking of the transesophageal echocardiography probe, helps to obtain real-time 2D and 3D echocardiographic pictures, which can be fused with fluoroscopic pictures [18]. Markers can be placed, in real-time, on points of interest on the echocardiographic images and displayed on the fluoroscopy images. More recently, real time 3D images can be merged and displayed with the fluoroscopy images. This can help to facilitate complex interventional procedures and reduce the radiation dose and shorten the procedure time.

Medical image fusion: technical aspects

Medical image fusion covers a broad number of hot topic areas, including image processing, computer vision, pattern recognition, machine learning and artificial intelligence. Image fusion applies several techniques of image registration. This is a very active field in medical image computing. Development of automatic registration techniques for accurate spatial alignment of images of the same heart obtained by different imaging modalities at different points of time is a particular focus of research [19]. Interactive, manual image registration is used in many applications and is often used as a reference for validation purposes in more advanced algorithms. Interobserver and intraobserver variations have to be addressed in manual registration methods.

Registration involves finding the optimal transformation that optimizes the ‘fit’ across two images, the first being the ‘reference’ image and the ‘floating’ image. A variety of techniques dependent on the imaging modalities used have been developed, each with particular advantages and disadvantages.

Generally, a coarse-to-fine approach is used, where in the initial stages an under sampled image of the original is used and the resolution is progressively refined to achieve the best possible fitting.

Limitations of image fusion

Despite extensive research and proven bench studies, image-fusion techniques have not been completely translated into clinical routine. The main reason is that important parts of the registration techniques still require intensive manual interaction. Anatomical information extraction requires expert knowledge of the differing imaging modalities and their interaction with different aspects of anatomical structures. Techniques for image quantification and automatic segmentation of the myocardium and blood pool are far from established, particularly for complex, variable anatomies seen in congenital cardiology. Multimodality image fusion systems lack the strength of quantitative evaluations necessary in borderline conditions. In addition, respiratory and cardiac motion artefacts need to be identified and minimized. Finally, the distortion of cardiac structures due to stiff guidewires and catheters tend to be non-systematic and difficult to predict accurately and may be a significant limiting factor in certain situations.

Conclusion

Image fusion may be a very useful tool for accurate guidance during congenital cardiac interventions, especially for defects with complex anatomy, and changes caused by previous multiple procedures. They may improve outcomes and reduce the dose of radiation for the procedure. Further technical developments with automatic image registration, segmentation, motion compensation, device and catheter tracking, prediction of cardiac distortion and dynamic collimation of x-ray during the procedure will allow the procedure to be better automated and become routine standard of care.

Disclaimer

The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

Acknowledgements

The authors thank T Krasemann, R Razavi and K Rhode for reviewing this commentary.

Financial & competing interests disclosure

The research was funded by the NIHR Invention for Innovation programme (i4i). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.