Communication of medical images to diverse audiences using multimodal imaging

Images of the human body (or its parts) in vivo are now routinely produced to reveal, diagnose, or examine disease. Ex vivo imaging of excised organs and tissues is also routinely performed for medical purposes, as a part of pathological screening. A large number of sophisticated in vivo imaging modalities exist including radiography, MRI and ultrasound, and for removed tissues the standard procedure employed is histological staining, increasingly supported by mass spectrometry [1–6].

Matrix-Assisted Laser Desorption/Ionisation-Mass Spectrometry (MALDI-MSI) is a powerful technique allowing visualisation of the spatial distribution of a particular species, within a biological tissue sample. Multiple single mass spectra can be combined together to generate molecular maps of an ion of interest. MALDI-MSI has been frequently utilised for the direct protein profiling of tumour tissue samples, including tumour margin analysis [7–10].

To study the issues arising when multimodal imaging data sets are combined a model data set concerned with induced tumour haemorrhaging was studied. In vivo MRI studies have shown that hemosiderin deposits can be visualised via T2-weighted imaging in patients with un-ruptured aneurysms [11]. Considering this, the model system chosen for study was the pharmacodynamic action of the vascular disrupting agent combretastatin A-4 phosphate (CA4P) on a mouse fibrosarcoma model. We have previously reported observation of the tumour haemorrhaging induced by this compound by matrix-assisted laser desorption ionisation-mass spectrometry imaging (MALDI-MSI) and conventional histology [12]. It was hoped that in all three imaging modalities a haemorrhagic response would be detectable, along with evidence of the presence of necrotic and viable tissue regions within the CA4P treated fibrosarcoma.

Following an initial MRI scan of the tumour sample, MALDI-MS images of lipid/metabolite/peptide distributions in the tumours were obtained. Histological staining (haematoxylin and eosin) was then performed on the section post-MALDI-MSI.

In the interpretation of medical images, from any imaging modality, segmentation is a difficult task because of issues such as spatial resolution, poor contrast, ill-defined boundaries, noise, or acquisition artefacts [12, 13]. The images generated require substantial experience to detect and categorise lesions. Tissue abnormality is usually related to a dissimilar part of an otherwise homogeneous image. The goal of post-processing is to separate the image into regions that are meaningful for a specific task. Choosing the optimal post-processing threshold value can be difficult because the image composition may vary depending on the acquisition parameters and the type of tissue. The aim of the hierarchical clustering-based segmentation (HCS) process is to identify the more appropriate edges of a lesion and the heterogeneous regions within, using the novel procedure border pixel reclassification [14, 15] and it was felt here that segmentation might aid correlation of features between the modalities.

Correlation is not the only issue facing the amalgamation of combined data sets; other issues include the dissemination and communication of the wealth of information contained in such data sets to not only medical professionals but also to patients and a lay audience. Here this was addressed by examining improvements to medical imaging software and design issues concerning the communication of medical images to different audiences. A defined target set for this part of the study was to ascertain whether or not the characteristic heterogeneity of tumour tissue could be correlated between the imaging modalities employed and whether these data could be presented in a combined and easily understandable format.

The workflow here using MRI through to the MALDI image acquisitions is shown in Fig. 2a–c. As mentioned previously, studies using MRI have shown that hemosiderin deposits due to haemorrhaging can be visualised via this imaging modality [11].

Figure 2c shows a MALDI-MS image of the distribution of a haemoglobin ion (m/z 1529) in a 12 µm section of the fibrosarcoma tissue obtained in this manner. The distribution of haemoglobin in the sections is also diagnostic of haemorrhaging and is a signal with the potential to be correlated with the MRI data. The area of haemorrhaging can be seen in the top left hand corner of the sample in the MRI slice data (Fig. 2a), the MALDI-MSI data (Fig. 2c) and is also clearly visible in the dark pigmented area in the optical image of fresh-frozen tumour tissue block, taken while it was being sectioned (Fig. 2b, as indicated by the arrows).

The MALDI-MSI workflow employed is displayed in Fig. 3. Frozen cryosectioned tissue sections are mounted on a microscope slide then sprayed with MALDI matrix for small molecule analysis or prior to this step, enzymatically digested if peptides are the desired species of interest. After MALDI-MSI acquisition, the spatial distribution of numerous ions can be observed which relate to a chosen peak within the mass spectra generated (example of MALDI images shown in Fig. 3c). Tissue abnormality in a medical image is usually related to a dissimilar part of an otherwise homogeneous image. The dissimilarity may be subtle or strong depending on the medical modality and the type of abnormal tissue. Hence, a dissimilarity highlighting process that yields a hierarchy of segmentation output for the user to choose from will be more useful.

Segmentation can be thought as a process of grouping visual information, where the details are grouped into objects, objects into classes of objects, etc. Thus, starting from the composite segmentation, the perceptual organisation of the image can be represented by a tree of regions, ordered by inclusion. The root of the tree is the entire scene, the leaves are the finest details and each region represents an object at a certain scale of observation [18]. Since the early days of computer vision, the hierarchical structure of visual perception has motivated clustering techniques towards segmentation [19], where connected regions of the image domain are classified according to an inter-region dissimilarity measure.

Hierarchical clustering-based segmentation (HCS) (Fig. 1) implements the traditional bottom-up approach of agglomerative clustering, where the regions of an initial partition are iteratively merged. HCS automatically generates a hierarchy of segmented images. The hierarchy of segmented images is generated by partitioning an image into its constituent regions at hierarchical levels of allowable dissimilarity between its different regions. At any particular level in the hierarchy, the segmentation process will cluster together all the pixels and/or regions which have dissimilarity among them less than or equal to the dissimilarity allowed for that level. Normally in agglomerative clustering methods, the cluster structure depends on the order in which the regions are considered [20]. The brute force approach, followed by the HCS process, where only those regions with the smallest overall dissimilarity are merged in each step, is the only solution to overcome this effect [21].

To identify possible parts of the MR image slice which might correlate with the MALDI images, HCS (as described in Fig. 1) was applied to the MRI scan data shown in (Fig. 4a). Figure 4b–e shows the HCS highlighted major regions which are not obvious in the original MRI. More regions appear within the images as the cluster size increases post-HCS processing, subsequently revealing areas not observable from the MRI scan alone.

The aim of the analysis in Fig. 5 was to establish possible further correlations between the MRI slices, MALDI-MSI and the haematoxylin and eosin performed using the embedded excised tumour tissue. Signals from the MALDI-MSI data representative of lipid peaks Fig. 5b, e at m/z 725 show strong correlation with the MRI data here in Fig. 5a, d, respectively. The ion at m/z 725 can be assigned to a sphingolipid species. Altered levels of sphingolipid species have been found between tumour and normal tissues and the lipid species (m/z 725) shown here has been known to be increased in viable tumour tissue [22]. The haematoxylin and eosin staining shown here in Fig. 5c, f shows agreement with the spatial distribution of the potential sphingolipid at m/z 725 (Fig. 5b, e) in the MALDI-MSI data. The dark blue regions in this image indicate nucleated cells and hence viable tumour tissue regions and show good agreement with the signal at m/z 725 which correspondingly is very weak or not visible in the tissue regions deemed necrotic (showing as white in the MRI images, Fig. 5a, d) according to histological staining.

One limitation of the current study was that MRI images were acquired at a spatial resolution of 500 micron, dictated by the magnetic field strength. Future studies will aim to utilise higher field strength devices capable of delivering 100 μm which will improve post-processing capability due to the reduced pixel size. However, even so, necrotic and viable tissue regions could be seen in the correlations between the MRI, MALDI-MSI and haematoxylin and eosin staining. These findings hold great promise to potentially assess efficiency of treatment in tumours.

A current challenge faced at all stages of the acquisition and processing of multimodal imaging data is how to present it in a manner that simply and clearly conveys the information contained within it. A ‘mock up’ of proposed multimodal imaging software was developed in this project and is presented in Fig. 6. This interactive tool would comprise a scrolling bar to view image planes within a 3D image environment and tissue overlay function. The software should also allow the user to seamlessly switch between imaging modes to perform data correlations between MALDI, MRI and histology data.

Considering the three imaging modes to be integrated (MRI, MSI and histology) individually, current imaging provision tends to rely on highly specific expertise both in terms of sample preparation and analysis. Early concept designs consider the information to be conveyed in a number of ways and at a number of levels. This is both understandable and acceptable given the highly specific nature of each technique. However, with regard to viewing and communicating each mode to lay persons, such technical information would require various barriers or ‘levels’ to ease accessibility and understanding. Pertaining to the parameters of each mode, and secondly, in communicating the outcomes.

The first pertains to the viewing modality and its graphical user interface, where and how visual control elements are placed, manipulated, and how the aspect of an image mode are ‘switched’ from one data view to another. These barriers can be categorised as graphical user interface barriers. The second set of barriers to efficient use and understanding relate to what the image set is communicating, in any particular imaging mode. These can relate to issues of scale, what the colour mapping presents or tangibility issues associated with sample position or shape.

Tangibility or having some form of spatial reference to sample data was felt to be important for all imaging modes because of an apparent disconnect between what was seen on screen and what that information meant to a viewer, especially a lay viewer. This was true of both the actual sample data and the means by which the image is manipulated.

The graphic communication mode of the aforementioned data sets is seen as crucial to bridging gaps in understanding for both lay and professional readers. In the proposed model, we have assigned a ‘plane’ to each imaging mode (MRI, MSI and Histology). The planes are intended to ground the three-dimensional nature of the sample within a tangible environment and provide an interface by which aspects of each mode (say, for the MSI mode, peptide intensity) can be selected for a given slice through the sample.

Currently there are no multimodal medical imaging software packages commercially available which have multifunctional co-registration capabilities. Software that could provide an ‘up-sampling’ feature for each imaging mode using HCS along with co-registration functionality would provide an exciting educational and diagnostic tool. Over 15 years ago, Calamai et al. [23] reported on the requirement of artificial vision systems in biomedicine and the effects of a computer segmentation approached on breast angiograms. There has since been some progression towards these methods of image manipulation within research groups that have sought to undertake such computational approaches, including, a recent research output by a group using medical imaging data and computer-graphics from MRI and computerised tomography (CT) scans to estimate patient specific lumbar spine muscle forces [24].

The ability to visualise viable tissue from an excised treated tumour could provide medical professionals and patients with information relating to the success or failure of an anti-cancer treatment (a complimentary test for tumour boundary analysis). Increasing the amount of information available to patient if requested would indeed enhance patient after care services.