Communication of medical images to diverse audiences using multimodal imaging

A study has been completed examining design issues concerning the interpretation of and dissemination of multimodal medical imaging data sets to diverse audiences. To create a model data set mouse fibrosarcoma tissue was visualised via magnetic resonance imaging (MRI), Matrix-Assisted Laser Desorption/Ionisation-Mass Spectrometry (MALDI-MSI) and histology. MRI images were acquired using the 0.25T Esaote GScan; MALDI images were acquired using a Q-Star Pulsar I mass spectrometer. Histological staining of the same tissue sections used for MALDI-MSI was then carried out. Areas assigned to hemosiderin deposits due to haemorrhaging could be visualised via MRI. In the MALDI-MSI data obtained the distribution sphingomyelin species could be used to identify regions of viable tumour. Mathematical ‘up sampling’ using hierarchical clustering-based segmentation provided a sophisticated image enhancement tool for both MRI and MALDI-MS and assisted in the correlation of images.

Images of the human body (or its parts) in vivo are now routinely produced to reveal, diagnose, or examine disease. Ex vivo imaging of excised organs and tissues is also routinely performed for medical purposes, as a part of pathological screening. A large number of sophisticated in vivo imaging modalities exist including radiography, MRI and ultrasound, and for removed tissues the standard procedure employed is histological staining, increasingly supported by mass spectrometry [1–6].

Matrix-Assisted Laser Desorption/Ionisation-Mass Spectrometry (MALDI-MSI) is a powerful technique allowing visualisation of the spatial distribution of a particular species, within a biological tissue sample. Multiple single mass spectra can be combined together to generate molecular maps of an ion of interest. MALDI-MSI has been frequently utilised for the direct protein profiling of tumour tissue samples, including tumour margin analysis [7–10].

To study the issues arising when multimodal imaging data sets are combined a model data set concerned with induced tumour haemorrhaging was studied. In vivo MRI studies have shown that hemosiderin deposits can be visualised via T2-weighted imaging in patients with un-ruptured aneurysms [11]. Considering this, the model system chosen for study was the pharmacodynamic action of the vascular disrupting agent combretastatin A-4 phosphate (CA4P) on a mouse fibrosarcoma model. We have previously reported observation of the tumour haemorrhaging induced by this compound by matrix-assisted laser desorption ionisation-mass spectrometry imaging (MALDI-MSI) and conventional histology [12]. It was hoped that in all three imaging modalities a haemorrhagic response would be detectable, along with evidence of the presence of necrotic and viable tissue regions within the CA4P treated fibrosarcoma.

Following an initial MRI scan of the tumour sample, MALDI-MS images of lipid/metabolite/peptide distributions in the tumours were obtained. Histological staining (haematoxylin and eosin) was then performed on the section post-MALDI-MSI.

In the interpretation of medical images, from any imaging modality, segmentation is a difficult task because of issues such as spatial resolution, poor contrast, ill-defined boundaries, noise, or acquisition artefacts [12, 13]. The images generated require substantial experience to detect and categorise lesions. Tissue abnormality is usually related to a dissimilar part of an otherwise homogeneous image. The goal of post-processing is to separate the image into regions that are meaningful for a specific task. Choosing the optimal post-processing threshold value can be difficult because the image composition may vary depending on the acquisition parameters and the type of tissue. The aim of the hierarchical clustering-based segmentation (HCS) process is to identify the more appropriate edges of a lesion and the heterogeneous regions within, using the novel procedure border pixel reclassification [14, 15] and it was felt here that segmentation might aid correlation of features between the modalities.

Correlation is not the only issue facing the amalgamation of combined data sets; other issues include the dissemination and communication of the wealth of information contained in such data sets to not only medical professionals but also to patients and a lay audience. Here this was addressed by examining improvements to medical imaging software and design issues concerning the communication of medical images to different audiences. A defined target set for this part of the study was to ascertain whether or not the characteristic heterogeneity of tumour tissue could be correlated between the imaging modalities employed and whether these data could be presented in a combined and easily understandable format.

Chemicals and materials

a-Cyano-4-hydroxycinnamic acid (CHCA), aniline (ANI), ethanol (EtOH), chloroform (CHCl3), acetonitrile (ACN), octyl-a/b-glucoside (OcGlc), tri-fluoroacetic acid (TFA), ammonium bicarbonate, haematoxylin, eosin, xylene and DPX mountant were from Sigma-Aldrich (Dorset, UK). Modified sequence grade trypsin (20 µg lyophilised) was obtained from Promega (Southampton, UK).

Tissue samples

All animal procedures were carried out in accordance with the United Kingdom Animals (Scientific Procedure) Act 1986, with local ethics committee approval and following published guidelines for the use of animals in cancer research [16]. Mice were injected sub-cutaneously in the rear dorsum with a 50 µl tumour cell suspension containing 1 × 106 fs188 tumour cells in serum-free medium. These fibrosarcoma (fs) cells are engineered to express only the VEGF188 isoform of vascular endothelial growth factor A (VEGF) [17]. Tumours were allowed to grow to approximately 500 mm³, before CA4P treatment (a 50 µl single dose of 100 mg/kg i.p in saline). Mice were killed and tumours excised at various times after treatment before being snap frozen in liquid nitrogen-cooled isopentane and stored at −80 °C for later processing. The tumour used for the experimental work described here was excised 24 h after treatment with CA4P.

Tissue preparation

Tumours were embedded in gelatin blocks for MRI acquisition and wooden rods were placed as fiducial markers in all four corners of the gelatine blocks to aid spatial recognition and co-registration between the following imaging modalities. The samples were transported in dry ice following MRI acquisition. Samples were then re-frozen and cryosectioned to a thickness of 10 µm prior to MALDI-MSI data capture. 10 µm-thick frozen tissue sections were cut, using a Leica CM3050 cryostat (Leica Microsystems, Milton Keynes, UK). The sections were then freeze–thaw mounted on poly-l-lysine glass slides. Mounted slides were either used immediately or stored in an airtight tube at −80 °C for subsequent use.

In situ tissue digestion and trypsin deposition

The tissue samples were washed initially with 70 % (v/v) and then 90 % (v/v) ethanol for 1 min then left to dry. Subsequently, slides were immersed in chloroform for 10 s. Prior to matrix application, in situ tissue digestion was performed with trypsin solution prepared (from lyophilised trypsin), at 20 µg/ml, by addition of 50 mM ammonium bicarbonate (NH4HCO3) pH 8, containing 0.5 % octyl-a/b-glucoside (OcGlc, w/v).

The ‘‘Suncollect’’ (SunChrom, Friedrichsdorf, Germany) automatic pneumatic sprayer was used to spray trypsin in a series of layers. The sections for MALDI-MS and MALDI-MSI were incubated overnight in a humidity chamber containing H₂O 50 %: methanol 50 % (v/v) overnight at 37 °C and 5 % CO₂.

Matrix deposition

The matrix, α-cyano-4-hydroxycinnamic acid (CHCA) and aniline in acetonitrile:water:TFA (1:1:0.1 by volume), was applied using the Suncollect (at 5 mg/ml) (SunChrom, Friedrichsdorf, Germany) automatic pneumatic sprayer was used to spray trypsin in a series of layers. Identical coordinate settings to those used for trypsin deposition were employed, to ensure sample uniformity. Equimolar amounts of aniline were added to the CHCA solution, i.e. 1 ml of 5 mg/ml CHCA solution contained 2.4 µl of aniline.

Instrumentation

MRI images were acquired using the 0.25T Esaote GScan. The sample was centrally placed with a dedicated wrist coil and a range of sequences performed with 160 × 160 mm² field of view (FOV). Optimal results were achieved from the T2 weighted Gradient Echo (3NEX) and XBone (4NEX) sequences employing 2 mm slices.

MALDI-IMS/MSI were performed using a Q-Star Pulsar i hybrid quadrupole time-of-flight mass spectrometer (Applied Biosystems/MDS Sciex, Concorde, Ontario, Canada) fitted with a variable repetition rate Nd:YV04 laser set at 5 kHz. Image acquisition was performed using raster/spot to spot imaging mode at 100 µm spatial resolution. Biomap 3.7.5.5 software (http://www.maldi-msi.org/) was used for image generation.

Image co-registration

Images were imported, visualised and co-registered using BioMap B3.7.5.5 software (http://www.maldi-msi.org/). The embedded wooden fiducial markers were used for alignment and augmentation of image co-registration.

Matrix removal and haematoxylin and eosin staining

Sections were washed at a 45° angle using 5 ml of 100 % EtOH (v/v). To rehydrate the sections the procedure was as follows: 100 % EtOH for 3 min, 95 % EtOH for 3 min, 70 % EtOH (v/v) for 3 min, de-ionised H₂O for 2 min. The slides were then immersed in haematoxylin for 45 s, tap water for 3–5 min, 70 % EtOH for 3 min, 95 % EtOH for 3 min (v/v), 20 s 5 % Eosin (w/v). To dehydrate for the purpose of mounting: 95 % EtOH for 3 min ×2, 100 % EtOH (v/v) for 3 min ×2, Xylene 5 min ×2. Finally, DPX mountant was applied to the tissue for analysis and subsequently left to dry in the fume hood overnight.

Multimodality hierarchical clustering-based segmentation (HCS)

HCS was firstly applied to MRI data to improve resolution and visual interpretation of regions within the tumour tissue scans. The following is a high-level description of the HCS process [14] (Fig. 1):

Flowchart to illustrate the overall operation of the HCS process

Full size image

1. 1.

   Give each pixel in the image a region label as follows:

   If an initial segmentation of the image is available, label each pixel according to this pre-segmentation.

   If no initial segmentation is available, label each pixel as a separate region.

   Set current dissimilarity allowed between regions, dissimilarity_allowed, equal to zero.

1. 2.

   Calculate the dissimilarity value, (dissimilarity_value), between all pairs of regions in the image.

   Set threshold_value equal to the smallest dissimilarity_value.

1. 3.

   If the threshold_value found, in step 2, is less than or equal to the current dissimilarity_allowed value, then merge all those regions having dissimilarity_value, between them, less than or equal to the threshold_value.

   Otherwise go to step 6.

1. 4.

   If the number of regions merged in step 3 is greater than 0, then reclassify the pixels on the border of the merged regions with the rest of the regions until no more reclassification is possible. After all the possible border pixels are reclassified, among the merged regions, store the region information for this iteration as an intermediate segmentation and go to step 2.

   Otherwise, if the number of regions merged in step 3 is equal to 0 then, go to step 5.

1. 5.

   If the current number of regions in the image is less than the pre-set value, check_no_regions, go to step 7.

   Otherwise, go to step 6.

1. 6.

   If the current value of dissimilarity_allowed is less than the maximum possible value then increase the dissimilarity_allowed value by an incremental value, and go to step 2.

   Otherwise go to step 7.

1. 7.

   Save the region information from the current iteration as the coarsest instance.

The above steps ensure that the segmentation does not depend on the order in which the image regions are processed.

Sacrificial design concepts of interactive multimodal imaging software

The proposed interactive multimodal software model integrated Histology, MRI and MSI techniques into one interface. Different levels were considered of how each mode of information would be represented, controlled, interpreted and combined. The challenge of how to address analogue, digital, scale, colour, resolution and tangibility issues and the presentation of concepts that showed promise in resolving these issues for both professional lay and public lay readers and operators were considered.

The workflow here using MRI through to the MALDI image acquisitions is shown in Fig. 2a–c. As mentioned previously, studies using MRI have shown that hemosiderin deposits due to haemorrhaging can be visualised via this imaging modality [11].

Acquisition workflow beginning with MRI (a), then tissue cryosectioning (b) to enable generation of MALDI-MSI (c) prior to histological staining. The pigmented area of the tumour tissue indicated by the arrows can be clearly identified at each stage of this workflow. The MALDI-MSI displays the spatial distributing of an ion relating to a haemoglobin peptide at m/z 1529

Full size image

Figure 2c shows a MALDI-MS image of the distribution of a haemoglobin ion (m/z 1529) in a 12 µm section of the fibrosarcoma tissue obtained in this manner. The distribution of haemoglobin in the sections is also diagnostic of haemorrhaging and is a signal with the potential to be correlated with the MRI data. The area of haemorrhaging can be seen in the top left hand corner of the sample in the MRI slice data (Fig. 2a), the MALDI-MSI data (Fig. 2c) and is also clearly visible in the dark pigmented area in the optical image of fresh-frozen tumour tissue block, taken while it was being sectioned (Fig. 2b, as indicated by the arrows).

The MALDI-MSI workflow employed is displayed in Fig. 3. Frozen cryosectioned tissue sections are mounted on a microscope slide then sprayed with MALDI matrix for small molecule analysis or prior to this step, enzymatically digested if peptides are the desired species of interest. After MALDI-MSI acquisition, the spatial distribution of numerous ions can be observed which relate to a chosen peak within the mass spectra generated (example of MALDI images shown in Fig. 3c). Tissue abnormality in a medical image is usually related to a dissimilar part of an otherwise homogeneous image. The dissimilarity may be subtle or strong depending on the medical modality and the type of abnormal tissue. Hence, a dissimilarity highlighting process that yields a hierarchy of segmentation output for the user to choose from will be more useful.

The MALDI-MSI workflow employed here for the analysis of excised tissue (a) post-MRI consisted of freeze thaw mounting of cryosectioned tissue (b), on tissue digestion for peptide analysis (omitted for lipids) prior to matrix application, and finally generation of MALDI-MSI (c) to enable the mapping and co-registration of peaks of interest found in the corresponding MALDI spectrum (d)

Full size image

Segmentation can be thought as a process of grouping visual information, where the details are grouped into objects, objects into classes of objects, etc. Thus, starting from the composite segmentation, the perceptual organisation of the image can be represented by a tree of regions, ordered by inclusion. The root of the tree is the entire scene, the leaves are the finest details and each region represents an object at a certain scale of observation [18]. Since the early days of computer vision, the hierarchical structure of visual perception has motivated clustering techniques towards segmentation [19], where connected regions of the image domain are classified according to an inter-region dissimilarity measure.

Hierarchical clustering-based segmentation (HCS) (Fig. 1) implements the traditional bottom-up approach of agglomerative clustering, where the regions of an initial partition are iteratively merged. HCS automatically generates a hierarchy of segmented images. The hierarchy of segmented images is generated by partitioning an image into its constituent regions at hierarchical levels of allowable dissimilarity between its different regions. At any particular level in the hierarchy, the segmentation process will cluster together all the pixels and/or regions which have dissimilarity among them less than or equal to the dissimilarity allowed for that level. Normally in agglomerative clustering methods, the cluster structure depends on the order in which the regions are considered [20]. The brute force approach, followed by the HCS process, where only those regions with the smallest overall dissimilarity are merged in each step, is the only solution to overcome this effect [21].

To identify possible parts of the MR image slice which might correlate with the MALDI images, HCS (as described in Fig. 1) was applied to the MRI scan data shown in (Fig. 4a). Figure 4b–e shows the HCS highlighted major regions which are not obvious in the original MRI. More regions appear within the images as the cluster size increases post-HCS processing, subsequently revealing areas not observable from the MRI scan alone.

Using the MRI scanned tissue (a), the HCS process outputs (b–e) are displayed with indication of the number of regions detailed. The regions are assigned a grey colour based on the average pixel value of the pixels contained within the regions. The more regions that are ‘up-sampled’, the more details can be observed

Full size image

The aim of the analysis in Fig. 5 was to establish possible further correlations between the MRI slices, MALDI-MSI and the haematoxylin and eosin performed using the embedded excised tumour tissue. Signals from the MALDI-MSI data representative of lipid peaks Fig. 5b, e at m/z 725 show strong correlation with the MRI data here in Fig. 5a, d, respectively. The ion at m/z 725 can be assigned to a sphingolipid species. Altered levels of sphingolipid species have been found between tumour and normal tissues and the lipid species (m/z 725) shown here has been known to be increased in viable tumour tissue [22]. The haematoxylin and eosin staining shown here in Fig. 5c, f shows agreement with the spatial distribution of the potential sphingolipid at m/z 725 (Fig. 5b, e) in the MALDI-MSI data. The dark blue regions in this image indicate nucleated cells and hence viable tumour tissue regions and show good agreement with the signal at m/z 725 which correspondingly is very weak or not visible in the tissue regions deemed necrotic (showing as white in the MRI images, Fig. 5a, d) according to histological staining.

Correlations between MRI (a, d), MALDI-MSI (b, e), and haematoxylin and eosin staining (c, f). Examples of the spatial distribution of an ion at m/z 725 (tentatively identified as the lipid species, sodiated sphingomyelin) are shown in the MALDI-MSI (b, e). This species appears to be localised within the viable tumour tissue regions evident in the haematoxylin and eosin staining post-MALDI image acquisition. Correlations are observable, indicated by the brackets which show a high intensity of m/z 725 (e) along with dark blue haematoxylin staining of the nucleated cells within this viable tumour tissue area

Full size image

One limitation of the current study was that MRI images were acquired at a spatial resolution of 500 micron, dictated by the magnetic field strength. Future studies will aim to utilise higher field strength devices capable of delivering 100 μm which will improve post-processing capability due to the reduced pixel size. However, even so, necrotic and viable tissue regions could be seen in the correlations between the MRI, MALDI-MSI and haematoxylin and eosin staining. These findings hold great promise to potentially assess efficiency of treatment in tumours.

A current challenge faced at all stages of the acquisition and processing of multimodal imaging data is how to present it in a manner that simply and clearly conveys the information contained within it. A ‘mock up’ of proposed multimodal imaging software was developed in this project and is presented in Fig. 6. This interactive tool would comprise a scrolling bar to view image planes within a 3D image environment and tissue overlay function. The software should also allow the user to seamlessly switch between imaging modes to perform data correlations between MALDI, MRI and histology data.

A ‘mock up’ of the proposed design concept for multimodal imaging software, this interactive programme would allow the user to switch between imaging modalities to serve as an informative diagnostic tool

Full size image

Considering the three imaging modes to be integrated (MRI, MSI and histology) individually, current imaging provision tends to rely on highly specific expertise both in terms of sample preparation and analysis. Early concept designs consider the information to be conveyed in a number of ways and at a number of levels. This is both understandable and acceptable given the highly specific nature of each technique. However, with regard to viewing and communicating each mode to lay persons, such technical information would require various barriers or ‘levels’ to ease accessibility and understanding. Pertaining to the parameters of each mode, and secondly, in communicating the outcomes.

The first pertains to the viewing modality and its graphical user interface, where and how visual control elements are placed, manipulated, and how the aspect of an image mode are ‘switched’ from one data view to another. These barriers can be categorised as graphical user interface barriers. The second set of barriers to efficient use and understanding relate to what the image set is communicating, in any particular imaging mode. These can relate to issues of scale, what the colour mapping presents or tangibility issues associated with sample position or shape.

Tangibility or having some form of spatial reference to sample data was felt to be important for all imaging modes because of an apparent disconnect between what was seen on screen and what that information meant to a viewer, especially a lay viewer. This was true of both the actual sample data and the means by which the image is manipulated.

The graphic communication mode of the aforementioned data sets is seen as crucial to bridging gaps in understanding for both lay and professional readers. In the proposed model, we have assigned a ‘plane’ to each imaging mode (MRI, MSI and Histology). The planes are intended to ground the three-dimensional nature of the sample within a tangible environment and provide an interface by which aspects of each mode (say, for the MSI mode, peptide intensity) can be selected for a given slice through the sample.

Currently there are no multimodal medical imaging software packages commercially available which have multifunctional co-registration capabilities. Software that could provide an ‘up-sampling’ feature for each imaging mode using HCS along with co-registration functionality would provide an exciting educational and diagnostic tool. Over 15 years ago, Calamai et al. [23] reported on the requirement of artificial vision systems in biomedicine and the effects of a computer segmentation approached on breast angiograms. There has since been some progression towards these methods of image manipulation within research groups that have sought to undertake such computational approaches, including, a recent research output by a group using medical imaging data and computer-graphics from MRI and computerised tomography (CT) scans to estimate patient specific lumbar spine muscle forces [24].

The ability to visualise viable tissue from an excised treated tumour could provide medical professionals and patients with information relating to the success or failure of an anti-cancer treatment (a complimentary test for tumour boundary analysis). Increasing the amount of information available to patient if requested would indeed enhance patient after care services.

HCS processing confirms the heterogeneous nature of seemingly homogeneous tissue by processing the images performed using MRI. This facilitates the accurate mapping of the MR images with the corresponding MALDI and HDI image data. Utilisation of this niche for a software interface to merge imaging modalities could provide initial steps towards the implementation of MALDI-MSI into the clinical pathological workflow, bringing a new technique to enhance the ‘bench to bedside’ concept.

LMC carried out the MALDI-MSI and the histology and drafted the manuscript, ANS wrote the hierarchical clustering software and carried out data analysis, RP and HR contributed to the study design and the design of the final user interface, CW contributed to the study design and the MRI work, MRC contributed to the study design, the interpretation of the MALDI-MSI and MRI data and the final manuscript draft. All authors read and approved the final manuscript.

Acknowledgements

We thank Imagine Connectivity for funding this pilot study, Manchester Metropolitan University for use of their MRI facility, Prof GM Tozer, University of Sheffield for providing the samples for this project.

Compliance with ethical guidelines

Competing interests The authors have no competing interests.

Affiliations

1. Biomedical Research Centre, Sheffield Hallam University, Sheffield, UK

   • Laura M. Cole
   •  & Malcolm R. Clench

2. Materials and Engineering Research Institute, Sheffield Hallam University, Sheffield, UK

   • Arul N. Selvan

3. Art and Design Research Centre, Sheffield Hallam University, Sheffield, UK

   • Rebecca Partridge
   •  & Heath Reed

4. Health and Wellbeing, Sheffield Hallam University, Sheffield, UK

   • Chris Wright

Corresponding author

Correspondence to Laura M. Cole.

Reprints and Permissions