Reduced-dose and high-speed acquisition strategies for multi-dimensional electron microscopy

Abstract

Multi-dimensional electron microscopy has recently gained considerable interest thanks to the advent of microscopes with unprecedented analytical and in situ capabilities. These information-rich imaging modes, though, are often subject to long acquisition times and large data generation. In this paper, we explore novel acquisition strategies and reconstruction algorithms to retrieve reliable reconstructions from datasets that are limited in terms of both per image and tilt series angular sampling. We show that inpainting techniques are capable of restoring scanning transmission electron microscopy images in which a very restricted number of pixels are scanned, while compressed sensing tomographic reconstruction is capable of minimising artefacts due to angular subsampling. An example of robust reconstruction from data constituting a dose reduction of 10× is presented, using an organic/inorganic core-shell nanowire as a test sample. The combination of these novel acquisition schemes and image recovery strategies provides new avenues to reduced-dose and high-speed imaging.

Background

Electron tomography (ET) is an invaluable technique for the three-dimensional (3D) characterisation of nanomaterials, frequently playing a unique role in elucidating nanostructure-property relationships [1]. ET entails acquiring a set of transmission electron microscopy (TEM) images at different tilt angles and subsequently employing a reconstruction algorithm to retrieve the 3D object [2]. Typically, bright field TEM is used in the life sciences, while high-angle annular dark-field scanning TEM (HAADF-STEM) is preferred in materials science because of its relative insensitivity to diffraction effects and atomic-number (Z) contrast [3]. ET based on these imaging modes is nowadays often performed in an automated or semi-automated manner and is routinely employed for the study of morphology and distribution of features in 3D (e.g. [4-7]).

Recently, ‘multi-dimensional electron microscopy’ (often ET plus a further dimension) has gained significant interest, with the desire to follow structural changes in space and time and to generate chemically sensitive volume reconstructions that can be analysed quantitatively [8]. More specifically, with the availability of high-brightness electron sources, monochromators, X-ray detectors with improved efficiency and energy loss spectrometers with high speed and dynamic range, ground-breaking new insights are within reach.

Nonetheless, 3D imaging coupled with dynamic and/or spectroscopic dimensions faces challenges with regard to achieving sufficient signal-to-noise ratio (SNR) at an acceptable acquisition time and electron dose. To date, the reconstruction fidelity has generally been optimised by acquiring many projections over as wide a tilt range as possible. Conventional reconstruction algorithms, such as weighted backprojection (WBP) or the simultaneous iterative reconstruction technique (SIRT), yield reasonable tomograms when the dataset is acquired with a high SNR and a fine angular sampling increment [1,2 and references therein]. Unfortunately, such standard acquisition parameters cannot easily be extended to multi-dimensional experiments. STEM-based spectroscopies such as energy dispersive X-ray spectroscopy (EDX) or electron energy loss spectroscopy (EELS) generally require long per-pixel acquisition times and, due to the additional spectral dimension, generate large amounts of data. Extension of these methods to 3D spectroscopic imaging using TEM or STEM requires the sample to be beam-resistant (e.g. [9]) and/or a sacrifice of the image resolution by reducing the size of the spectrum-images. Using state-of-the art TEMs, voxel spectroscopy by simultaneous EELS and EDX tomography was demonstrated by Haberfehlner et al. in [10], after generating 2,920 spectral volumes individually. Likewise, in situ dynamic tomography requires high-speed tilt series acquisition, ideally of multiple tilt series, in order to capture 3D snapshots. In Kwon and Zewail’s work, time resolved 4D ET was performed on a multi-walled carbon nanotube, with the acquisition of 4,000 spatiotemporal projections, followed by alignment, reconstruction and rendering of each 3D temporal volume separately [11].

There is a strong need to develop both acquisition and processing tools that are specifically adapted to low-dose and high-speed ET and for STEM-based techniques in particular. Different approaches can be explored for this purpose, with various consequences on the quality of the signal and hence the reliability of the reconstruction. These approaches can be classified as tilt-downsampling (T-DS) and image-downsampling (I-DS) techniques.

T-DS techniques

T-DS techniques rely on either reducing the tilt range or increasing the tilt increment. In general, datasets with limited tilt range tend to result in reconstructions with artefacts that are harder to correct than those related to large tilt increments. Indeed, a limited tilt range results in a large ‘missing wedge’ of information in the Fourier space of the object. Consequent artefacts, such as blurring and elongation of features in the missing wedge direction, deteriorate the quality of the reconstruction and make quantification difficult (e.g. [12]). So far, therefore, tilt series acquisition times have generally been reduced by acquiring fewer images over the widest tilt range available and employing advanced reconstruction algorithms such as discrete algebraic reconstruction technique (DART) or compressed sensing ET (CS-ET). The common feature of these techniques is the incorporation of prior knowledge in the reconstruction process. DART has been introduced to ET by Batenburg et al. [8] for the reconstruction of objects with a known number of grey levels [13]. Recently, a variant of DART was employed for the 3D atomic reconstruction of a crystalline nanoparticle, using only two HAADF-STEM projections and strong assumptions about its crystallographic structure [14]. CS-ET is a more general technique that takes advantage of any prior knowledge about the sparsity of the object in a chosen transform domain and seeks a sparse solution that is consistent with the acquired data [15]. It has been successfully applied to heavily undersampled datasets, using different sparsifying domains depending on the nature of the signal [16-18].

I-DS techniques

I-DS techniques include (1) reduction of the frame size, (2) reduction of the per-pixel dwell time, (3) reduction of the beam current density and (4) recording only a subset of pixels of the full frame (e.g. a random selection of pixels). Reducing the frame size (option 1), by increasing the pixel size to reduce the total number of pixels per frame, is the most straightforward approach but leads to an irretrievable loss of resolution. Options (2) and (3), recently explored in [19] for imaging beam-sensitive materials, are becoming increasingly popular in the area of high resolution STEM (HR-STEM) in which the acquisition of a series of low-dose images is performed, followed by registration and averaging [20,21].

Option (4) is commonly explored for example in magnetic resonance imaging (MRI), where various patterns are tested and implemented, with the aim to accelerate dynamic examinations [22]. Similar manipulation of scanning patterns has attracted the interest of the microscopy community, with the difference that MRI (sub-)sampling is in the Fourier domain, while the microscopy counterpart takes place in the image domain. Notably, Lissajous scan patterns were recently applied to atomic force microscopy for time-lapsed imaging and were shown to yield reduced distortions compared to fast raster-based scanning [23]. Random subsampling has also been explored for high-speed electron microscopy data collection, mainly for dose rate considerations. Anderson et al. simulated such an experiment in the scanning electron microscope (SEM) by selecting a random subset of pixel locations and recovering the full frame image by interpolation, using image smoothness as a prior knowledge [24]. This approach was also simulated on HR-STEM images of grain boundaries and lower magnification STEM images of extremely beam-sensitive materials, with the scanning of only 5% of the total number of pixels in the original image and recovery achieved by Bayesian dictionary learning technique [25].

Although automated low-dose ET acquisition packages have been developed and algorithms that tackle datasets with large tilt increments have been successfully implemented, I-DS techniques based on recording a subset of pixels have not yet, to our knowledge, been tested on ET datasets.

In this paper, we explore a combination of T-DS and I-DS strategies in order to lower the electron dose and acquisition time. I-DS restoration is achieved by total variation (TV) inpainting, while the tomographic reconstructions are performed by CS-ET. The approach is tested on an organic/inorganic core-shell CoPc/ZnO nanowire (NW), where ET is employed to examine the integrity of the core and the porous structure of the shell.

Methods

HAADF-STEM tomography of a CoPc/ZnO NW

CoPc/ZnO NWs are 1D organic/inorganic core-shell materials formed by an inner organic single-crystal NW of cobalt phthalocyanine (CoPc) conformally covered with a zinc oxide (ZnO) layer. The synthesis procedure entails a two-step vacuum methodology. First, single-crystal CoPc NWs are grown by physical vapour deposition, using substrates decorated with metal nanoparticles that act as nucleation centres. Then, a ZnO shell is grown at room temperature using plasma-enhanced chemical vapour deposition in a remote configuration. The properties of CoPc/ZnO NWs depend strongly on the integrity of the organic core after the plasma deposition of the shell [26,27]. This property, along with the detailed morphology of the ZnO shell along the full NW, was investigated here by HAADF-STEM tomography. For this purpose, the sample was dispersed on a commercial TEM grid possessing a 5 nm-thick carbon film on a thicker 30 nm holey carbon film (Pacific Grid Tech, San Francisco, CA, USA). Using a probe-corrected TITAN microscope (FEI, Columbus, OH, USA) operating at 300 kV, a NW oriented parallel to the rotation axis was selected, and a HAADF-STEM tilt series was acquired from −70° to +68° with 2° increment. The frame size was 1024 × 1024, with a pixel size of 1.54 nm and a frame time of 15 s. The tilt series was exported to Inspect3D (FEI, Hillsboro, OR, USA) for alignment by cross-correlation. This full dataset was then used as a reference and subsequently downsampled for I-DS and T-DS simulations. In addition to the CS-ET reconstructions, a reference reconstruction was also generated using the full dataset and a conventional SIRT algorithm, implemented in Inspect3D.

Simulation of I-DS and T-DS set-ups

To simulate randomised subsampling regimes across the full 1024×1024 frame size, patterns of ones and zeros, following Bernoulli distribution, were generated in Matlab (MathWorks, Natick, MA, USA). The patterns were then used as masks over the fully sampled projections. Experimentally, this set-up simulates a reduction in dose (and acquisition time) per image, without altering the SNR of the sampled pixels. Additionally, the angular increment can be increased in order to further reduce the total dose for the tilt series. This effectively results in fewer pixels scanned in the dataset, but each could achieve a higher SNR than in a conventional low-dose tomography experiment where dose fractionation approach is commonly used (i.e. all pixels are sampled and a maximum permissible number of projections are acquired but with a lower dose and SNR per pixel).

Restoration of the I-DS projections

In this work, in-house code written in Matlab is used to perform TV inpainting, following the approach described in [28]. In order to interpolate the missing pixels in the projection images, we compute a solution of the optimisation problem:

$$ \widehat{u}= arg\ mi{n}_{\tilde{u}}TV\left(\tilde{\boldsymbol{u}}\right)\ subject\ to\kern0.5em S\tilde{u} = f $$

(1)

Here, f denotes the sampled pixels of the projection image, S is the subsampling operation that selects only those pixels of a full sized image û that are known and TV(û) is the discrete isotropic TV of û (see [29] for explicit notation). Thus, the optimisation problem (1) is an interpolation method that aims at finding an image û for which the given pixels of the projection image are unaffected, while missing pixels are filled such that the TV of the whole image û is minimal. In order to solve (1) computationally, we reformulate it to:

$$ \left(\widehat{u},\widehat{v}\right) = argmi{n}_{\tilde{u},\tilde{v}}\kern0.5em {\left\Vert \tilde{v}\right\Vert}_{\ell 1}\ subject\ to\kern0.5em \left(\begin{array}{c}\hfill S\hfill \\ {}\hfill \nabla \hfill \end{array}\right)\tilde{u}\ \hbox{--} \left(\begin{array}{c}\hfill 0\hfill \\ {}\hfill I\hfill \end{array}\right)\tilde{v}=f $$

(2)

Note that other regularisation functions (such as the squared two-norm of the gradient of the image, the one-norm of a wavelet transformation of the image or the total generalised variation of the image (cf. [30]) for instance) are possible and yield different interpolation results.

CS-ET reconstruction of the T-DS datasets

The CS-ET algorithm was also implemented in Matlab. For a comprehensive description, the reader is referred to [15]. In summary, the projections are first Fourier transformed to obtain radial samples of the object in the Fourier domain. An initial reconstruction is then obtained from the radial Fourier data using the non-uniform fast Fourier transform (NUFFT) developed by Fessler and Sutton [31]. In conjunction with the NUFFT, the conjugate gradient descent algorithm of Lustig et al. [32] is then used to solve the optimization problem defined by:

$$ {\widehat{x}}_{\lambda }= \arg \kern0.2em mi{n}_{\tilde{x}}\kern1.5em \left\{{\left\Vert \varPhi \tilde{x}-b\right\Vert}_{\ell_2}^2\kern1.75em +\kern1.5em \lambda {\left\Vert \varPsi \tilde{x}\right\Vert}_{\ell_1}\right\} $$

(3)

where $\widehat{x}$ is the reconstruction of the true signal x, Φ is the undersampled Fourier transform operator, b is the Fourier transform of the acquired tilt series and Ψ is the chosen sparsifying transform. Similar to the TV term in Equation 1 and the ℓ ₁-norm term in Equation 2, the ℓ ₁-norm term in Equation 3 promotes sparsity in the chosen transform domain and is defined as the sum of the absolute values. λ is a regularisation parameter that determines the relative importance of sparsity in the reconstruction. Essentially, the algorithm promotes sparsity in the Ψ basis, subject to consistency with the measured data. For the CoPC/ZnO NW, sparsity was promoted in both the image and gradient domains. In this case, Equation 3 can be reformulated as:

$$ {\widehat{x}}_{\lambda_I,\ {\lambda}_{TV}}= \arg \kern0.2em mi{n}_{\tilde{x}}\kern2em \left\{{\left\Vert \varPhi \tilde{x}-b\right\Vert}_{\ell_2}^2\kern0.5em +\kern1.5em {\lambda}_I{\left\Vert \tilde{x}\right\Vert}_{\ell_1}\kern1.25em +\kern1.5em {\lambda}_{TV}TV\left(\tilde{x}\right)\right\} $$

(4)

where similarly λ _I and λ _TV are regularisation parameters that reflect the degree of sparsity imposed in the image and gradient domains, respectively. These λ _I and λ _TV values are chosen by visual assessment of reconstruction quality on selected slices, aiming for the optimal trade-off between minimization of reconstruction artefacts and loss of genuine signal. In the CS-ET implementation used here, TV $\left(\widehat{x}\right)$ is calculated as the isotropic TV, although other variants are applicable.

Results

Reconstruction of the fully sampled dataset

Figure 1a shows the zero tilt HAADF-STEM projection of the NW from the tilt series, where Z-contrast gives insight into the different components present (see the full tilt series in Additional file 1). A voxel projection visualisation of a conventional SIRT reconstruction shows the porous nanocolumnar structure of the ZnO shell (Figure 1b). As can be appreciated in the cross-sectional view through the tomogram in Figure 1c, the columns do not lie perpendicular to the organic core of the NW but at an inclination of ca. 30°. This is due to the relative orientation of the organic NW and the direction of the incoming plasma species during the synthesis process. The organic core was found to have a rectangular cross-section (Figure 1d), with no evident damage throughout the length of the NW (Figure 1c).

Figure 1

HAADF-STEM tomography of a CoPc/ZnO nanowire. (a) HAADF-STEM image of the CoPc/ZnO NW taken at 0° tilt angle. (b) Voxel projection visualisation of the SIRT reconstruction, showing the integrity of the organic core (*) and the nanocolumnar structure of the ZnO shell (**). (c) xy cross-sectional slice, showing a 30° inclination of the branches in the shell. (d) xz cross-sectional slice, showing the rectangular shape of the core and the porous structure of the shell.

The movie in Additional file 2 shows the CS-ET reconstruction of the fully sampled dataset, used as ground truth in the rest of the paper. Note that a better signal-to-background ratio is achieved with CS-ET than with SIRT (Figure 1c,d), although qualitatively both of these reconstructions permit similar levels of structural details to be discerned. As shown in e.g. [15,16], however, CS-ET can substantially outperform SIRT when the tilt increment is large and when seeking to analyse the reconstruction quantitatively.

TV inpainting of I-DS datasets

Figure 2b,e,h shows the 0° tilt projections obtained with only 60%, 30% and 10% of pixels randomly sampled, as compared to the fully sampled image (Figure 2a). Figure 2c,f,i shows the restored images after TV inpainting.

Figure 2

Downsampling by simulated random scanning of the full frame image (a), using a Bernoulli pattern. (b,e,h) show images where only 60%, 30% and 10% of the pixels are sampled from the image in (a), respectively. (c,f,i) show the corresponding TV inpainting results and root mean square errors. Image grey levels have been normalised to the maximum of the fully sampled image in order to ease the comparison. (d,g,j) are line profiles through the yellow line in the inset in (d), comparing the reference, subsampled and TV-inpainted images. Scale bar: 200 nm.

Good recovery is achieved for the 60% and 30% I-DS simulations, with a root mean square error (RMSE) of 1.09% and 2.82%, respectively. The quality of the inpainting can also be appreciated qualitatively in the insets in Figure 2c,f, and in the line profiles in Figure 2d,g, where a good match between the ground-truth image (full black line) and the recovered image (dashed red line) is observed.

Details of the nanoporous structure of the shell are not as well recovered with 10% I-DS (inset in Figure 2i, RMSE = 5.63%). The overall shape of the core-shell NW is, however, still successfully retrieved, as seen in the line profile (Figure 2j).

Combination of I-DS and T-DS

For simulating reduced-dose ET based on both I-DS and T-DS, tilt series are generated by changing the Bernoulli subsampling pattern for each tilt increment in order to reduce coherent artefacts in the reconstruction (see Additional files 3, 4 and 5). I-DS projections are restored individually (see Additional files 6, 7 and 8), and CS-ET with sparsity in the image and TV domains is employed to reconstruct the NW under T-DS settings.

Figure 3 shows a selected xz orthoslice obtained with various degrees of I-DS (rows) and T-DS (columns). The fidelity of the reconstructions is affected differently by the two downsampling schemes. In all T-DS scenarios, CS-ET suppresses well-known ‘streaking’ artefacts that would occur in conventional reconstructions from few projections. For a given I-DS setting, details of the outer branching in the shell are consistently retrieved, while small pores become more difficult to discern as the T-DS increases (see for example the insets in Figure 3a,d and the corresponding line profiles in Figure 4). This is consistent, for example, with the difference in quantified resolution achieved using a fully sampled tilt series with 2° tilt increment and one with 8° tilt increment (6.31 and 11.7 nm, respectively) or between 30% I-DS tilt series with 4° and 6° tilt increment (8.21 and 8.8 nm, respectively). The resolution was estimated by Fourier shell correlation, wherein two volumes are reconstructed, each based on half of the available tilt series. We note though that Fourier shell correlation of reconstructions with large tilt increments can lead to errors in the resolution estimation and must be carefully interpreted and complemented with qualitative evaluation; for example, the reconstruction with 100% I-DS and 8° tilt increment is visually of higher fidelity than the reconstruction with 10% I-DS and 2° tilt increment (based on Figures 3 and 4), even though the estimated resolution is worse.

Figure 3

CS-ET reconstructions from a combination of I-DS (rows) and T-DS (columns) settings. (a-d) correspond to 100% I-DS setting, (e-h) to 60%, (i-l) to 30% and (m-p) to 10%. Each row shows reconstructions obtained with 2°, 4°, 6° and 8° T-DS settings. I-DS tilt series were restored by TV inpainting prior to CS-ET reconstruction. Red insets in (d) and (f) show the reconstructed pores obtained with a relative dose of 0.25 and 0.3, respectively, compared to the fully sampled reconstruction (yellow inset in (a)). Green insets in (k) and (m) both show the reconstructed pores obtained with a factor of 0.1 dose reduction but using different I-DS and T-DS settings. Scale bar: 200 nm.

Figure 4

Intensity profiles from the CS-ET reconstructions using different I-DS and T-DS settings. The profiles are taken at the position across two pores of the shell, as indicated by the yellow line on the inset orthoslice. Scale bar: 200 nm.

In general the level of T-DS permissible is highly dependent on the complexity of the particular object under study, the prior knowledge incorporated in the CS-ET optimization and the level of detail desired in the ET reconstruction.

I-DS can also have a noticeable impact on the reconstruction, potentially even more than T-DS. For this data set, 60% and 30% I-DS lead to similar results. The small pores are recovered in both settings, but with a better visibility in the case of 60% I-DS, which correlates with the lower RMSE of the TV-inpainted tilt series (see Figure 2c,f). This impact of I-DS is also reflected by the higher resolution achieved with 60% I-DS and 4° increment (7.69 nm) compared to 30% with 4° increment (8.21 nm). The 10% I-DS reconstructions retrieve the overall shape of the NW but fail to recover details of both the outer branching of the shell and the pores, even with 2° tilt increment (green inset in Figure 3m). This is because the TV-inpainted projections (Figure 2i) do not contain the information about such fine structures. Nonetheless, 10% I-DS might be sufficient for ET of nanoparticles with basic morphology and may be necessary for 3D dynamic studies.

It is worth noting that various I-DS and T-DS settings with similar dose lead to reconstructions with different resolution (Table 1). For example, 60% I-DS with 4° increment yields a markedly better reconstruction than 100% I-DS with 8° increment (red inset in Figure 3d,f and corresponding line profiles in Figure 4), even though the relative dose is similar (0.3 and 0.25, respectively). Likewise, both 30% I-DS with 6° increment and 10% with 2° increment correspond to a relative dose of 0.1, but the first setting leads to higher fidelity reconstruction than the latter (see green insets in Figure 3k,m and corresponding line profiles in Figure 4). A CS-ET reconstruction of the entire NW using the 30% I-DS and 6° increment setting is displayed in Figure 5 and is comparable to the fully sampled reconstruction in Figure 1 - recovering the salient details of the nanoporous structure of the shell and the integrity of the core (see cross-sections through the NW in Figure 5c,d,e and the movie in Additional file 9).

Table 1 Dose reduction by I-DS and T-DS approaches

Figure 5

CS-ET reconstruction of the entire NW from a dataset with 30% I-DS and 6° T-DS, corresponding to a dose reduction of 10×, compared to Figure 1. (a) voxel projection visualisation of the CS-ET reconstruction. (b) xy cross-sectional slice. (c, d, e) xz cross-sectional slices, at y positions indicated in (b).

From these results, it seems crucial, for a successful dose reduction in ET experiments, to select I-DS and T-DS settings prudently, taking into account the structural complexity of the object to be reconstructed.

Discussion

In this paper, TV inpainting and CS-ET were combined to simulate low-dose ET experiments with random STEM pixel subsampling and reduced numbers of tilt series projections. It was found that a dose reduction of 10× still enabled high fidelity reconstructions to be made. Although these proof-of-principle simulations have been performed on conventional HAADF-STEM projections, an important impact on multi-dimensional microscopy is expected: EDX and EELS maps are often acquired with a short dwell time, leading to low SNR signals and difficulties in applying spectral analysis techniques. Both I-DS and T-DS approaches may provide the signal level required for reliable processing of the sampled pixels. In the case of in situ tomography, I-DS and T-DS strategies may enable the study of changes at finer time scales. There is also the potential, with both the TV inpainting and CS-ET reconstruction, to incorporate prior knowledge (viz. sparsity) over time to further enhance the reconstructions and/or enable even greater subsampling.

The proposed approach may be further optimised by exploring other tilting strategies such as the Saxton scheme [33] or equally sloped tomography [34] or other tilting regimes specifically adapted for particular scanning trajectories or sample geometry. Moreover, the TV inpainting and CS-ET algorithms are closely related, making it possible to incorporate the inpainting approach into the CS-ET reconstruction. In general, performing the CS-ET optimization in 3D, with prior knowledge about the full object rather than individual 2D slices, should yield higher fidelity ET reconstructions (see [10] for an illustrative demonstration).

It must be emphasised that the degree of downsampling is strongly dependent on the level of detail in the object to be reconstructed: While a dose reduction of 10× was demonstrated on the CoPC/ZnO NW, the dose can probably be reduced even further for constant-density nanoparticles with a regular shape but may need to be increased for identifying, for example, the distribution of small particles on a complex support. As seen in Figure 3 the quality of the reconstruction can depend heavily on the quality of the restored I-DS tilt series. Therefore, it is crucial to optimise downsampling by exploring various scan patterns, interpolation techniques and tilting regimes that are best tailored to the specimen under consideration and practical implementation.

Scanning trajectories can be scripted (for example, using the Gatan Digiscan system) and should be designed in such a way that mechanical and electronic instabilities are minimised.

Finally, in addition to enabling multi-dimensional ET of a wide range of nanomaterials in the physical sciences, I-DS and T-DS may be of significant interest to the life science community, where samples are often extremely dose-sensitive.

Conclusions

This paper has explored novel acquisition and recovery strategies for ET experiments with reduced dose and acquisition time. The combination of TV inpainting and CS-ET reconstruction was found to yield reliable reconstructions from significantly undersampled datasets. The next steps are to address the practical implementation of the proposed strategies and to further develop the recovery procedures.