Applying compressive sensing to TEM video: a substantial frame rate increase on any camera

CS background

In imaging problems, the signal has two spatial dimensions, so the basis must also have two spatial dimensions. Often, small two-dimensional images (and higher) are referred to as patches. Figure 2 shows the two-dimensional Haar wavelet basis alongside the discrete cosine basis (DCT)—the real part of the Fourier transform. The basis patches along the top and left sides are the same as the one-dimensional basis elements, except they have been copied to fill the second dimension. The interior of the table is formed by combining the basis patches along the top and left edges into all of the possible two-dimensional variants.¹

In order to omit pixels, there is a single 1 in each column; another way of stating this is that the rows are randomly selected from the identity matrix without replacement. The representation in Fig. 3 includes zero rows for illustrative purposes, but the sensing matrix does not have those zero rows. Because the sensing matrix is missing rows, it is short and wide, that is \(\boldsymbol {\Phi }_{i} \in \mathbb {R}^{Q\times P}, Q\ll P\), where Q is the dimension of compressed measurement, \(\boldsymbol {y}\in \mathbb {R}^{Q}\), and P is the dimension of the signal \(\boldsymbol {x}\in \mathbb {R}^{P}\). The inverse problem of recovering x from y is underdetermined, so further assumptions must be imposed to guarantee a solution.

In sensing problems where the signal is an image, the signals {x ₁,…,x _N } are patches from the full image. Usually the patches are overlapping so that each pixel has a corresponding patch, except for the right and bottom regions of the image. Figure 4 is an illustration of the patches and how they overlap. The sensing matrices, measurements, and signals are all obtained by extracting patches from the corresponding full-size images. In the case of the signal, the CS algorithm will recover the patches x _i and then the patches are put back together and the overlapping pixels are averaged.

Dictionary learning and sparse-CS

Dictionaries are another choice for the basis, but dictionaries do not have an analytical form like the Fourier or wavelet bases. Dictionary learning is a method to discover a frame⁴ from the data, which is referred to as the dictionary. The learned dictionary allows every patch to be represented by a weighted sum of a few⁵ dictionary elements or vectors (assuming overcompleteness). Because the overcomplete dictionary model enforces the use of only a few basis patches, the data is sparse under the dictionary. This approach is advantageous because the learned dictionary can guarantee a sparse representation, whereas choosing a Fourier basis, for example, does not guarantee sparsity. Two learned dictionaries are depicted in Fig. 5.

The first algorithm for dictionary learning was based on human vision [26]. More recently, a much faster variant was proposed, the K-SVD algorithm [27], and Mairal et al. have further improved the K-SVD-based approach and given a thorough review of dictionary learning [16]. Another approach, a part of the approach in this paper, is beta-process factor analysis (BPFA) [25]. BPFA has been used in compressive sensing of STEM images [13]. The relationship between optimization/maximum likelihood (K-SVD) approaches and Bayesian/sampling (BPFA) approaches is discussed after the details of the BPFA model are introduced.

Another approach that has been applied in image restoration tasks, and specifically to STEM image restoration is the non-local means algorithm [16, 28]. Non-local means uses all of the image patches simultaneously to find a reweighting of the central pixel of each patch. Sparse representation, on the other hand, finds a subset of elements from a dictionary and the corresponding weights to reconstruct an entire patch (dictionary learning simultaneously finds a dictionary). Non-local means is a kernel density estimation method, and when employing the Gaussian kernel, it is closely related to the GMM, which will be explained in detail.

One of the approaches to guarantee that the solution of the underdetermined system of Eq. (2) is the desired solution is to assume there is a sparse representation under some basis/frame (e.g., Fourier, wavelets, or a learned dictionary). This means that

$$\begin{array}{*{20}l} \boldsymbol{x}_{i} &= \boldsymbol{D}\boldsymbol{w}_{i}, \end{array} $$

(3)

$$\begin{array}{*{20}l} \boldsymbol{y}_{i} &= \boldsymbol{\Phi}_{i} \boldsymbol{D}\boldsymbol{w}_{i}, \end{array} $$

(4)

where the columns of D=[d ₁,…,d _K ] are the dictionary elements. The number of non-zero elements in w _i is much less than the size of the basis K (number of columns in D), nnz(w _i )≪K. The choice of basis is important since it should induce sparsity in the w _i . The issue of the CS inverse being underdetermined is alleviated by finding solutions w _i that are also sparse. In practical applications, the noise ε _i must also be considered

$$ \boldsymbol{y}_{i} = \boldsymbol{\Phi}_{i}(\boldsymbol{D}\boldsymbol{w}_{i} + \boldsymbol{\epsilon}_{i}). $$

(5)

In the Fourier example above, the signal is recoverable as long as the noise amplitude is not larger than the amplitude of the smallest signal component. The same idea holds for sparse CS.

There are a few applications of sparse-CS in electron microscopy. The first was using ℓ ₁ and total variation (TV) regularization to simulate compressive sensing on STEM images and speculate about the application to STEM tomography [12]. It has also been shown that TV regularization is useful in electron tomography [29]. Tomography is closely related to CS, and even more so in electron tomography where it is common to have a missing wedge of data due to the inability to acquire all of the projections. More recently, BPFA has been applied to STEM compressive sensing [13], and an optimization approach is reported for compressed STEM imaging and tomography in [24].

Manifold-CS

A more recent approach in CS is to assume that the signal is a manifold embedded in a high-dimensional space [30]. Essentially, the intrinsic dimension of the data is smaller than the ambient dimension. Manifold-CS enjoys higher accuracy because the model is more flexible than sparse-CS [31] (sparse-CS is a special case of manifold-CS). A simple example of a manifold is a tube or a sheet through a three-dimensional space that is not self-intersecting. The concept of two-dimensional materials, such as graphene, is similar to the concept of a manifold in an N-dimensional space. Another example of a manifold is face images [32]. As the face image changes from happy to angry, as the lighting changes from light to dark, or as the face turns from right to left, the coordinates of the data move along constrained sections of the ambient space—the face manifold. This is not the same as moving along the principal dimensions defined by a principal components analysis (PCA). Manifold approaches learn local structures, whereas PCA-like methods learn global structures.

The concept of compactness from mathematical topology ([33], Chapter 3) states that a set, such as a manifold, can be covered by a finite number of open sets from the N-dimensional space.⁶ There is no specific structure required for the covering sets, so they can be assumed to be Gaussian, i.e., ellipsoids. Figure 6 shows the covering of a one-dimensional manifold (a curve) through a two-dimensional space. It can be seen that in order to use this approach the centers, orientations, and radii of the ellipsoids must be determined. Furthermore, any point on the manifold can be approximated arbitrarily well by this method simply by increasing the number of ellipsoids and also shrinking them to have a tighter fit. Statistically, having too many ellipsoids can cause undesirable overfitting effects, and mathematically, the number of ellipsoids (if it can be determined) is closely related to the manifold condition number.

The manifold-CS model described above is known in statistics as a mixture of factor analyzers (MFA). MFA combines the Gaussian mixture model (GMM) and factor analysis. In MFA, the GMM determines the number of ellipsoids and the factor analyzer determines the statistics of each ellipsoid (location, orientation, and radii). Connecting the pixel omission example in Fig. 3 to the MFA is the final piece in CS-MFA. Figure 7 illustrates the omission of dimensions of the measured data. The compressed data lies along the x- and y-axes. The CS inversion process—recovering the signal from compressed measurements—must take compressed measurements and map them back to the signal manifold. The model parameters learned by the MFA make this feasible by constraining the inversion procedure to the manifold.

One difficulty with the standard version of the GMM and factor analysis is that the number of clusters and dimension of the basis must be set a priori. Cross-validation can be employed to determine the parameter settings, but it requires splitting the data into several sections and learning the model on each section. Bayesian nonparametrics [34] offers a solution to this problem by including these parameters in the inference of the model. The rest of this section will describe the mathematical details of the GMM, factor analysis, their nonparametric extensions, the MFA, and a description of the hardware needed for a TEM to collect data that can be inverted by CS-MFA.

Gaussian mixture model

The approach in this paper for manifold-CS is to model the manifold as an MFA. The mixture part of the MFA finds the number of ellipsoids needed to cover the manifold. The mixture part of MFA is based on the GMM, a model for clustering real-valued data. Figure 8 shows a set of two-dimensional data that was generated from a GMM. The primary goal in clustering is to determine which cluster each item belongs to and once this has been determined, cluster statistics such as mean and variance can be determined. Meeting this primary goal is easily accomplished by methods such as K-means. But the GMM goes beyond the primary goal by also finding the uncertainty parameters in the cluster assignments. In Fig. 8, several points lie in the overlap of two ellipses, with K-means they would simply be assigned to the nearest ellipse. In some applications, it may be important to know how strongly the algorithm believes a data point belongs to a cluster; this information can be inferred with the GMM.

where t(i) is the cluster number of the ith data point and \(\mathcal {G}(\cdot,\cdot)\) is the gamma distribution, the conjugate prior for the precision of a normal distribution. The weight λ _i determines the proportion of the data in cluster i. In Eq. (7), the cluster is known, so the probability is simply defined by the statistics of that cluster. The mean and precision of each cluster are given by Eqs. (8)–(9). The hyperparameters a,b,c,d are usually determined using the mean and precision of the entire data set. The cluster proportions are sampled jointly from a symmetric Dirichlet distribution in Eq. (10). The Dirichlet distribution is a multivariate extension of the beta distribution, where each λ _t ∈[0,1] and \(\sum _{t=1}^{T} \lambda _{t} = 1\). The parameter α>0 determines the decay rate of λ ₁,…,λ _T and will be discussed more below. Finally, the latent cluster assignments are drawn from a multinomial distribution based on the cluster proportions. The multinomial distribution is a generalization of the Bernoulli distribution; n trials (data points) are performed with a chance of success in exactly one of k different categories (clusters).

where t(−i) is the list of all cluster assignments except the ith and n _{−i j} is the number of items in cluster j, excluding item i.

Returning to the number of clusters, it was previously mentioned that it is possible to infer the number of clusters using Bayesian nonparametrics. For the GMM, the nonparametric model is known as the infinite GMM and is produced by modifying the Dirichlet distribution to be a Dirichlet process (DP). There are a few analogies for the DP that have been well circulated in the statistics literature, the Chinese restaurant process (CRP) and the stick breaking process (SBP). In this paper, the CRP and SBP, which are equivalent to the DP, will be introduced; theoretical details of DP mixture models can be found in [17, 35, 36].

where n is the current number of customers, n _t is the number of customers at table t, and α is the parameter related to the rate new tables are set up. To form a draw from a CRP, the infinity of customers are seated at their tables sequentially and after every customer has been seated the proportion of customers at each table determines \(\{\lambda _{t}\}_{t=1}^{\infty }\). The CRP representation clearly shows the influence of α on the thickness of the tail of the proportions; increasing α increases the tail thickness. This countably infinite set of proportions replaces the finite number of proportions in the GMM. Informally, if T→∞ in Eq. (12), then limiting cases are given by Eq. (13). Once the proportions have decayed past a certain level, the remaining proportions are set to zero and the number of tables (clusters) can be determined. Figure 9 depicts the seating arrangement and assignment probabilities for a new customer after several customers have been seated.

and replaces Eq. (10) in the infinite GMM. As with the CRP, the SBP can be terminated when the proportions are sufficiently small. Figure 10 illustrates the stick breaking process.

Factor analysis

Using the singular value decomposition (SVD), \(\boldsymbol {DD}^{\top } = \sum _{k=1}^{K} \sigma _{k} \boldsymbol {v}_{k} \boldsymbol {v}_{k}^{\top }\), where the singular vectors v _k are orthonormal and the singular values σ _k >0. The singular values are the radii of a K-dimensional ellipsoid and the singular vectors determine the orientation of each dimension (assuming \(\gamma _{\epsilon }^{-1} < \sigma _{K}\)). Figure 11 illustrates the singular values and the mean. Note that probabilistic PCA is different from PCA, which is simply a projection onto the top K principal components (either via SVD of the data or eigen-decomposition of the data covariance matrix) [37].

where K is the number of dictionary elements and a,b are hyperparameters. For each \(\boldsymbol {x}_{i}\in \mathbb {R}^{P}\), the latent binary vector \(\boldsymbol {z}_{i}\in \mathbb {R}^{K}\) encodes which dictionary elements are used by x _i . The proportion π _k is the sharing mechanism and encodes the average use of basis vector k across all of the selection vectors z _i .

The metaphor used to describe the BeBP is the Indian Buffet Process (IBP). In the IBP, customers (data points) enter the restaurant and choose dishes (dictionary elements) from the buffet. The first customer chooses Poisson(a) dishes. The ith customer samples each old dish with probability #(previous samples)/i and samples Poisson(a/i) new dishes. This is the single parameter IBP with b=1. Figure 12 illustrates the process. As the number of customers i tends to infinity, the number of new dishes tends to zero. In practice, the IBP is truncated to a number of dishes sufficiently large (i.e., large enough that some dishes are unused with high probability—this is data dependent) and any dishes that are unused can be removed from the representation. Details about the IBP and BeBP can be found in [17, 25].

where Eqs. (23)–(25) have replaced the expression for w _i in the PCA model, ∘ is the element-wise Hadamard product, and the product notation in 26 and 27 denotes independent draws. The mean μ has been omitted in (19), since in the case of a single factor analyzer, the mean can simply be subtracted from the data as a pre-processing step. When implementing the algorithm, the hyper-parameters a,…,f are set to so-called non-informative values.

which is minimized to find the latent parameters. The first term is the least square error between the inferred parameters and the data while the second and third terms are commonly used as smoothing regularizers. The fourth term is the sparsifying regularizer, similar to the ℓ ₁ norm. The BPFA model is commonly implemented using Gibbs sampling or variational Bayesian methods [25, 30]. It must be emphasized that Eq. (28) is not used by sampling algorithms and cannot be optimized with traditional approaches. For more details about beta process dictionary learning including the application to three-dimensional data, see [38].

Mixture of factor analyzers

where γ _ε,t ,γ _s,t ,τ _tk ,τ ₀ all have gamma hyperpriors. Equation (30) says that data point i is in a cluster with statistics given by factor analyzer t(i). Equations (31)–(33) give a basis representation where Σ _t(i) is a diagonal matrix similar to a singular value matrix that weights the contributions of each basis vector. If some of the (diagonal) elements of Σ _t are small relative to the noise variance, then that component t(i) will be low rank.

where only one of the vectors w _t is non-zero. In this way, only a single block or group is active, which also makes the representation sparse. If there is only a single ellipsoid in the model, then the sparse-CS formulation is recovered as a special case.

In addition to having a block-sparse structure, the nonparametric MFA usually infers bases that are low-rank, K<P. Low-rank Gaussian bases correspond to localized tubular manifolds. In [30] the fact that the signal is 1-block sparse is used to prove the reconstruction guarantee. Theorems for the separability of the components and satisfaction of the restricted isometry property (RIP) can also be found in [30]. Essentially, the number of measurements should be greater than a constant times the largest rank among all of the D _t plus the log of the number of components. The largest rank is the intrinsic manifold dimension, while the number of components T is related to the manifold condition number.

CS-MFA

The prior predictive distribution is obtained when ξ _t =0 and Λ _t =I _P , however this is usually inaccurate, so the posterior parameters are obtained by calculating the mean and covariance of the Gibbs samples. The bases \(\tilde {\boldsymbol {D}}_{t}\) are also taken as the mean of the Gibbs samples.

The representation in Eq. (44) admits an analytic CS inversion procedure, that is, once the model parameters are learned (either offline or online [22, 41]), new signals are recovered by matrix–vector operations.

Description of CS-TEM hardware

where the image size is N _x ×N _y pixels. As previously mentioned, the images are broken down into patches so the data points x _i in the MFA model are of size 4×4×L.

In order to obtain compressed measurements suitable for CS-MFA, the coded aperture compressive temporal imaging (CACTI) approach described in [23, 42] is used. CACTI was developed for optical video CS. In the CACTI camera system, the signal passes through a coded aperture that changes at a faster rate than the camera obtains images. This causes multiple coded images to be integrated into a single image. The aperture is set on a piezoelectric stage. The stage moves along either the x- or y-axis according to a triangle wave. During an up-stroke, a set of coded images are integrated and then another set are integrated during the down-stroke. A function generator is used to drive the piezo stage and trigger the image capture on the camera at the troughs and peaks of the triangle wave. The same setup is possible in TEM. The major difficulty in moving this approach to TEM is designing an aperture to block electrons rather than photons. Figure 13 shows an illustration of the TEM-CACTI system.

The benefit of placing the mask on a moving stage is that moving the mask creates a new encoding—essentially a new mask. If the position of the mask is known, then the encoding is known. This overcomes a difficulty in CS of using a new mask for every measurement. The compression ratio is determined by the range of motion of the mask. Effectively, moving n pixels (mask feature size) will give a factor of n compression, or n frames from 1.

Another difficulty—present in CS for TEM, but not in optical CS—is that the part of the mask blocking the signal must be supported by a material transparent to electrons. Example masks that allow approximately 50 % of electrons to pass are shown in Fig. 14. An issue that might be raised about this approach is that 50 % of the image is discarded. The intent of our approach, however, is to increase the acquisition rate. It has been shown that image data can be discarded and subsequently recovered, both generally [25] and in electron microscopy [13]. Moreover, it might be possible to place the aperture before the specimen, which would give a decrease in dose and an increase in acquisition rate.

Palladium nanoparticle oxidation

To demonstrate the applicability of coded aperture CS video reconstruction for atomic resolution imaging, we show observations from Pd nanoparticles during exposure to elevated temperature and an oxidizing environment. Supported Pd nanoparticles are used extensively in catalytic applications under high temperatures and in reactive gas environments. The ability to visualize and characterize morphological, structural, and surface transformations associated with environmental exposure under in-situ conditions at high temporal resolution is critical for rationalization of structure–property relationships and thus essential for future advancement of catalytic technologies.

The observations here focus on characterization of atomic level processes associated with a formation of a surface oxide in the initial stage of oxidation. In particular, the observations show how adsorption of oxygen and interaction with a SiN_x support lead to subtle morphological changes, and subsequently to a formation of surface oxides. The observations were performed with an environmental FEI Titan 80–300. The microscope is equipped with CEOS aberration corrector for the image-forming lens, which allows imaging with Ångström resolution. The images were acquired with Gatan’s Ultra-Scan 1000S CCD camera, and the acquisition was performed in Digital Micrograph (DM) at the frame rate of 1.1 frames/s. The observations were performed at oxygen partial pressure of 10⁻² mbar at 500 °C. Heating of the samples was done with an Aduro Protochips heating holder.

From the originally recorded video of Pd oxidation, which is available in the supplementary information, CS video reconstruction was simulated by integrating every 10 aperture-coded frames into a single measurement frame. A subset of 10 original images, the integrated coded image, and reconstructed images are shown in Fig. 15. The comparison in Fig. 15 shows very good agreement between the original and reconstructed images. Figure 16 shows the last frame recovered from a set of compressed frames—the peaked feature is accurately recovered. The reconstruction preserves the atomic resolution in the bulk portion of the nanoparticles with a small loss of resolution observed in the interfacial region. The peak signal to noise ratio (PSNR) for each recovered frame is shown in Fig. 17. The large drop in PSNR is due to misalignment, and after registering the reconstructed image with the original, the PSNR is 16.95 dB. The reason for the relatively low PSNR overall (despite the fact that the reconstruction looks good, Fig. 18) is due to the fact that the reconstructed image is denoised as a side effect of reconstruction. Moreover, the top and left edges of the image (10 pixels) are mostly lost due to the coding process.

Silver nanoparticle coalescence

Using aberration-corrected environmental TEM, heterogeneous catalysts surface restructuration by gas molecules [43], the sintering mechanisms of supported metal catalysts [44], and other structural changes in a gaseous environment [45], can be studied at the atomic scale under gas pressures of up to 20 Torr. For gas pressures closer to catalytic conditions, up to 1 Atm, subnanometer resolution can be achieved by using dedicated gas cell holders [46, 47]. In order to gain in-situ information at the atomic level, highly magnified imaging is required. Typically, an increase in magnification results in the electron beam having to be focused onto a smaller area in order to keep the number of electrons per pixel constant. This increase in the electron dose will ultimately lead to an increase of possible beam damage effects that can influence the process. Here we show an example of metallic particle coalescence induced merely by parallel electron beam illumination in TEM. While our experiments have been done for 60 nm Ag particles, we expect additional or more pronounced beam effects for the case of smaller particles. This is most relevant for catalysis applications, since particle mobility during sintering will be higher.

Figure 19 shows a sequence of bright field TEM images of the electron beam-induced coalescence of six Ag nanoparticles supported on amorphous carbon. Commercially available 60-nm-diameter Ag particles (0.02 mg/mL in aqueous buffer, Sigma-Aldrich) were drop cast on a holey carbon film (Ted Pella, Inc.). As with the previous example, the in-situ TEM videos were acquired using an 80–300 keV FEI Titan environmental TEM equipped with an objective-lens spherical aberration corrector and operating at 300 keV and in high vacuum mode. Changes in image contrast are observed, indicating particular dynamic processes, such as the formation of cavities, localized areas with lighter contrast within the particles and adjacent to areas displaying surface expansion, and diffraction contrast due to recrystallization, apparent as broad linear contours. Mass transport is apparent as progressive changes in contrast from the darker particles to the lighter inter-particles and the surface of newly formed areas. After about 10 min of electron-beam irradiation, a recrystallization front is formed and advances from the top left corner of the forming crystal down. After 13 min of irradiation, formation of facets on the recrystallized surface is also observed. The mass transport during irradiation, as shown in the snapshots for the first 2 min of the process, occurs first at the sintering neck between particles and homogeneously around their surfaces on the outermost particles. This indicates that surface diffusion is a main mechanism driving the coalescence process under the electron beam. This observation is in good agreement with previous works [48, 49].

A set of reconstructed frames are shown for comparison in Fig. 20. The images in Fig. 20 are qualitatively accurate when compared to Fig. 19. In the last 200 frames, the specimen begins to drift up and left. The reconstruction quality diminishes during this phase, as shown by Fig. 21, since the training data did not include drift dynamics. Of note, however, is a bright flash that occurs near frame 850, the reconstruction completely eliminates this transient effect (the image became mostly white in a few frames and then returned to the original contrast over the same period). Moreover, the speckled noise is also removed—it is especially apparent in the background of the original data. The reconstruction PSNR of the Ag nanoparticle experiment is relatively higher than the PSNR of Pd reconstruction because the noise in the original Ag data is much lower.

Compression versus reconstruction quality

The final experiment compares the reconstruction quality over several compression levels. Figures 22 and 23 show the average PSNR across all video frames as a function of the compression factor. These curves are approximately logarithmic. As the compression factor increases, the average PSNR decreases more slowly. This kind of saturation occurs because the reconstructed image is increasingly smooth, but still maintains the average image, thus it cannot have a very low PSNR.

For comparison, the average PSNR of linear interpolation is also plotted. This is simply a baseline, it would be difficult to do worse with a principled approach. For example, when the video has been subsampled with compression factor 2 (i.e., every other frame is missing), the interpolated result is the average of the previous frame and the next frame. The compressed video used for the interpolation results is simply subsampled at the rate corresponding to the compression factor. To compute the average PSNR, all of the sampled frames are omitted, since their PSNR is infinite. Therefore, the comparison is between the inferred frames of both methods.

The comparison between CS-MFA and interpolation shows that the compressed frames contain significant information. CS-MFA is able to exploit this information to achieve accurate results for a wide range of compression factors. Moreover, the variance in the reconstruction PSNR is relatively small and does not increase with the compression factor.

Finally, reconstructed frames from both movies at 10 ×, 20 ×, and 30 × compression can be seen alongside the original frames in Figs. 24 and 25. As the compression level increases, the image contrast decays. Many of the important structures are still visible in reconstructed images, and the reconstructed images are denoised. The edge artifacts in the palladium video are from an image alignment that occurred prior to the CS simulation.

It is difficult to decide from Figs. 22 to 23 what the maximum compression factor should be. The reconstructed images degrade very smoothly. Upon inspection of the reconstructed videos (included in the supplementary material), a compression of about 15 × seems feasible for the palladium nanoparticle video and about 20 × for the silver nanoparticle video. The compression factor also depends on what image features are important; this is a tradeoff between speed and image clarity.