Poisson noise removal from high-resolution STEM images based on periodic block matching

Abstract

Scanning transmission electron microscopy (STEM) provides sub-ångstrom, atomic resolution images of crystalline structures. However, in many applications, the ability to extract information such as atom positions, from such electron micrographs, is severely obstructed by low signal-to-noise ratios of the acquired images resulting from necessary limitations to the electron dose. We present a denoising strategy tailored to the special features of atomic-resolution electron micrographs of crystals limited by Poisson noise based on the block-matching and 3D-filtering (BM3D) algorithm by Dabov et al. We also present an economized block-matching strategy that exploits the periodic structure of the observed crystals. On simulated single-shot STEM images of inorganic materials, with incident electron doses below 4 C/cm ², our new method achieves precisions of 7 to 15 pm and an increase in peak signal-to-noise ratio (PSNR) of 15 to 20 dB compared to noisy images and 2 to 4 dB compared to images denoised with the original BM3D.

Background

Modern electron microscopy allows for atomic resolution images of crystalline structures [1]. However, the full scope of resolution can be exploited only for materials with little electron beam sensitivity. Lowering the electron dose decreases the signal-to-noise ratio (SNR) of the acquired micrographs accordingly, degrading the quality or even completely prohibiting the extraction of desired information from the noisy micrographs. Examples of inorganic materials with high beam sensitivity, where scanning transmission electron microscopy (STEM) images of poor SNR have to be used, are both oxide [2] and metallic [3] catalysts. One important quantity that may be extracted from atomic-resolution electron micrographs is the positions of the atoms. The precision with which these can be determined is crucial for the understanding of certain material properties [4,5]. For single-shot STEM images, the best reported precision is about 15 pm [6]. A common technique to increase the precision is to register a series of frames and extract the atom positions from the average of the registered frames. Kimoto et al. [7] achieved a precision of 5 pm using rigid registration. More recently, a non-rigid registration scheme has been developed by Berkels et al. [2] that has achieved sub-picometer precision for STEM series [8]. Both registration schemes require many individual frames of moderate SNR resulting in a very-high overall electron dose applied to the material. Thus, to widen the applicability of STEM to more beam-sensitive inorganic materials, a central objective is to develop effective denoising methods that increase the single-shot image quality. This would enable the extraction of desired information from individual frames or the use of the individual frames for registration, while using a lower electron dose.

Denoising in materials, electron microscopy is often accomplished by simple spatial filters like a Gaussian blur or median filter or frequency space low-pass or Wiener filters (cf. [9-11]). These examples are all taken from materials robust enough under the electron beam to survive relatively high electron dose to obtain images with high spatial oversampling (small pixels). More sophisticated approaches that take advantage of the highly redundant nature of some high-resolution STEM images have been employed [12], building on tools developed for very beam-sensitive biological samples [13-16]. While these methods, mostly based on manipulations of the image in Fourier space, are well established and have proven to perform well on high-resolution transmission electron microscopy (HRTEM) images of organic materials, they are tightly linked to the periodicity of the data and have thus not been widely adopted in STEM studies of aperiodic defects in inorganic materials.

The most successful generic image denoising method for arbitrary images available today rely on non-local detection and averaging of self-similar image regions. The first algorithm based on this strategy is the non-local means filter (NLM) by Buades et al. [17]. Due to the richness in self-similarity of electron micrographs of crystals, NLM is in principle very well suited for denoising such micrographs [18], and it has been employed in biological electron microscopy [19]. However, two main properties of low-dose electron micrographs of crystals are disregarded by the original design and implementation of NLM, namely (1) NLM removes additive Gaussian white noise (AGWN) instead of Poisson noise, which is the dominant form of noise in low-dose STEM images; and (2) NLM uses a local similarity search to reduce the computational cost, whereas the distance between self-similar points in images of crystals is at least as large as the inter-atomic distance. In the case of STEM, there is an additional difficulty caused by image distortions resulting from the serial acquisition of the pixel data. In view of these issues, we proposed an enhanced version of NLM tailored to STEM imaging [20]. The most significant enhancement was the development of an efficient non-local similarity search, based on the generation of periodic lattices in Fourier space [21]. However, the basic NLM principle can be further improved by replacing the simple weighted average of intensities of pixels with similar neighborhoods by a more advanced collaborative filtering of the corresponding image parts. Several methods of this type have been proposed over the past few years, e.g., optimized block-wise NLM by Coupé et al. [22] and NLPCA by Salmon et al. [23], which is based on principal component analysis. Among the most successful variants is the block-matching and 3D-filtering algorithm (BM3D) developed by Dabov et al. [24], which has become a benchmark for image denoising algorithms in the field of image processing. The extension of BM3D to Poisson noise removal via application of the Anscombe variance-stabilizing transformation [25] has already been proposed by Mäkitalo and Foi [26] who have contributed an exact unbiased inverse transformation that increases the accuracy especially in the low-count regime. In [20], it was found that despite being restricted to a local similarity search, the original BM3D filter with extension to Poisson noise outperformed the proposed periodic search NLM filter. Thus, the starting point here is the BM3D filter.

In this paper, we discuss how modifications introduced in STEM-tailored NLM [20] can be incorporated into the state-of-the-art denoising algorithm BM3D in order to tap its full potential on electron micrographs of crystals. More precisely, as this method has been designed for Gaussian white noise, a central task is to develop suitable modifications that effectively deal with Poisson noise. Another focus is to exploit the atomic lattice structure that entails a repeated appearance of self-similar image components within a single frame. While we focus on the application of STEM imaging of inorganic materials, the key features of Poisson noise and oversampling are shared by HRTEM images as well, so the proposed method should also be applicable to HRTEM images. Also note that the procedures proposed in this paper can be applied subsequently for registering a series of low-dose frames to a single reference frame [2].

The paper is organized as follows. First, we briefly recall the original BM3D algorithm. Then, we present two strategies for Poisson noise removal, namely variance stabilization based on the Anscombe transform and using Poisson maximum-likelihood similarity measures due to Deledalle et al. [27] for the block matching. The main contribution is an adaptive non-local periodic block matching. It exploits the repetitive structure within the micrograph while keeping the computational cost affordable. Since the spatial distribution of the resulting block estimates is highly non-uniform, potentially resulting in a poor reconstruction in regions with few available block estimates, we discuss possible remedies of this deficiency. Finally, we evaluate the proposed method on simulated data, where the ground truth, i.e., the true, noise-free mean electron count per pixel, is available, showing the improvement both in terms of visual image quality and in quantitative measures such as peak signal-to-noise ratio (PSNR), precision, and fidelity achieved by the proposed method.

Methods

Block matching and 3D filtering

BM3D was developed on the basis of NLM. It was originally designed to remove additive Gaussian white noise from natural images or other images exhibiting a sufficient amount of self-similarity. The main idea of the algorithm is to find similar parts within an image and remove noise from those parts by sparsifying their common representation in a 3D transform domain. After processing all image parts in this manner, one receives an overcomplete representation of an estimate of the noise-free input image. Averaging all partial estimates in overlapping regions provides an estimate of the full underlying ground truth image. The denoising process in the 3D transform domain is performed in two stages, namely an initial hard thresholding succeeded by a final Wiener filtering. In the following, we briefly recall the definition of the original BM3D filter as published in [28], which serves as a basis for the methods developed in this work. We use a slightly simplified version of the algorithm described in [28] but with the exception of a change of the block size from 8×8 to 16×16, all relevant parameters, as well as the unitary transforms in the 3D domain, are chosen in exact agreement with the original implementation of the BM3D filter [29] when using the default profile, i.e., normal profile.

The original BM3D algorithm for Gaussian noise removal

Let us regard an image:

$$ \begin{array}{cc}z:X& =\left\{1,\dots, N\right\}\times \left\{1,\dots, M\right\}\to \mathbb{R},\\ {}x& \mapsto f(x)+\eta (x),\end{array} $$

((1))

where f is the (noise-free) ground truth and $\eta \left(\cdotp \right)\sim \mathcal{N}\left(0,{\sigma}^2\right)$ is AGWN, i.e., a normally distributed random variable with zero mean and standard deviation σ.

Let us fix some $n\in \mathbb{N}$ with n<N and n<M and some reference pixel:

$$ x\in \overset{\kern0.60em \circ }{X}:=\left\{\left(i,j\right)\in X:1\le i\le N-n+1,1\le j\le M-n+1\right\} $$

((2))

and denote by Z _x a block of size n×n with upper-left corner $x\in \overset{\kern0.60em \circ }{X}$. The size n (in units of pixels) should be chosen such that an n×n block is roughly between 0.25 and 1.25 times as large as an atom within the image. Furthermore, we denote with Z _x (y) for y∈X the values of the block in global coordinates, and with Z _x [ i,j] :=Z _x (x ₁+i−1,x ₂+j−1) for $i=1,\dots, n$, its values in local coordinates. Note that we set Z _x (y)=0 outside the block’s support. The block-matching part of BM3D consists of a local search for a tuple of blocks that are similar to Z _x in the sense that their normalized L ²-distance from Z _x is less than some threshold τ>0. More precisely, denoting by N _S the size of the local search window and by ${\mathcal{N}}_{N_S}(x)$ the N _S ×N _S neighborhood centered around $x\in \overset{\kern0.60em \circ }{X}$, let:

$$ {S}_x:=\left\{y\in {\mathcal{N}}_{N_S}(x):d\left({Z}_x,{Z}_y\right)<\tau \right\}, $$

((3))

where:

$$ \begin{array}{cc}d\left({Z}_x,{Z}_y\right)& :=\frac{1}{n^2}\parallel {Z}_x-{Z}_y{\parallel}_2^2\\ {}:=\frac{1}{n^2}\sum_{i,j=1}^n{\left({Z}_x\left[i,j\right]-{Z}_y\left[i,j\right]\right)}^2.\end{array} $$

((4))

We then sort the set S _x by block similarity, i.e., the set S _x becomes a tuple with $d\left({Z}_x,{Z}_{{\left({S}_x\right)}_i}\right)\le d\left({Z}_x,{Z}_{{\left({S}_x\right)}_j}\right)$ for 1≤i≤j≤#(S _x ). From all matched coordinates in S _x , we now choose the ordered subset ${\check{S}}_x=\left(\right.{\left({S}_x\right)}_i{\left)\right.}_{i=1}^{N_x}$ consisting of the ${N}_x:= \min \left\{\#\left({S}_x\right),{N}_{3\mathrm{D}}\right\}$ most similar blocks. Here, N _3D is some small number that controls the computational cost. Note that the choice of N _3D may also implicitly change the original threshold τ in (3) to a possibly smaller one. Stacking the best-matched blocks ${Z}_y,y\in {\check{S}}_x$ on top of each other yields an n×n×N _x 3D tensor:

$$ {\mathcal{Z}}_x\left[i,j,k\right]:={Z}_{{\left({\check{S}}_x\right)}_k}\left[i,j\right],\kern1em k=1,\dots, {N}_x, $$

((5))

with high correlation along the coordinate k. Figure 1 shows such a column of stacked blocks using a STEM image as an example. Due to the correlation along the coordinate k (indicated by a black-dotted arrow in Figure 1), an efficient denoising of the tensor ${\mathcal{Z}}_x$ can now be performed as follows. Let be some 3D unitary transformation (e.g., Fourier or wavelet transform), and let Υ _θ denote the hard-thresholding operator with threshold θ, which is chosen with some fixed proportionality to the standard deviation σ of the noise. The denoising procedure for the 3D tensor ${\mathcal{Z}}_x$ consists of a hard thresholding of the coefficients of the transformed tensor $\mathcal{T}\left({\mathcal{Z}}_x\right)$, followed by the inverse transformation. This leads to the following estimator:

$$ {\widehat{\mathcal{Y}}}_x:={\mathcal{T}}^{-1}\left({Y}_{\theta}\left(\mathcal{T}\left({\mathcal{Z}}_x\right)\right)\right) $$

((6))

Figure 1

Illustration of block-matching and 3D-filtering algorithm. The BM3D algorithm is demonstrated using a STEM image of a gallium nitride crystal as an example. The magenta/cyan squares resemble exemplary 16×16 reference blocks (two out of 58,081 possible ones in the 256×256 image). The red/blue squares mark blocks that were matched to the magenta/cyan reference. The matched noisy blocks are stacked into 3D tensors (top right), which is then denoised, e.g., by replacing all pixels with mean values along the column direction (indicated by the black-dotted arrow). The resulting denoised blocks (bottom right) are then aggregated at their original positions within the image (bottom left image), and overlapping parts (green) of adjacent blocks are averaged, yielding a single denoised image as the result.

for the tensor ${\mathcal{Z}}_x$. A typical transformation would be based, e.g., on a wavelet basis. To this estimator, one assigns a weight ${w}_x={N}_{\theta}^{-1}$, where N _θ is the number of retained non-zero transform coefficients after hard thresholding.

Such block estimators are calculated for each reference point in a subset ${X}_R\subset \overset{\kern0.60em \circ }{X}$ of all available block corners, given by:

$$ {X}_R:=\left\{\left(i,j\right)\in \ddot{X}:i\kern-1.0pt \equiv \kern-1.0pt 1\kern-1.0pt \mod \kern-1.0pt {N}_{\mathrm{step}}\wedge j\kern-1.0pt \equiv \kern-1.0pt 1\kern-1.0pt \mod \kern-1.0pt {N}_{\mathrm{step}}\right\}. $$

((7))

Choosing N _step<n, the corresponding reference blocks overlap. This guarantees an overcomplete representation of the estimated image. An intermediate estimator ${\widehat{f}}^{\mathrm{basic}}$ for the desired image f is then defined as a weighted average over all overlapping block estimations. More precisely, let ${\hat{Y}}_{x,k}:={\widehat{\mathcal{Y}}}_x\left[\cdotp, \cdotp, k\right]$, k=1,…,N _x , denote the kth slice of the estimator for ${\mathcal{Z}}_x$, which is a 2D block estimate corresponding to the block ${Z}_{{\left({\check{S}}_x\right)}_k}$ and set:

$$ {\widehat{f}}^{\mathrm{basic}}(y):=\frac{\sum_{x\in {X}_R}\sum_{k=1}^{N_x}{w}_x{\hat{Y}}_{x,k}(y)}{\sum_{x\in {X}_R}\sum_{k=1}^{N_x}{w}_x{\chi}_{Z_{{\left({\check{S}}_x\right)}_k}}(y)}, $$

((8))

where χ _Z denotes the characteristic function of the block $Z\subset X$. Note that each block Z _y may be matched to different references ${Z}_x,{Z}_{x^{\prime }}$, resulting in different estimates ${\hat{Y}}_{x,k},{\hat{Y}}_{x^{\prime },{k}^{\prime }}$ for the same block, i.e., ${\left({\hat{S}}_x\right)}_k={\left({\hat{S}}_{x^{\prime }}\right)}_{k^{\prime }}=y$ in this case.

The final estimate $\widehat{f}$ of the desired image f is now calculated in a second iteration, where the whole procedure described above is repeated with some modifications briefly indicated next, referring to [28] for the details. Denoting by F _x , the n×n block with upper-left corner x extracted now from the basic estimate ${\widehat{f}}^{\mathrm{basic}}$, the above procedure is applied to the blocks F _x in place of Z _x , where the analogs to the sets S _x in (3) are formed, however, with respect to a possibly different threshold parameter τ ^′. Moreover, the analog to (6) may involve a different unitary transformation (e.g., using local cosine transforms) and, perhaps more importantly, hard thresholding is replaced by Wiener filtering. Again, we refer to [28] for the details.

Figure 1 illustrates the BM3D procedure using a STEM image as an example. It demonstrates the three main steps of BM3D, which are performed for each reference block: 1) matching blocks to a reference and stacking the matched blocks into a 3D tensor, 2) denoising this column of matched blocks, and 3) aggregating the resulting denoised blocks at their original positions and averaging overlapping parts of adjacent blocks (green). For simplicity, the illustration uses simple averaging along the column of matched blocks (black dotted arrow) as an example procedure for denoising the 3D tensors. In practice, the approaches described above provide better performance.

Like any block-averaging or non-local means approach, our method has reduced performance near the edges of the image. Since N _step<n for every y∈X, there is a block with corner x∈X _R such that y∈Z _x . In other words, for any pixel in the image (even at the boundary), there is a block containing this pixel, resulting in an estimated intensity of the pixel after the corresponding block has been denoised in 3D transform domain. But because there exist fewer overlapping blocks near the image boundaries, there are fewer available estimates of boundary pixels that can be used for further averaging. Therefore, both the basic and final estimates ${\widehat{f}}^{\mathrm{basic}}$ and $\widehat{f}$, i.e., the intermediate result and the final denoised image, will show a slightly reduced quality towards the image boundaries. However, the denoising of each block together with its matched blocks in 3D transform domain contributes much more to the image quality than the averaging of spatially overlapping blocks. Thus, the reduced quality at the boundaries of images denoised with BM3D is usually not substantial in practice.

Extension to Poisson noise removal via variance stabilization

As mentioned before, BM3D is designed to remove additive Gaussian white noise of some uniform standard deviation σ from an input image. In order to adapt this filter to Poisson noise removal, Mäkitalo and Foi [26] proposed to use a variance-stabilizing transform (VST) that approximately transforms any given image affected by Poisson noise, i.e., of the form:

$$ v:X\to {\mathbb{N}}_0,\kern2.77626pt v(x)\sim \mathrm{Pois}\left(\lambda (x)\right), $$

((9))

into an image with AGWN of standard deviation one. Here, v∼Pois(λ) denotes a Poisson distributed random variable with mean (and variance) λ, i.e.:

$$ \mathrm{P}\left(v=k\right):=p\left(k\Big|\lambda \right)=\frac{\lambda^k{e}^{-k}}{k!}\mathrm{f}\mathrm{o}\mathrm{r}k\in {\mathbb{N}}_0. $$

((10))

The transformation:

$$ A(k):=2\sqrt{k+\frac{3}{8}} $$

((11))

is known as the Anscombe transform [25]. Mäkitalo and Foi proposed an exact unbiased inverse transform of the form [26]:

$$ {A}^{-1}:\mathbb{E}\left\{A(v)\Big|\lambda \right\}\mapsto \mathbb{E}\left\{v\Big|\lambda \right\}. $$

((12))

Here, given random variables V,W, $\mathbb{E}\left\{V\Big|W\right\}$ denote the expectation of V given W. In their implementation [30], Mäkitalo and Foi provide tabulated values of this inverse for arguments within the range [ A(0),100]. For smaller arguments, the inverse is extended by zero, and for higher arguments, the asymptotic unbiased inverse transform is used, see [25].

The procedure for Poisson noise removal is then given by the following three-step procedure: first, apply the forward Anscombe transform on the input image v, receiving an input image z=A(v) with a noise model similar to AGWN; second, apply the BM3D filter to this image z, receiving an estimate $\widehat{f}$ of E{A(v)|λ}; and lastly, apply the exact unbiased inverse Anscombe transform on $\widehat{f}$ to obtain an estimate $\widehat{\lambda}:={A}^{-1}\left(\widehat{f}\right)$ of λ.

Direct block matching for Poisson noise statistics

As pointed out by Salmon et al., the Anscombe transform is not accurate in the extreme low-count regime, i.e., if the average number of counts λ(x) is smaller than three for some x∈X (cf. [23]). STEM images with such a small number of counts are generally deemed useless, so they are rarely acquired or published. However, the ability to extract information from extreme low-dose images would be a substantial advantage for a variety of problems, such as characterization of metallic catalyst particles [3] and polymers and molecular crystals [31].

The performance of two variants of non-local means on images affected by Poisson noise are compared in [20]. The first one features the Anscombe transform and the standard L ²-patch similarity measure, while the second one uses a Poisson maximum-likelihood-based patch similarity measure from [27] without a preceding VST on the input data. The results show that when the PSNR of the input image is below 10 dB, the Poisson maximum-likelihood ratio-based patch distances outperform the strategy of using standard L ²-distances on the Anscombe transformed data.

The idea of using a Poisson maximum-likelihood-based similarity measure can be easily adopted to block matching when calculating the basic estimate via hard thresholding within the BM3D algorithm, see (3), (6), (8). To this end, we replace the L ²-distance (4) by the following distance:

$$ {d}_{\mathrm{P}}\left({Z}_x,{Z}_y\right):=\frac{1}{n^2}\sum_{i,j=1}^nf\left({Z}_x\left[\kern0.3em i,j\right],{Z}_y\left[\kern0.3em i,j\right]\right), $$

((13))

where f is defined through the Poisson maximum-likelihood ratio:

$$ \mathrm{p}\mathrm{m}\mathrm{l}\left({k}_1,{k}_2\right):=\frac{\underset{\lambda \in \left(0,\infty \right)}{ \max }p\left({k}_1\Big|\lambda \right)p\left({k}_2\Big|\lambda \right)}{\underset{\lambda \in \left(0,\infty \right)}{ \max }p\left({k}_1\Big|\lambda \right)\underset{\lambda \in \left(0,\infty \right)}{ \max }p\left({k}_2\Big|\lambda \right)} $$

((14))

as follows [27]:

$$ \begin{array}{cc}\kern-15.0pt f\left({k}_1,{k}_2\right)& :=- \log \left(\mathrm{p}\mathrm{m}\mathrm{l}\left({k}_1,{k}_2\right)\right)\\ {}={k}_1 \log {k}_1\kern0.3em +{k}_2 \log {k}_2\kern0.3em -\kern0.3em \left({k}_{1\kern0.3em }+{k}_2\right) \log \left(\frac{k_1+{k}_2}{2}\right).\end{array} $$

((15))

Using the distance (13), we then replace the set of matched blocks S _x (cf. (3)) with:

$$ {S}_x^P:=\left\{y\in {\mathcal{N}}_{N_S}(x):{d}_{\mathrm{P}}\left({Z}_x,{Z}_y\right)<- \log \left({\tau}^P\right)\right\}, $$

((16))

where τ ^P is a new threshold related to the maximum-likelihood ratios (14). Note that ${d}_P\left({Z}_x,{Z}_y\right)<- \log \left({\tau}^P\right)$ is equivalent to:

$$ \overline{\mathrm{pml}}\left({Z}_x,{Z}_y\right):={\left(\prod_{i,j=1}^n\mathrm{p}\mathrm{m}\mathrm{l}\left({Z}_x\left[i,j\right],{Z}_y\left[i,j\right]\right)\right)}^{1/{n}^2}>{\tau}^P, $$

((17))

i.e., τ _P is in fact a threshold on the geometric mean of the maximum-likelihood ratios. Now, since:

$$ \underset{\lambda \in \left(0,\infty \right)}{ \max }p\left({k}_1\Big|\lambda \right)p\left({k}_2\Big|\lambda \right)\le \underset{\lambda \in \left(0,\infty \right)}{ \max }p\left({k}_1\Big|\lambda \right)\underset{\lambda \in \left(0,\infty \right)}{ \max }p\left({k}_2\Big|\lambda \right) $$

((18))

we have that 0≤pml(k ₁,k ₂)≤1 for any ${k}_1,{k}_2\in \mathbb{N}$. From this, it follows that also $\overline{\mathrm{pml}}\left({Z}_x,{Z}_y\right)$, the geometric mean of the maximum-likelihood ratios between the blocks Z _x and Z _y takes values between 0 (very bad match) and 1 (perfect match). Thus, the corresponding threshold τ ^P should be chosen between 0 (no thresholding) and 1 (only accepting identical blocks). We found that the choice τ ^P=0.55 works well in practice and thus we use this value wherever the Poisson maximum-likelihood ratios are employed in this work. The reduced set ${\check{{S}_{x}^{P}}}$ and the block stack ${\mathcal{Z}}_x^P$ are defined analogously to ${\check{S}}_x$ and ${\mathcal{Z}}_x$, respectively, just replacing S _x by ${S}_x^P$.

Since the denoising in 3D transform domain is designed to remove AGWN, we still have to use the Anscombe transform before applying to the 3D block, i.e., the estimated 3D block stacks (6) are replaced by:

$$ \widehat{{\mathcal{Y}}_x^P}:={\mathcal{T}}^{-1}\left({Y}_{\theta}\left(\mathcal{T}\left(A\left({\mathcal{Z}}_x^P\right)\right)\right)\right). $$

((19))

In other words, when using this strategy, the Anscombe transformed data is used to fill the 3D block stacks with the values of the matched blocks but not to determine the positions of the matched blocks in the 3D stacks.

The weights ${w}_x^P={N}_{\theta}^{-1}$ are again equal to the number of non-zero transform coefficients after hard thresholding the Anscombe transformed block stack in the 3D transform domain. Accordingly, the basic estimate (8) is replaced by:

$$ {\widehat{f}}_{\mathrm{P}}^{\mathrm{basic}}(y):=\frac{\sum_{x\in {X}_R}\sum_{k=1}^{N_x}{w}_x^P{\hat{Y}}_{x,k}^P(y)}{\sum_{x\in {X}_R}\sum_{k=1}^{N_x}{w}_x^P{\chi}_{Z_{{\left(\widehat{S_x^P}\right)}_k}}(y)}. $$

((20))

Here, ${\hat{Y}}_{x,k}^P(y)$ is defined in analogy to (8). Also note that due to the application of the Anscombe transform to the 3D block stacks in (19), the values of the basic estimate (20) after Poisson maximum-likelihood-ratio-based block matching are still in the Anscombe transform domain. Thus, the Wiener filtering iteration remains unchanged.

Adaptive non-local periodic block matching

Images of crystals at atomic scale acquired by electron microscopy exhibit highly repetitious patterns of only a few basic objects corresponding to the different types of atoms. Therefore, such images are rich in self-similarity; however, blocks of equal structure are typically not found in local neighborhoods. Based on this observation, we developed a periodic search strategy for non-local means that is based on a Fourier analysis of the input image and distributes small non-local search windows periodically throughout the entire image [20]. In our previous work, we found that a significant increase in both visual image quality and of quantitative measures such as the PSNR can be achieved using this periodic search strategy. Nevertheless, we also found that BM3D, even with local search, in most cases still outperforms our proposed periodic search non-local means. Still, the local search used in the original BM3D filter is expected not to be well suited for most electron microscopy images of crystals. Hence, we now propose to combine our periodic search strategy for block matching with the BM3D filter for electron microscopy images. First, we recall the definition of the non-local periodic search grid [20]:

$$ \begin{array}{cc}\kern-10.0pt {\pi}_i^0(x)& :=x,\kern1em i=1,2,\\ {}\kern-10.0pt {\pi}_i^{\pm k}(x)& :=\left\{\underset{y\in {\mathcal{N}}_{\underset{S}{\overset{\pi }{N}}}\left(\underset{i}{\overset{\pm \left(k-1\right)}{\pi }}(x)\right)}{ \arg \min }{d}_{\mathrm{P}}\left({Z}_x,{Z}_y\right)\right\}\\ {}\pm \varDelta {x}_i\left(\begin{array}{c} \cos {\alpha}_i\\ {} \sin {\alpha}_i\end{array}\right),\kern1em i=1,2,\\ {}\kern-10.0pt {\pi}_{N_S^{\pi }}(x)& :=\left(\bigcup_{k_1,{k}_2}{\mathcal{N}}_{N_S^{\pi }}\left({\pi}_2^{k_2}\left({\pi}_1^{k_1}(x)\right)\right)\right)\bigcap X.\end{array} $$

((21))

Here, α ₁,α ₂ denotethe angles or directions of the periodic pattern within the input image and Δ x ₁,Δ x ₂ the spacings between the self-similar objects along those axes and ${N}_S^{\pi }$ is the size of the small local search windows (in units of pixels). For an automatic estimation strategy of these parameters from the input image, we refer to [20]. Note that the $\arg \min$ expression causes an adaptive reset of the search pattern that gives some robustness against errors in both the estimation of the grid parameters and slight variations of the periodic pattern within the image.

The search grid (21) is used to replace the local search window ${\mathcal{N}}_{N_S}(x)$ in the set of matched blocks (3) in the hard thresholding iteration and analogously in the Wiener filtering iteration with the sets:

$$ \begin{array}{cc}{\varPi}_x^{\mathrm{ht}}& :=\left\{y\in {\pi}_{N_S^{\pi }}(x):{d}_{\mathrm{P}}\left({Z}_x,{Z}_y\right)<{\tau}^{\mathrm{ht}}\right\},\\ {}{\varPi}_x^{\mathrm{wie}}& :=\left\{y\in {\pi}_{N_S^{\pi }}(x):{d}_{\mathrm{P}}\left({F}_x,{F}_y\right)<{\tau}^{\mathrm{wie}}\right\}.\end{array} $$

((22))

Here, the super scripts ht and wie refer to the first and second stage involving hard thresholding and Wiener filtering. In this paper, we have used small ${N}_S^{\pi}\times {N}_S^{\pi }$ local search windows of size ${N}_S^{\pi }=5$. We found that this choice retains reasonable computational efficiency, while being large enough (about 0.1 to 0.25 times as small as the atoms in the images we used) to correct the positions of the non-local periodic search steps within each atom. ${\check{\varPi}}_x^{\mathrm{ht}}$ and ${\check{\varPi}}_x^{\mathrm{wie}}$ are defined analogously to ${\check{S}}_x$. Note that for crystals at atomic scale the self-similarity within the corresponding image is so rich that with a non-local search like this, for nearly all x∈X _R , one finds many more than N _3D blocks that are similar to the reference at x, i.e., N _x =N _3D holds for nearly all x∈X _R .

In order to reduce the computational cost of the non-local periodic search, we made the following technical adjustments to its implementation in comparison with its description in [20] and the expression (21). First, for each reference point x among the two directions α ₁ and α ₂, the corresponding axis with the largest intersection with the image X is declared to be the primary search axis. Then, after performing one step along the primary search axis, non-local periodic search steps along both positive and negative directions of the secondary search axis are carried out only until the image boundary is reached. This process is repeated until the image boundary is also reached along the positive and negative direction of the primary search axis. On the one hand, this implementation requires less computational cost than computing the whole set of points within the image that could be reached by steps along either of the two periodicity axes, while on the other hand, it still gives a sufficiently large subset of all of these points. This efficient periodic block-matching strategy increases the computational cost by a factor of about 1.5 to 2 (depending on the density of the atoms within the image) compared to our implementation of BM3D with local block matching. Note that due to the overhead produced by unoptimized parts of our implementation this relative comparison may not be exact. Nevertheless, we believe that the factor between the run-times of local and periodic block matching would be of the same order of magnitude in optimized code as well.

Uniform distribution of block estimates

A key difference between the non-local means algorithm and BM3D is that the estimated intensity $\widehat{f}(y)$ in some pixel y∈X does not only depend on the similarity between the block Z _y (if $y\in \overset{\kern0.60em \circ }{X}$) and other blocks Z _x , $x\in \overset{\kern0.60em \circ }{X}$, in the image but also on the similarity between overlapping blocks ${Z}_{x^{\prime }}\ni y$, ${x}^{\prime}\in \overset{\kern0.60em \circ }{X}$, and other blocks Z _x . This is due to the mechanism in BM3D that aggregates the intensities of all pixels of all denoised blocks at their original positions in order to obtain the estimate (8) from the overcomplete representation given by the collection of all block estimates. As noted before, there can even be different block estimates corresponding to the same position, if they originate from denoised 3D block stacks matched to different references. Therefore, when applying the BM3D filter, the preciseness of the intensity estimate $\widehat{f}(y)$ may not only depend on the block distances of all references to their matched blocks but also on the distribution of the positions corresponding to all block estimates. For adaptive periodic block-matching, this is:

$$ {n}_{\mathrm{aggr}}^{\pi }(y)=\sum_{x\in {X}_R}\sum_{x^{\prime}\in {\check{\varPi}}_x}{\chi}_{Z_{x^{\prime }}}(y). $$

((23))

Analogously, we define n _aggr for local block matching but with ${\check{\varPi}}_x$ replaced by ${\check{S}}_x$. Figure 2 shows these quantities for both the hard thresholding and Wiener filtering iteration. The images B and D in Figure 2 reveal that the periodic block matching may lead to a significant concentration of block estimates in certain regions. Consequently, the number of block estimates has to be very low elsewhere. The effect is much weaker with the local block matching (cf. Figure 2A,C) since the local search windows naturally restrict blocks from one area to be matched to a small set of blocks within their neighborhood, which prevents the number of aggregates to concentrate significantly in one specific area of the image.

Figure 2

Number of aggregates for local and periodic block matching. Number of aggregates n _aggr, i.e., after local block matching (A) and (B) and ${n}_{\mathrm{aggr}}^{\pi }$, i.e., after periodic block matching (B) and (D) for hard thresholding (A) and (B) and Wiener filtering (C) and (D) applied to the gallium nitride lattice in Figure 3.

Figure 3

Simulated STEM images. Example frames from simulated series of electron microscopy images. Left column shows ground truths including STEM scan distortions but no shot noise. Crystals with average detected electrons per pixel are as follows: (A) perfect gallium nitride (0.66 to 3.47), (C) and (E) perfect silicon (0.28 to 1.94) and (0.37 to 6.68), (G) silicon with dislocation (1.4 to 9.7). Right column shows the same images with Poisson shot noise. The respective peak electron counts per pixel are as follows: (B) 12, (D) 9, (F) 21, and (H) 15.

Here, we propose a modification to the adaptive periodic block-matching strategy that aims at steering it towards a more uniform distribution of the positions of all block estimates as observed for the original local block-matching method. The task is to maximize the minimal number of aggregates while still choosing blocks with least distances during block matching. As we consider block similarity to be the more important than uniformity of the block estimates, we still primarily sort the matched blocks by block distance. However, we create additional degrees of freedom by choosing a slightly larger limit ${N}_{3\mathrm{D}}^{\ast }>{N}_{3\mathrm{D}}$ of the number of matched blocks than before. This gives larger sets ${\varPi}_x^{\ast }={\left({\varPi}_x\right)}_{i=1}^{N_x^{\ast }}$, where ${N}_x^{\ast }:= \min \left\{\#\left({\varPi}_x\right),{N}_{3\mathrm{D}}^{\ast}\right\}$. Reducing these sets for all reference points x∈X _R to the final size N _x while driving the distribution of the resulting block estimates throughout the image towards a more uniform one gives rise to the following optimization problem:

$$ {\left({\varPi}_x^{U^{\ast }}\right)}_{x\in {X}_R}=\underset{{\left({\varPi}_x^{\prime}\right)}_{x\in {X}_R}\in \mathcal{P}}{ \arg\ \max}\underset{y\in X}{ \min}\sum_{x\in {X}_R}\sum_{x^{\prime}\in {\varPi}_x^{\prime }}{\chi}_{Z_{x^{\prime }}}(y), $$

((24))

where $\mathcal{P}:=\left\{{\left({\varPi}_x^{\prime}\right)}_{x\in {X}_R}:{\varPi}_x^{\prime}\subset {\varPi}_x^{\ast },\#\left({\varPi}_x^{\prime}\right)={N}_x,\kern0.60em \forall x\in {X}_R\right\}$ denotes the feasible set. Unfortunately, this optimization problem is of combinatorial type and due to its global coupling of all reference coordinates would be computationally too costly to solve. Hence, we propose the following simplification of the uniform block matching through an iterative procedure.

Here, the reference coordinates have been numbered ${X}_R=\left({x}^1,\dots, {x}^{\#\left({X}_R\right)}\right)$ and block matching is performed for one after the other in an iterative fashion. In this paper, we use the number of local ${N}_S^{\pi}\times {N}_S^{\pi }$ search windows as the limit ${N}_{3\mathrm{D}}^{\ast }$. Note that as the number of search window is dependent on the reference position x∈X _R , so is the new limit ${N}_{3\mathrm{D}}^{\ast }={N}_{3\mathrm{D}}^{\ast }(x)$, unlike the limit N _3D chosen before, which is independent of x. This choice can be motivated as follows: if the adaptive periodic search succeeds in placing the local search windows roughly according to the pattern of the observed crystal, then for a reference pixel within an atom, each local search window within the corresponding similarity search should overlap a similar part of another atom. Therefore, we expect to find at least one well-matching block per local search window. Note that we define ${n}_{\mathrm{aggr}}^U$ in analogy to (23) but with ${\check{\varPi}}_x$ replaced by ${\check{\varPi}}_x^U$.

Results and discussion

We have presented several modifications of the original BM3D filter as it was proposed by Dabov et al. [24], including the extension to Poisson noise removal due to Mäkitalo and Foi [26]. The main intention was to improve the filter within the context of electron microscopy images of crystals at atomic scale. To ensure a clear understanding of which modifications or settings were used to produce the results presented in this section, we use the following prefixes:

• np- ⋆: normal profile and Anscombe-based three-step procedure

• pml- ⋆: Poisson maximum-likelihood ratio-based block matching within hard thresholding iteration

• ⋆-l ⋆: local block matching with N _S =39

• ⋆- π⋆: adaptive periodic block matching with ${N}_S^{\pi }=5$

  • ⋆- π N⋆: selection of N _3D best-matched blocks with thresholds N _3D=16 (HT) and N _3D=32 (Wiener)

  • ⋆- π U⋆: uniformly distributed selection of N _3D blocks from ${\widehat{N}}_{3\mathrm{D}}$ best-matched blocks as in Algorithm 1

• ⋆- ⋆n: block size is n×n

A commonly used quantitative measure to compare the image quality after application of different image processing algorithms is the PSNR of the corresponding estimates:

$$ \mathrm{PSNR}\left(\lambda, \widehat{\lambda}\right)=10\underset{10}{ \log}\left(\frac{\underset{x\in X}{ \max }{\left(\lambda (x)\right)}^2}{\frac{1}{\left|X\right|}\sum_{x\in X}{\left(\lambda (x)-\widehat{\lambda}(x)\right)}^2}\right). $$

((25))

This quantitative measure requires the knowledge of the underlying ground truth, i.e., the exact average counts λ(x) in each pixel x∈X, which is not available in experimental STEM images. Therefore, we have simulated STEM images of various materials using the frozen phonon multislice algorithm [32] to obtain representative ground truth micrographs that are free of Poisson noise (cf. Figure 2A,C,E,G). To make these images representative of experimental images, typical image distortions that are caused by instabilities of the sample and electron beam during experimental STEM image acquisition have been artificially introduced to these simulated images. Known material crystallographic data was used to create the gallium nitride and silicon atomic models. Molecular dynamics was used to calculate the silicon dislocation atomic model [8]. In order to compare the algorithms at different noise levels, we scaled the intensities of the ground truth images to different average electron counts per pixel before applying random Poisson noise. This simulates the usage of different beam currents, resulting in different electron doses. Examples of Poisson noise affected versions of the simulated images can be found in Figure 3B,D,F,H.

In this work, we use our implementation of BM3D with local search as the ‘original’ BM3D benchmark. This comparison is justified, since we verified our implementation against the implementation provided by the original authors [29] and found the results to be consistent both in terms of the resulting peak signal-to-noise ratios and a visual comparison of the retrieved estimates. Note that our implementation even gives slightly better PSNRs on most images, since it uses a full local search for each reference coordinate instead of reducing the size of the local search window based on previous block-matching results, as it was suggested in [24]. However, due to both this fact and the lack of code optimization, our implementation is currently also slower by a factor of between 5 and 10, depending on the input.

At http://nmevenkamp.github.io/ELMA/, we provide a current version of our proposed method as C++ source code, as well as Windows and MacOSX applications with graphical user interface.

Adaptive non-local periodic block matching

A major goal of this work is to show that local block matching, with a search window small enough to warrant practical efficiency, is not able to properly benefit from globally recurring self-similar features. Instead, in such cases, real non-local block-matching strategies are able to exploit the capabilities of the BM3D filter to a significantly higher extent.

We compare the PSNRs of the BM3D estimates for local block matching to those attained with our proposed non-local adaptive periodic block matching in Table 1 for the block sizes n=8 and n=16. The results clearly show that our proposed non-local periodic block matching leads to significantly better estimates (up to 4 dB gain) than the local block matching for all input images, even in the case of the silicon lattices with a defect, where the atoms are not arranged in a perfectly periodic pattern. This shows that the adaptive reset within the small local search windows defined in (21) yields sufficient robustness to cope with slight deviations from a perfect periodic structure of the input image. Figure 4 compares the resulting visual image quality when using local (left column) or periodic (right column) block matching within the BM3D filter. Comparing with the ground truths in Figure 3, one sees that both the local and the periodic block-matching methods smooth out the scan distortions. This effect is due to the non-local averaging of blocks extracted from essentially identical atoms that just differ slightly in shape due these distortions. Since the local distortion of a scan line with respect to the atom center is mostly random, these distortions cancel out on average. In Figure 4A,B,C,D,E,F, one sees that for the cases of perfect gallium nitride and silicon lattices at very low PSNR, the adaptive periodic block matching leads to a significantly better estimate than the local block matching. The example shown in Figure 4G,H, where a silicon lattice with a dislocation was used, shows visually that even in the regions where the silicon atom pairs are slightly tilted, and where therefore self-similarity does not coincide with the global periodic pattern of the crystal, our proposed adaptive periodic block matching is robust enough to still find well-matching blocks. The denoised images of the silicon lattice with a dislocation indicate that non-local averaging does not noticeably affect the orientation of the silicon atoms around the dislocation, despite the fact that most silicon dumbbells within the image have a vertical orientation.

Figure 4

Comparison between local and adaptive periodic block matching. Denoised versions of the Poisson noise affected images from Figure 3 after application of local block-matching BM3D (A), (C), (E), and (G) and non-local adaptive periodic block-matching BM3D (B), (D), (F), and (H). The block size is N ₁=16 for both methods.

Table 1 PSNRs of local vs. adaptive periodic block matching

Comparing the two middle columns with the two right columns in Table 1, one clearly sees that using the larger block size n=16 consistently yields significantly better results than using n=8. Note that there is one exception: the silicon lattice with dislocation at lowest PSNR (2.7 dB). However, the denoised images shown in Figure 5 confirm the importance of a larger block size for robust block matching, especially within the low electron count regime, as the estimates for n=16 (cf. Figure 5E,F) show an image quality far superior to the ones for n=8 (cf. Figure 5C,D). Again, regarding the case of the silicon lattice with dislocation (left column in Figure 5), one sees that the shapes of the atoms are still estimated more correctly using the larger block size n=16 (cf. Figure 5E). Notice however, that there appears to be a blurring of whole sub-regions within the estimate for n=16, which is not present for n=8 (cf. Figure 5C). This effect will be discussed in the next section.

Figure 5

Block matching with different block sizes and (non-) uniform distribution of block estimates. (A) Noisy silicon lattice with dislocation (peak 6, PSNR: 2.70 dB), (B) noisy perfect silicon lattice (peak: 6, PSNR: 3.62 dB), (C) and (D) corresponding estimates from adaptive periodic block-matching BM3D using N ₁=8, (E) and (F) using N ₁=16, and (G) and (H) using N ₁=16 and uniform distribution of block estimates.

Uniform distribution of block estimates

As described earlier, the adaptive periodic block matching, due to its true non-local similarity search, may lead to local concentrations of the number of available block estimates, thereby leaving fewer block estimates for other regions of the image. This effect can lead to an overall poor image quality in the image regions with few block estimates, as was observed in the case of the silicon lattice with dislocation in Figure 5E. In comparison to this, we see in Figure 5G,H the estimates retrieved from the uniform block-matching strategy described earlier. Figure 5G shows an example of the improvement obtained by steering the distribution of the number of available block estimates towards global uniformity. Exemplary distributions of the number of aggregates after uniform periodic block matching are shown in Figure 6 for the gallium nitride lattice in Figure 3B. In Table 2, we compare our proposed plain adaptive periodic block matching versus the one with additional uniform distribution of the number of available estimates per pixel. First of all, the previous anomalous result for the silicon lattice with dislocation at lowest PSNR (2.7 dB), in which a block size of n=8 outperformed the larger block size of n=16 using plain adaptive periodic block matching, is resolved. Furthermore, the results in Table 2 show that for both the block sizes n=8 and n=16 uniform block matching increases the quality of the estimate in the majority of the cases, compared to plain periodic block matching without this addition, except for the silicon dislocation images with n=16. Our proposed threshold for the number of blocks that are regarded equally well matched (the number of non-local search windows) is not well suited for the rotated atoms near the dislocation. Therefore, for such images, reduced reliability is expected from the uniform block-matching approach.

Figure 6

Number of aggregates for uniform periodic block matching. Number of aggregates ${n}_{\mathrm{aggr}}^U$ after uniform adaptive periodic block matching for (A) hard thresholding and (B) Wiener filtering applied to the gallium nitride lattice in Figure 3.

Table 2 PSNRs of periodic vs. uniform periodic block matching

Direct block matching for Poisson noise statistics

Table 3 shows a comparison of (a) the PSNR of estimates retrieved by application of the BM3D filter in a three-step procedure using the Anscombe transform with (b) replacing the L ²-distances in the Anscombe transformed data within the block matching of the hard thresholding iteration by the Poisson maximum-likelihood-based distances within the original data. The results indicate that at low PSNR of the input image, employing Poisson maximum-likelihood ratios is slightly better than using the L ²-distance within the Anscombe transformed data. For intermediate PSNR however, there seems to be very small or no difference between the two distance measures.

Table 3 PSNRs of Anscombe and L ² distance based block similarity vs. Poisson maximum-likelihood ratios

Although the improvement so far is small, we have just modified a small part of the BM3D algorithm to work directly on the original Poisson statistics, namely the block-matching part within the initial hard thresholding iteration. The results we presented in [20] show a more significant advantage of the Poisson maximum-likelihood ratio-based similarity measure over the Anscombe transformed L ²-distances. However, this was for the NLM algorithm, where in the case of the Poisson maximum-likelihood ratio-based similarity measure, no Anscombe transform of the input data was required at all. Therefore, we believe that future research towards adopting the whole BM3D algorithm directly to Poisson noise statistics may lead to more substantial improvements of the reconstruction in the extreme low-count regime.

Atomic column detectability, position precision, and reconstruction fidelity

As mentioned in the beginning, the positions of the atoms are an important quantity that material scientists would like to extract from atomic-resolution electron micrographs. In the following, we analyze how well the positions can be estimated on the noisy images referred to in the previous sections and by how much our proposed denoising algorithm improves this estimation.

In order to extract the atom positions, we adopt a two-step procedure: first, the individual atoms within the image are detected through segmentation and the geometrical centers of the resulting atomic regions are used as an initial guess for the atom centers; second, a 2D Gaussian function (or a sum of two 2D Gaussian functions in the case of the silicon lattices) is fit on a small area around each atomic region via non-linear regression in order to determine the final estimate of the position of each atom. For further details regarding this procedure, we refer to [2,33].

The detectability of the atomic columns in the noisy and denoised images can be assessed by the detection fraction. It is defined as follows. Let c _i =c _i (λ)∈[ 1,N]×[ 1,M], $i=1,\dots, {N}_{\lambda }$ denote the centers of the atoms as detected from the ground truth λ and ${\hat{c}}_j={\hat{c}}_j\left(\widehat{\lambda}\right)\in \left[\kern0.3em 1,N\right]\times \left[\kern0.3em 1,M\right]$, $j=1,\dots, {N}_{\widehat{\lambda}}$ the ones detected from the corresponding noisy (or denoised) image $\widehat{\lambda}$, using the same atom-finding procedure as described above. Furthermore, let $\varDelta {c}_{\min }:=\underset{i,j=1,\dots, {N}_{\lambda }}{ \min}\left|{c}_i-{c}_j\right|$ denote the shortest inter-atomic distance within the centers detected from the ground truth. Then, the detection fraction is defined as the fraction of atoms detected in the ground truth, which could be matched to a corresponding atom detected in the noisy (or denoised) image:

$$ \begin{array}{c}\kern-15.0pt {\rho}_{+}\left(\lambda, \widehat{\lambda}\right):=\frac{\#\left(\left\{i\in \left\{1,\dots, {N}_{\lambda}\right\}\kern0.3em :\kern0.3em \underset{j=1,\dots, {N}_{\widehat{\lambda}}}{ \min}\left|{c}_i-{\hat{c}}_j\right|\kern0.3em <\kern0.3em \frac{1}{2}\varDelta {c}_{\min}\right\}\right)}{N_{\lambda }}.\end{array} $$

((26))

Here, the $\min$ expression selects the atom detected in the noisy (or denoised) image, which is nearest to the one detected in the ground truth and the comparison with Δ c _min ensures that atoms are only matched if their distance is less than half the minimum inter-atomic distance. The latter is required for an unambiguous correspondence. Table 4 gives the detection fractions for noisy and denoised versions of low dose examples of perfect gallium nitride and silicon lattice images, as well as images of silicon lattices with a dislocation, with peak electron counts in the range of 6 to 49 electrons per pixel, corresponding to incident electron doses in the range of 0.55 to 28.47 C/cm ². These are low electron doses for inorganic materials, which is the main application intended for the algorithms presented in this paper. For organic and biological samples, however, a low electron dose is on the order of 0.01 C/cm ². The detection fraction is very small for the lower dose noisy images (0.55 to 5.68C/cm ²), as well as for the local block-matching BM3D estimate of the lowest dose gallium nitride (1.54 C/cm ²) and silicon dislocation (00.55 C/cm ²) lattices. For the other images (especially the silicon lattice), the detection fraction is significantly increased by denoising the image (even with local block matching). Furthermore, the periodic search BM3D method achieves detection fractions near or equal to 100% in all cases, except the lowest dose silicon dislocation lattice. In terms of the detection fraction, the uniform periodic block matching is superior to plain periodic block matching. While the difference is slight in most cases (up to 1%), the detection fraction is significantly increased by uniform block matching on the two lowest dose images, namely from 94.2% to 98% on the perfect silicon lattice (0.79 C/cm ²) and from 48.4% to 92.5% on the silicon lattice with dislocation (0.55 C/cm ²). Note that the detection fraction alone is not sufficient to assess the quality of the atomic column detection, as can be seen from the following example: say that due to the noise or poor quality of the image, the segmentation falsely detects many more atoms within the noisy (denoised) image than in the ground truth and that some of them are, by coincidence, near some of the atoms within the ground truth. In this case, the detection fraction could still be large. In order to cover this issue, we additionally give corresponding misdetection fractions in Table 5:

$$ \begin{array}{c}\kern-16.0pt {\rho}_{-}\left(\lambda, \widehat{\lambda}\right)\\ {}:=\frac{N_{\widehat{\lambda}}\kern0.3em -\kern0.3em \#\left(\left\{i\in \left\{1,\dots, {N}_{\lambda}\right\}:\underset{j=1,\dots, {N}_{\widehat{\lambda}}}{ \min}\left|{c}_i\kern0.3em -\kern0.3em {\hat{c}}_j\right|\kern0.3em <\kern0.3em \frac{1}{2}\varDelta {c}_{\min}\right\}\right)}{N_{\widehat{\lambda}}}\end{array} $$

((27))

Table 4 Detection fraction ( 26 ) of the noisy (denoised) images

Table 5 Misdetection fraction ( 27 ) of the noisy (denoised) images

These specify the fraction of atoms detected from the noisy (denoised) image that could not be matched to any of the atoms detected from the ground truth. As seen in Table 5, these fractions are zero, except for the noisiest input images (6 to 15 peak electrons/pixel) and the local block-matching BM3D estimate of the silicon images with doses 0.79 and 7.85 C/cm ², as well as the lowest dose silicon dislocation image (0.55 C/cm ²). Note that for the periodic block-matching BM3D the misdetection fractions are zero in all cases, except for the lowest dose silicon dislocation image. While the local block-matching BM3D has a misdetection fraction of 100% in this case, the periodic and uniform periodic block-matching BM3D methods achieve misdetection fractions of 12% and 4%, respectively.

A known quality measure for the atomic column position estimation is the so-called precision. It is defined as the standard deviation of the inter-atomic distances and ideally should be as small as possible for perfect single crystal samples because all the inter-atomic distances should be identical. In STEM, the precision is often regarded separately parallel and perpendicular to the scan (which is horizontal and vertical for all of the images here). However, we just intend to show that denoising improves the quality of the atomic column detection. Hence, we simply regard the total precision:

$$ \begin{array}{c}\kern-17.0pt \mathrm{Precision}\left(\widehat{\lambda}\right)\kern0.3em =\kern0.3em \sqrt{\frac{1}{N_{\varDelta x}}\sum\limits_{i=1}^{N_{\varDelta x}}{\left(\varDelta {x}_i\kern0.3em -\kern0.3em \bar{\varDelta }x\right)}^2+\frac{1}{N_{\varDelta y}}\sum\limits_{j=1}^{N_{\varDelta y}}{\left(\varDelta {y}_j-\bar{\varDelta }y\right)}^2},\end{array} $$

((28))

where N _{Δ x} ,N _{Δ y} are the number of, respectively, horizontal and vertical inter-atomic distances between neighboring atoms detected in the image $\widehat{\lambda}$, Δ x _i ,Δ y _j , $i=1,\dots, {N}_{\varDelta x}$, $j=1,\dots, {N}_{\varDelta y}$ are the corresponding distances and $\bar{\varDelta }x,\bar{\varDelta }y$ are their mean values, respectively. In Table 6, the precision, in picometers, is given for all images used in the atomic column detectability analysis above, except for the silicon dislocation images, where this measure is not meaningful due to the aperiodicity of the true atom positions. For images, where not a single pair of neighboring atoms could be detected, the precision is denoted by ‘n/a.’ Table 6 shows that this is the case for the noisy images with peak electron counts in the range of 6 to 12, as well as the local block-matching BM3D estimate for the lowest dose gallium nitride lattice (1.54 C/cm ²), which is in line with the small corresponding detection fractions seen in Table 4. For all other images, the according detection fractions in Table 4 are above 90% and the misdetection fractions in Table 5 are below 2%, which means that the precision is of relevance in these cases. Revisiting Table 6, we see that, except for the moderate-dose gallium nitride image (3.85 C/cm ²), where the precision is already quite high after local block matching, our proposed (uniform) periodic block-matching BM3D is able to consistently enhance the precision by a factor of two, compared to local block-matching BM3D. Even for the lowest dose input (0.79 C/cm ²), we observe a precision below 15 pm, which is the best precision reported for single shot STEM images before [6]. However, our high precision is obtained in a simulated image, and to be completely relevant, it must be confirmed in experimental images. In the case of the gallium nitride lattice (3.85 C/cm ²), where individual atoms could not reliably be detected in the noisy input, the uniform adaptive periodic BM3D estimate achieves a precision of below 7.5 pm, improving the precision by a factor of two compared to those reported in [6].

Table 6 Precision ( 28 ), in picometers, of the noisy and denoised images

We conclude the ‘Results and discussion’ section with the analysis of another quality measure for the atom detection, which we call fidelity. We define this measure as the root mean square of the distances between corresponding atoms, as detected in both the ground truth and the noisy (denoised image):

$$ \begin{array}{c}\kern-15.0pt \mathrm{Fidelity}\left(\lambda, \widehat{\lambda}\right):=\sqrt{\frac{N_{\lambda }}{\rho_{+}\left(\lambda, \widehat{\lambda}\right)}\sum_{i=1}^{N_{\lambda }}{Y}_{\frac{1}{2}\varDelta {c}_{\min }}{\left(\underset{j=1,\dots, {N}_{\widehat{\lambda}}}{ \min}\left|{c}_i-{\hat{c}}_j\right|\right)}^2}.\end{array} $$

((29))

As in the definition of the detection fraction (26), here the $\min$ expression selects the nearest atom and the thresholding operator ${Y}_{\frac{1}{2}\varDelta {c}_{\min }}$ ensures that atoms are only matched if their distance is less than half the minimum inter-atomic distance. Note that unlike accuracy, where the exact atom positions used to simulate the ground truth would replace the detected ground truth centers c _i , the fidelity does not measure the error in the actual positions of the atomic columns. Instead, it measures how much the atomic column detection is influenced by the addition of shot noise and, more importantly, by the reconstruction. Looking at the corresponding values given in Table 7 for the examples discussed before, we see the same effect that for the very noisy input images (6 to 12 peak electrons/pixel) the fidelity is very low. Again, the local block-matching BM3D estimates show a significantly increased fidelity compared to the noisy input images. Furthermore, except for the moderate dose gallium nitride (3.85 C/cm ²) and silicon dislocation (2.8 to 28.47 C/cm ²) images, the fidelity is also enhanced by a factor of two by using our proposed (uniform) periodic block matching instead of the local block-matching strategy. For the single shot STEM images with an incident electron dose above 1.57 C/cm ², we achieve sub-picometer fidelity by our proposed BM3D filter with uniform adaptive periodic block matching. Note that this sub-picometer fidelity is also observed on the aperiodic silicon lattices with a dislocation, indicating that the adaptiveness of the proposed periodic search strategy is able to cope with localized irregularities in the crystal structure. Overall, this result shows that the denoising with our proposed method does not introduce artifacts exceeding the sub-picometer level. Again, it should not be taken as evidence of sub-picometer precision or accuracy in locating the atom columns in the reconstructed images, since it does not account for the STEM instabilities built in to the ground truth images. Removing instabilities still requires an approach like non-rigid registration of a series of images [2].

Table 7 Fidelity ( 29 ), in picometers, between ground truths and noisy (denoised) images

A comparison with linear filters

In practice, simple linear filters, such as the median or low-pass Wiener filter, are still commonly used for denoising in electron microscopy. In the following, we show that such filters provide a substantially worse denoising performance than the proposed modified BM3D method. In particular, we demonstrate that the application of such linear filters does not enable a proper analysis of extremely noisy images (less than 10 peak electron counts per pixel). We used the MATLAB functions medfilt2 for median filtering and wiener2 for low-pass Wiener filtering to produce the results presented here. In both cases, default parameters were used, which results in a windows size of 3×3 pixels and, in case of the Wiener filter, the noise power being automatically estimated.

Figure 7 shows denoised version of the gallium nitride and perfect silicon lattices from Figure 3 after application of a median filter (left column) and a low-pass Wiener filter (right column). A comparison with the corresponding estimates provided by our proposed method shown in Figure 5B,D reveals that the visual quality after denoising is much better when the modified BM3D method is used instead of either the median or low-pass Wiener filter.

Figure 7

Median and Wiener filter applied to noisy images. Gallium nitride image (cf. Figure 3B) denoised with (A) median filter and (B) low-pass Wiener filter, and silicon image (cf. Figure 3D) denoised with (C) median filter and (D) low-pass Wiener filter.

A quantitative analysis based on the measures described above is shown in Table 8. The best results among the linear filters were achieved in the case of the noisy gallium nitride lattices processed with a median filter. Here, all atoms were correctly detected, i.e., the detection fraction is one and the misdetection fraction is zero. However, note that the recorded precisions in these two cases are worse by a factor of about two compared to those achieved with the proposed modified BM3D. In the other cases, i.e., where the median filter was applied to the silicon lattices and in all cases where the low-pass Wiener filter was used, the results indicate that atom detection is not feasible in the denoised images at all, which is either due to the detection fraction ρ ₊ being significantly less than 100% or the misdetection fraction ρ ₋ being significantly larger than 0% or both. The proposed BM3D filter achieved detection fractions of 100% on all noisy perfect crystal images except for the lowest SNR silicon lattice and misdetection fractions of 0% in all periodic crystal cases. In the case where the detection fraction after application of the modified BM3D filter is just 98%, the median filter yields a similar result with 96%. However, the misdetection fraction after application of the median filter is 81%. The low-pass Wiener filter yields worse results, with a detection fraction of 4% and a misdetection fraction of 50%. Overall, the results indicate that the proposed modified BM3D method is far superior to simple linear filters such as the median or Wiener filter in terms of both atom detectability and precision of atom positions.

Table 8 Performance of median and Wiener filters

A remark on further quantitative measures

We are aware that other quantities like the intensity and shape of atomic columns are also important for materials science. In future work, we plan to conduct a survey in this direction, especially treating the question whether the non-local averaging procedure will be able to retain different intensities in atomic columns of the same type, thus enabling the determination of the number of atoms in each column within the denoised image.

Conclusions

We proposed key modifications of the block-matching and 3D-filtering algorithm, which were aimed at enhancing the filter when applied to atomic-resolution electron micrographs of periodic crystals. We have shown that, through the proposed modifications, the denoising performance is significantly improved compared to the original BM3D on all tested images. It also substantially outperforms common linear filters such as median-filtering and low-pass Wiener filtering. The major advances are the adoption of a Fourier-based periodic similarity search [20] within the non-local means setting to the BM3D algorithm, as well as the treatment of an issue regarding spatial block concentration, which only occurs in the new BM3D setting. Furthermore, we showed that the proposed filter with its uniform adaptive periodic block matching, specifically tailored to perfect crystal structures, is able to significantly enhance both visually and quantitatively the image quality of low-dose electron micrographs. Quantitative measures of interest to the material science community, namely atomic column detectability and position precision, are significantly improved by application of the new denoising algorithm, without the introduction of artifacts such as false-positive identification of atomic columns or shifts in the atomic column image positions beyond the sub-picometer level.

The proposed algorithm for steering the spatial distribution of block estimates towards global uniformity achieves a significant improvement over the periodic block-matching BM3D for certain images. However, we also observed cases where the results are slightly worse compared to the periodic block matching without this addition. Note that our uniform adaptive periodic block-matching BM3D is still significantly better than the original BM3D in all cases. Nevertheless, we plan to further investigate this phenomenon. The goal is to find a strategy that will at least sustain the quality of the estimate compared to plain periodic block matching while improving it in the majority of the cases.

According to the results we presented for the case of a silicon dislocation, the adaptiveness of the periodic block matching copes fairly well with localized irregularities in the crystal structure. Nevertheless, the proposed periodic block matching is generally limited to the assumption of a perfectly periodic crystal, which is usually of less interest to material scientists than crystals with (possibly multiple) defects or changes in the lattice orientation. Thus, we plan to improve and properly extend the block-matching strategy to these more complex geometries.

While the presented methods can be easily adopted to a series of images and thus might directly enable the registration of a series of low-dose electron micrographs, we expect a direct coupling of denoising and registration to be superior. Developing such a combined algorithm and analyzing its performance on series of low dose frames will be a main goal of our future work.