3D reconstruction of biological structures: automated procedures for alignment and reconstruction of multiple tilt series in electron tomography

Abstract

Transmission electron microscopy allows the collection of multiple views of specimens and their computerized three-dimensional reconstruction and analysis with electron tomography. Here we describe development of methods for automated multi-tilt data acquisition, tilt-series processing, and alignment which allow assembly of electron tomographic data from a greater number of tilt series, yielding enhanced data quality and increasing contrast associated with weakly stained structures. This scheme facilitates visualization of nanometer scale details of fine structure in volumes taken from plastic-embedded samples of biological specimens in all dimensions. As heavy metal-contrasted plastic-embedded samples are less sensitive to the overall dose rather than the electron dose rate, an optimal resampling of the reconstruction space can be achieved by accumulating lower dose electron micrographs of the same area over a wider range of specimen orientations. The computerized multiple tilt series collection scheme is implemented together with automated advanced procedures making collection, image alignment, and processing of multi-tilt tomography data a seamless process. We demonstrate high-quality reconstructions from samples of well-described biological structures. These include the giant Mimivirus and clathrin-coated vesicles, imaged in situ in their normal intracellular contexts. Examples are provided from samples of cultured cells prepared by high-pressure freezing and freeze-substitution as well as by chemical fixation before epoxy resin embedding.

Appendices

Appendix 1: Automatic alignment

The averaging method proposed in this work relies on automating the EMT reconstruction process for any number of tilt series. This appendix describes in detail our procedure steps: (i) the marker detection, (ii) the single tilt alignment, (iii) the multiple tilt alignment. The back-projection routine in the case of the wSIRT approach is explained in the next appendix. The general alignment flow is summarized from Figs. 10–13.

Fig. 10

Demonstration of the gold marker detection scheme outlined in "Appendix: Marker detection." This filtering procedure makes use of gaussian derivatives to calculate the second-order derivatives and allows to selectively enhance markers of a specific size (see recipe [21]). A A projection micrograph of the clathrin specimen at zero tilt is displayed. B The corresponding detector image is shown. The biological background behind the 5-nm gold markers is largely attenuated after applying this filter.

Fig. 11

Flow chart describing the automation process for a single tilt series reconstruction. After detecting the marker projection peaks throughout the micrographs, 3D marker positions $\{\mathbf{X^\tau }\}_\tau$ are inferred using a cluster analysis; this analysis is carried out on a set of 3D points located at the cross-over nodes of back-projected rays (Fig. 12). A trace ($\equiv$ set of matching peaks in a micrograph series) is then estimated for each marker, and the projection maps refined with a bundle adjustment procedure. This process is repeated iteratively to maximize the average trace size $\langle l_\tau \rangle _\tau$

Marker detection

The precise alignment of a tomographic series typically depends on fiducial markers; it is achieved by spreading gold particles on the specimen surfaces. By tracking the marker projections throughout a series of micrographs, the specimen deformation can be closely monitored and accounted for in the final alignment. This enables accurate electron micrograph registration, and ultimately leads to high-quality reconstructions.

The first step towards fully automated EMT consists in detecting the gold particles for each electron micrograph. We follow the procedure outlined in reference [21]. Since gold particles on a micrograph correspond to small black disks, the method consists in finding local maximum in the second-order derivatives (or equivalent) of the image gray levels. Following [21], this detection scheme is optimized to be selective for a certain length scale $\sigma$ related to the marker size. Hence, the gaussian derivative concept is introduced. Second-order gaussian derivatives with a scale level $\sigma$ are simply obtained by convoluting images with three kernels $G^{\sigma }_{xx},$ $G^{\sigma }_{xy}$ and $G^{\sigma }_{yy}$:

$${I^{\sigma}}_{ij}(x,y) = I(x,y) * G^{\sigma}_{ij}(x,y)$$

(2)

with i, j in $\{x,y\};$ x and y denote for the two coordinates of an image pixel.

$$\begin{aligned} \begin{array}{l} G^{\sigma }_{xx}(x,y) = \frac{x^2-\sigma ^2}{2 \pi \sigma ^4} \exp { \left( - \frac{x^2+y^2}{2 \sigma ^2} \right), } \\ G^{\sigma }_{yy}(x,y) = \frac{y^2-\sigma ^2}{2 \pi \sigma ^4} \exp { \left( - \frac{x^2+y^2}{2 \sigma ^2} \right) ,} \\ G^{\sigma }_{xy}(x,y) = \frac{x y}{2 \pi \sigma ^4} \exp { \left( - \frac{x^2+y^2}{2 \sigma ^2} \right). } \\ \end{array} \end{aligned}$$

(3)

The detector image is then built from the eigenvalues $\lambda _\text{min}$ and $\lambda _\text{max}$ of the 2 by 2 image matrix $\left\{ I_{ij} \right\}$. In our case, we choose to detect particles from the following quantity $D=\left| \lambda _\text{min} \lambda _\text{max} \right|$.

Peaks in image D are considered positive if their normalized correlation coefficient is above a certain threshold. The threshold value can be readjusted afterwards (in a second pass) so the number of selected peaks, i.e., the number of gold markers, remains consistent throughout the tilt series.

Iterative alignment for single axis tomography

Immediately after the detection, the set of peaks is still an unclassified collection of points. It is not possible to distinguish any trace corresponding to a specific marker within the micrograph series.

This is resolved using a procedure based upon a cluster analysis to identify possible gold particles 3D locations and their associated projections. Precisely, when back-projecting rays arising from detected peaks, markers should be located on the convergence nodes (Fig. 12A). Once a marker position is assessed, its projection trace can be rebuilt by picking the most probable matching peak on each micrograph—this is the closest peak to a micrograph projection point within a given distance.

Fig. 12

The 3D seeding procedure. A The gold markers are inferred as the main convergence nodes of back-projected rays arising from the detected projection peaks. B In practice, a cluster analysis is performed on the mid-points of the shortest segment joining selected pair of back-projected rays. Note that the two paths $\omega _1$ and $\omega _2$ do not necessarily cross in 3D

Fig. 13

Flow chart summarizing the automation strategy for multiple tilt series. Each series i is first reconstructed independently as outlined in Fig. 11, generating multiple sets of gold marker 3D positions $\{ \mathbf{X_i^{\tau _i}} \}_{\tau _i}$. Their correspondences are established through an iterative process, allowing to identify the matching traces between tilt series. A global alignment is then implemented

The overall automation scheme is outlined in Fig. 11. It is an iterative procedure that repeatedly builds projection traces from cluster seeds and refines the micrograph alignment until the average peak number in the traces becomes maximal. The initial state in this iterative routine corresponds to a rough alignment where micrographs are aligned by means of simple cross-correlations. It has consistently shown to lead to the correct final alignment.

In practice, the cluster analysis is carried out on a set of 3D points generated from pair of back-projected rays; for each pair of rays, the mid-point of the shortest segment joining them is retained; this would correspond for instance to mid-point H of segment $H_1 H_2$ in Fig. 12B. Narrow regions exhibiting a high point density are mapped to clusters; their centroid is taken as the 3D position of a marker and used as an input to determine the projection traces. Throughout this computation, the projection maps $P_{\omega }$ that associates a 3D point $\mathbf{X}$ to a 2D pixel $\mathbf{x}$ in micrograph $\omega$ are taken affine, i.e., rays are considered as straight lines. Also, to ensure the 3D seeding procedure is efficient, we do not consider all the possible pairs of rays, but only the ones separated by a fixed angle increment—for instance 20°. By doing so, aligning tilt series containing thousands of markers can be done in a matter of minutes on a regular workstation. To implement the cluster analysis, we used the scipy.cluster package [36].

After the seeding procedure has been initiated and the marker traces been identified, the alignment is finally optimized through a bundle adjustment procedure by minimizing a reprojection error. This allows to refine the projection maps $P_{\omega }$ ; this technique has been extensively discussed in the context of electron tomography [16–18].

When markers are numerous (for instance, more than $n_\text{c}=100$), the procedure outlined in Fig. 11 is implemented twice. At first, we restrict the number of markers to be less than a certain threshold (${<}n_\text{c}$) but still evenly distributed; this allows to quickly estimate the projection maps $P_{\omega }$. In a second pass, all the particles are considered: since the initial value of $P_{\omega }$ is already very accurate, the procedure remains efficient. A second pass might be unnecessary for single tilt series. This is not the case for multiple tilt series as we will see in the next paragraph. Also, note there is no guarantee every single particle is picked by our automation routine; it is nevertheless very effective because of the two-pass detection scheme—see Appendix "Marker detection".

Compared to other approaches, for instance the beadtrack utility in eTomo [20], our procedure uses a global (3D) framework; it does not try to connect peaks directly from one micrograph to the next one in a sequential manner. This is still possible to achieve with geometrical cross-over consideration, but probably not optimal: there is a large uncertainty on the cross-over location of two nearly parallel rays. Finally, we note that a sequential cross-correlation procedure is used during the built up of the initial projection map.

The seeding step depicted in Fig. 12 and its cluster analysis could be excluded from the iterative loop and implemented only once as an initial routine. Maintaining this step within the loop allows the number of markers to grow and thus more alignment flexibility, with only a limited impact on the algorithm performance.

Once the iterative routine has converged, a final bundle adjustment scheme can be implemented using non-linear projection maps [16, 17]. This would insure a more accurate alignment, and eventually correct for any morphological distortions [18]. For multiple tilt series however, this non-linear bundle adjustment should be carried out when the entire procedure is completed.

Iterative alignment for multiple axis tomography

The automated routine described above can be applied to any number of tilt series without substantial modification, just by considering them as a unique dataset/series. A more robust methodology consists in aligning first each series separately, and handle them globally once their individual signatures—projection map $\{P_\omega \}_\omega$ and marker locations $\left\{ \mathbf{X}^{\tau } \right\} _{\tau }$—are known. This is the approach we adopt; the workflow is summarized in Fig. 13.

For the global alignment, correspondences between markers from different series need to be established. This task is implemented with another iterative procedure.

In the following, $\left\{ \mathbf{X}^{\tau _a}_{a} \right\} _{\tau _a}$ and $\left\{ P_{\omega _a} \right\} _{\omega _a}$ indicate respectively the 3D marker positions and projection maps for tilt series a; $\tau _a$ indexes the markers and $\omega _a$ the specimen orientation.

For every pair of series (a,b), two functions $\alpha _{ab}$ and $\alpha _{ba}$ map the correspondences between the sets $\left\{ \mathbf{X}^{\tau _a}_{a} \right\} _{\tau _a}$ and $\left\{ \mathbf{X}^{\tau _b}_{b} \right\} _{\tau _b}$:

$$\begin{aligned} \begin{array}{l} \tau _b = \alpha _{ab}(\tau _a) \\ \tau _a = \alpha _{ba}(\tau _b). \end{array} \end{aligned}$$

(4)

Since reconstructing from a or b is equivalent, we introduce $T_{ab}$ (resp. $T_{ba}$), the invertible 3D transformation to switch from volume a to volume b (resp. b to a); $T_{ab}^{-1}=T_{ba}$. In this approach, the index mapping $\alpha _{ab}$ is set by comparing the point to point distance between $\{ T_{ab} ( \mathbf{X}^{\tau _a}_{a} ) \}_{\tau _a}$ and $\{ \mathbf{X}^{\tau _b}_{b} \}_{\tau _b}$; this allows to assign a unique $\tau _b$ for each $\tau _a$ as long the corresponding distance is short enough. A similar procedure is used with $\alpha _{ba}$. At the end, only tracks $\tau _a$ from a satisfying the condition $\tau _a = \alpha _{ba} \circ \alpha _{ab}(\tau _a)$ are kept. Those tracks and their b counterpart are then used to refine the transformation $T_{ab}$ with a 3D regression procedure. This process can be repeated iteratively until the number of shared markers between the two sets is maximized.

For the first iteration, the 3D transformation $T_{ab}$ is initialized as a simple 3D rotation around the Z axis. Its parameters, i.e., translation coefficients and rotation angle, are estimated by comparing the zero-tilt images from both series. The images are cross-correlated for different relative rotations; the highest correlation peak provides necessary elements to estimate $T_{ab}$.

This iterative operation is repeated for each pair of tilt series. Once all the shared markers are identified, the original marker sets are recollected as a unique set, and each projection maps redefined in relation to the first series a, that is $P_{\omega _b} \rightarrow P_{\omega _b} \circ T_{ab}$. Those quantities are used as inputs in a final bundle adjustment procedure for the entire system.

It is clear that having detected a maximum number of markers during the single tilt series alignment is important to establish the geometrical correspondence between tilt series.

Appendix 2: Monitoring the sample deformation

Part of our present knowledge about electron beam damage on plastic sections derives from studies relying on tracking fiducial markers [8]. It therefore comes as no surprise that the same information is inherently contained in any EMT process whose alignment procedure shares the same principles. In fact, the overall quality of a reconstruction technique depends on how well the alignment procedure compensates for sample deformation, among other issues.

We describe in this appendix how to simply extract some sample deformation parameters from the projection maps $\left\{ P_{\omega } \right\} _{\omega }$, once the bundle adjustment optimization procedure has been carried out.

In our procedure, the correspondence between a point $\mathbf{X}=(X,Y,Z)$ of the specimen at no tilt and a projection point $\mathbf{x}=(x,y)$ on a micrograph for a given specimen orientation $\omega$ is expressed as a polynomial expansion, which at the first-order writes:

$$\begin{aligned} \left\{ \begin{array}{rcl} x &{} = &{} t_{x,\omega } + n_{1X,\omega } X + n_{1Y,\omega } Y + n_{1Z,\omega } Z + ... \\ y &{} = &{} t_{y,\omega } + n_{2X,\omega } X + n_{2Y,\omega } Y + n_{2Z,\omega } Z + ... \\ \end{array} \right. , \end{aligned}$$

(5)

the linear portion of this development being characterized by a set of two vectors ${ \mathbf n}_{1,\omega }$ and ${ \mathbf n}_{2,\omega }$ at each sample tilt.

In an ideal system, with no sample warping and optics following straight line rules, the projection map in (5) would exactly correspond to projecting the result of two simple rotations: $\Pi _\mathbf{e_Z} \circ \mathcal{R}_\mathbf{{u},\theta } \circ \mathcal{R}_{\mathbf{e_Z},\phi }$; the first one $\mathcal{R}_{\mathbf{e_Z},\phi }$ modeling the sample replacement between each tilt series ($\phi$ is the azimuthal angle), while the second one $\mathcal{R}_\mathbf{{u},\theta }$ describes the sample tilt operation within a series, $\mathbf{{u}}$ is the tilt axis ($\theta$ is the tilt angle), and $\Pi _\mathbf{e_Z}$ the orthogonal projection along the optic axis $\mathbf{e_Z}$.

At a high enough magnification to avoid any optical distortion problem, and with a small electron beam dose rate, we expect the deviations from the ideal case to be simple. In those conditions, the sample would mostly shrink with two different compression factors, a larger one $\gamma _Z$ along the normal axis and a smaller one $\gamma _\perp$ within the specimen plane (if $\gamma _\perp \simeq \gamma _X \simeq \gamma _Y$). This can be mathematically summarized by modeling the projection map (5) with the following:

$$\begin{aligned} \mathbf{x} = \Pi _\mathbf{e_Z} \circ \mathcal{R}_\mathbf{{u},\theta } \circ \mathcal{R}_{\mathbf{e_Z},\phi } \circ \mathcal{W}_{\omega } (\mathbf X), \end{aligned}$$

(6)

The linear part of the affine transformation $\mathcal{W}_{\omega }$ which describes the sample shrinkage is

$$\begin{aligned} \left[ \begin{array}{ccc} \gamma _{X} &{} 0 &{} 0 \\ 0 &{} \gamma _{Y} &{} 0 \\ 0 &{} 0 &{} \gamma _{Z} \\ \end{array} \right] . \end{aligned}$$

(7)

Here, the warping transformation $\mathcal{W}_{\omega }$ is indexed by the sample orientation ${\omega }=(\theta ,\phi )$, but $\mathcal{W}_{\omega }$ really depends on the accumulated dose and the micrograph index of the acquisition order. As the micrographs are collected, the sample will shrink, we assume exponentially, towards a stable plateau whose value is mostly a function of the electron dose rate [8].

$$\begin{aligned} \left\{ \begin{array}{rcl} \gamma _X &{} = &{} \gamma _{X,\infty } + (1-\gamma _{X,\infty }) e^{-i_\omega /\tau _X} \\ \gamma _Y&{} = &{} \gamma _{Y,\infty } + (1-\gamma _{Y,\infty }) e^{-i_\omega /\tau _Z} \\ \gamma _Z &{} = &{} \gamma _{Z,\infty } + (1-\gamma _{Z,\infty }) e^{-i_\omega /\tau _Z} \\ \end{array} \right. , \end{aligned}$$

(8)

here, $\gamma _{X,\infty }$, $\gamma _{Y,\infty }$ and $\gamma _{Z,\infty }$ are the plateau values for both shrinkage coefficients $\gamma _X$, $\gamma _Y$ and $\gamma _Z$; $i_\omega$ is the index of an electron micrograph; $\tau _X$, $\tau _Y$ and $\tau _Z$ are the associated shrinkage rates. Note that during the data acquisition with SerialEM [14], there is always a tracking and an autofocus routine happening between two recorded micrographs. Nevertheless, we can safely assume that $i_\omega$ is directly proportional to the accumulated dose.

A crude linear model with no coupling between axes for the sample deformation, as outlined by the Eqs. (6) , (7) and (8), can already capture some coarse variations of the projection map throughout the tilt series. It is very clear, when monitoring the quantity:

$$\begin{aligned} n_{3,\omega } = \Vert \mathbf{n_{1,\omega } } \times \mathbf{n_{2,\omega } } \Vert \end{aligned}$$

(9)

which becomes

$$\begin{aligned} n_{3,\omega } = \left[ ( \gamma _X^2 \cos ^2 \phi + \gamma _Y^2 \sin ^2 \phi ) \gamma _Z^2 \sin ^2 \theta + \gamma _X^2 \gamma _Y^2 \cos ^2 \theta \right] ^{1/2} \end{aligned}$$

(10)

for the deformation model defined in Eq. (7). In this case, if there was no sample warping ($\gamma _X=1$, $\gamma _Y=1$ and $\gamma _Z=1$), the quantity $n_{3,\omega }$ should remain equal to 1. On the other hand, with more shrinkage along the normal axis of the section, that is $1-\gamma _X,1-\gamma _Y \ll 1-\gamma _Z$, $n_{3,\omega }$ would oscillate between two curves specified by ${\gamma ^-_{\perp }}^2 \left[ 3 \gamma _Z^2 + {\gamma ^+_{\perp }}^2 \right] ^{1/2}/2$ and $\gamma ^-_{\perp } \gamma ^+_{\perp }$, with $\gamma ^-_{\perp } = \text{min}(\gamma _X,\gamma _Y)$ and $\gamma ^+_{\perp } = \text{max}(\gamma _X,\gamma _Y)$ during the multiple axis scheme with a tilt variation between −60° and +60°.

Appendix 3: Weighted iterative approach

In this appendix, we elaborate the principles behind the weighted iterative approach.

Considering electrons traveling across a specimen, their trajectories $\gamma _{\mathbf{x},\omega }(t)$ depend on both their final destination $\mathbf{x}$ on the detector and the specimen orientation $\omega$; t is a variable that parametrizes a particle position along its path; $t=t_0$ and $t=t_1$ correspond to the specimen boundaries. The fundamental principle of electron tomography consists in inverting the following relation:

$$\begin{aligned} v(\mathbf{x},\omega ) = \mathcal{R}_\Gamma u(\mathbf{x},\omega ) \equiv \int\limits _{t_0}^{t_1} u[\gamma _{\mathbf{x},\omega }(t)] dt, \end{aligned}$$

(11)

which defines the Radon transform $\mathcal{R}_\Gamma$, $\Gamma$ being the set of all the possible electron trajectories; u is the volumic representation of the specimen; v is the projection micrograph density.

The formulation of the weighted iterative approach employed in this work involves the following operator $\bar{\mathcal{R}}^*_\Gamma$:

$$\begin{aligned} \bar{\mathcal{R}}^*_\Gamma v (\mathbf{X}) = \int\limits _{P_\omega (\mathbf{X})=\mathbf{x}} \lambda _{\mathbf{x},\omega }(\mathbf{X}) v(\mathbf{x},\omega ) \text{d}\omega , \end{aligned}$$

(12)

$\mathbf{X}$ is a variable representing the 3D positions in the specimen while $P_\omega$ are the projection maps associated with trajectories $\gamma _{\mathbf{x},\omega }(t)$; for every $(\mathbf{x},\omega ,t)$, we have $P_\omega (\gamma _{\mathbf{x},\omega }(t))=\mathbf{x}$; finally, $\lambda _\mathbf{x,\omega }$ is a function that modulates the back-projected densities depending on the position within the specimen and the source of the contribution, that is the originating micrograph.

In electron tomography, it is generally not possible to obtain the exact solution of the inverse problem outlined in Eq. (11), mostly because of the missing wedge problem. The mathematical system is ill-posed, and as a result, many artifacts hamper the reconstructions obtained via standard approaches.

The motivation behind introducing the weighting parameter in Eq. (12) is to enforce constraints on the back-projection operator that will ultimately lead to reconstruction bearing minimal flaw, but still be solution of Eq. (11). Possible choices for $\lambda _\mathbf{x,\omega }$ can be dictated by physics-based arguments. For example, to reduce the streaks effects surrounding dense objects, we can choose an expression that will effectively inflate their absorbance and attenuate the bleaching effect along the rays. Such an expression can be inferred using cross-validation ideas, with for instance,

$$\begin{aligned} \lambda _\mathbf{x,\omega } \sim - \int\limits _{P_\omega (X)=x,\omega \ne \omega '} v(x,\omega ') d \omega '. \end{aligned}$$

(13)

In addition, the original projection constraint along the rays must still be satisfied with:

$$\begin{aligned} \int\limits _{t_0}^{t_1} \lambda _\mathbf{x,\omega } \left( \gamma _\mathbf{x,\omega }(t)\right) dt =1, \end{aligned}$$

(14)

for every micrograph pixel $\mathbf{x}$ and every sample orientation $\omega$.

We have used for this work a SIRT-type iterative approach to find a solution for Eq. (11); the weighted-backprojection operator $\bar{\mathcal{R}}^*_\Gamma$ being used to correct for the discrepancies at each iteration. During the ith step, all the projections $v_i$ that correspond to the original specimen orientations are evaluated from the volume density $u_i$ through Eq. (11). Next, the volume density at iteration $i+1$ is evaluated by using

$$\begin{aligned} u_{i+1} = u_i + \alpha \bar{\mathcal{R}}^*_{\Gamma ,i} (v - v_i), \end{aligned}$$

(15)

with taking the following expression for the weighting operator in $\bar{\mathcal{R}}^*_{\Gamma ,i}$:

$$\begin{aligned} \lambda _{\mathbf{x},\omega ; i} (\mathbf{X}) = \frac{2}{\Delta t}- \frac{u_i(\mathbf{X})}{v_i(\mathbf{x},\omega )}, \end{aligned}$$

(16)

with $\Delta t = t_1-t_0$; $\alpha$ is just a mixing parameter for adjusting the convergence rate.

Equation (16) approximates Eq. (13) and satisfies relation (14). Expressed in this way, the system of Eqs. (15), (16) is similar to the adaptive approach developed in [22].

The iterative process is initialized by simply back-projecting the raw electron micrographs.

Finally, we note that the weighted iterative approach described in this appendix is tightly related to the multiplicative-type algorithm category in image reconstruction; for instance, the expectation maximization algorithm in emission tomography (see chap. 5 in  [24]). For this latter, reconstructions are obtained by maximizing a likelihood function characterizing the physical nature of the phenomenon, a Poisson distribution. The procedure is implemented through a multiplicative iterative scheme, and can be shown equivalent to Eq. (15) with $\alpha =1$. This statistical framework is also very appropriate in the case of electron tomography and its underlying physical principles. It also offers, as discussed above in the context of cross-validation, a more refined solution to artifacts compared to more standard approaches (FBP, SIRT, etc.).