Scale-aware monocular reconstruction via robot kinematics and visual data in neural radiance fields

2.1 Overview of NeRF-based scale-aware reconstruction

Considering robot-assisted endoscopy, the goal of our proposed pipeline, KV-EndoNeRF, is to achieve scale-aware monocular reconstruction from limited multi-modal data (i.e., kinematics, and endoscopic image sequences). It requires neither large numbers of endoscopic images for training, nor other imaging modalities, such as computed tomography (CT) and magnetic resonance image (MRI), for the ground truth labels. The key to our pipeline is to effectively incorporate the scale information from robot kinematics into NeRF-represented surgical scenes for optimization. Following the modeling in NeRF ^[15], we represent the surgical scene as a neural radiance field for further volume rendering (Section 2.2). As shown in Figure 1, we first extract the absolute scale from kinematics and then fuse it into sparse depth produced by SfM. Under sparse supervision, we fine-tune a monocular depth estimation network to the current endoscopic scene for scene-specific coarse depth (Section 2.3). After adjusting the scale of coarse depth estimation, we integrate it into the ray marching of NeRF and optimize the volumetric field to obtain the absolute depth (Section 2.4). Finally, the refined absolute depth maps are fused in a truncated signed distance functions (TSDF)-based volumetric representation according to the endoscopic trajectory (Section 2.5). This results in a reconstructed 3D model of the surgical scene with global-scale information.

Figure 1. Illustration of our proposed KV-EndoNeRF for scale-aware monocular reconstruction from the robotic endoscope. NeFR: Neural radiance fields.

2.2 Surgical scene representing and rendering by NeRF

NeRF has achieved impressive success in view synthesis by optimizing the neural implicit field. Our pipeline explores its potential for the optimization of depth estimates. We represent a surgical scene as a neural radiance field $$ F_\theta $$, which is an 8-layer multilayer perceptron (MLP) with network parameter $$ \theta $$. The field, $$ F_\theta : \left({\bf{x}}, {\bf{d}}\right) \rightarrow \left({\bf{c}}, {{\mathit{σ}}}\right) $$, maps a 3D point $$ {\bf{x}} \in \mathbb{R}^3 $$ and a viewing direction $$ {\bf{d}} \in \mathbb{R}^3 $$ to an RGB value $$ {\bf{c}}\left({\bf{x}}, {\bf{d}}\right) \in \mathbb{R}^3 $$ and space occupancy $$ \sigma \left({\bf{x}}\right) \in \mathbb{R} $$. With scene representations, we further adopt the volume rendering ^[21] in NeRF to generate rendered images for training. The volume rendering starts with shooting a batch of endoscope rays into the surgical scene from the endoscope center $$ {\bf{o}} $$ along the direction $$ {\bf{d}} $$. Each ray is constructed as $$ {\bf{r}}\left(s\right) = {\bf{o}} + s{\bf{d}} $$, where $$ s $$ is the ray parameter. We then proceed with ray marching to sample points in the space. Specifically, we partition each camera ray $$ {\bf{r}}\left(s\right) $$ into a batch of points $$ \left\{{\bf{x}}_k | {\bf{x}}_k = {\bf{r}}\left(s_k\right)\right\}_{k=1}^m $$. Then, the rendered color image $$ {\bf{C}} $$ and per-view depth value $$ {\bf{D}} $$ of a camera ray $$ {\bf{r}}\left(s\right) $$ can be computed using volume rendering as:

where $$ w_k=\left(1-\exp{\left(-\sigma\left({\bf{x}}_k\right) \triangle s_k\right)}\right) \exp{\left(-\sum_{l=1}^{k-1} \sigma\left({\bf{x}}_l\right) \triangle s_l\right)} $$, $$ \triangle s_k = s_{k+1}-s_k $$.

In this way, the 3D structure of the surgical scene can be encoded as a continuous implicit function, which enables memory-efficient geometric representation with infinite resolution.

2.3 Coarse depth adaptation and scale recovery with kinematics

We use the SfM to reconstruct the surgical scene as a set of 3D points $$ \mathcal{X} $$ and camera poses $$ \mathcal{T} = \{{\bf{T}}_i\in{\bf{SE}}\left(3\right)| i = 1 \cdots N\} $$ for the input images extracted from the unlabeled endoscopic video. To eliminate extreme outliers in the sparse reconstruction, point cloud filtering is utilized. For each endoscopic image pair, the rigid transformation matrix $$ ^I{\bf{T}}_i^{i+1} $$ from image $$ i $$ to $$ i+1 $$ can be computed by the camera poses $$ \mathcal{T} $$, where the left superscript $$ \left\{I\right\} $$ denotes the pose described under the image coordinate. As the endoscope is attached to a robot, the camera poses under the robot coordinate system can be calculated from kinematics information, considered as a reference to recover the absolute scale. Therefore, the relative pose $$ ^R{\bf{T}}_i^{i+1} $$ under the robot base $$ \left\{R\right\} $$ is computed. The absolute scale between the reconstructed structure and the real world can be estimated by:

where $$ {\bf{t}}_i $$ is the translation vector of the camera pose $$ {\bf{T}}_i $$. Although we can compute scale data for each frame, the noise in the kinematics data and the instability of the poses in $$ \mathcal{T} $$ introduce severe noise to each scale. To filter the scale, we employ a logarithmic moving average with a multiplicative error model. Based on the computed scale factor, we adjust the sparse 3D structure and camera poses to match the real-world values. Afterward, the scaled 3D point cloud $$ \mathcal{X}^{\prime} $$ is projected onto each image plane with the corresponding scaled camera pose $$ {\bf{T}}_i^{\prime} $$. The re-projected $$ z $$ values are concatenated as the sparse depth supervision $$ ^s{\bf{D}}_i $$, where the region with no points projected onto is set to zero.

To obtain scene-specific coarse depth from the current endoscopic data, we propose adapting a depth estimation network. This network is fine-tuned using the sparse depth supervision $$ ^s{\bf{D}}_i $$. However, due to the scale ambiguity in the predicted depth map, we utilize the scale-invariant log loss ^[22] for training the depth network. The scale-invariant log loss is defined as:

where $$ e_t=\log{y_t}-\log{y_t}^{\prime} $$, $$ y_t $$ represents the coarse depth value predicted by the proposed depth network, and $$ y_t^{\prime} $$ is the value of the corresponding sparse depth supervision $$ ^s{\bf{D}}_i $$. $$ M $$ denotes the number of pixels with valid supervision values, and $$ \alpha $$ is a weighting factor.

2.4 NeRF-based optimization for absolute depth

According to Equation (2), we can determine the absolute scale between the reconstruction and real-world values using the robot kinematics information. To incorporate this calculated scale into dense monocular reconstruction, we propose guiding the NeRF sampling process with our coarse depth estimation and scale information. First, we align the scale of the coarse depth map $$ ^c{\bf{D}}_i $$ based on the depth supervision $$ ^s{\bf{D}}_i $$. Moreover, we compute the confidence map of $$ ^c{\bf{D}}_i $$ by a geometric consistency check. The depth $$ ^c{\bf{D}}_i $$ is first projected onto all other views using the following equations:

where $$ {\bf{K}} $$ represents the endoscope intrinsic matrix, and $$ {\bf{p}} $$ denotes a pixel in the image. Subsequently, we calculate the depth reprojection error between $$ ^c{\bf{D}}_j^{\prime} $$ and $$ ^c{\bf{D}}_{i \to j} $$. The confidence map $$ {\bf{E}}_i $$ for each view is defined as the average value of the top $$ K $$ minimum cross-view depth reprojection errors.

Next, during ray marching, we sample points using a Gaussian distribution guided by the prior from the scaled coarse depth. Assuming the coarse depth value for a pixel $$ {\bf{p}} $$ to be $$ z_{\bf{p}} = \, ^c{\bf{D}}_i\left({\bf{p}}\right) $$, we sample the candidates using the distribution $$ s_k \sim \mathcal{N}\left(z_{\bf{p}}, \delta_{\bf{p}}^2\right) $$, where $$ \delta_{\bf{p}} = z_{\bf{p}} \cdot {\bf{E}}_i\left({\bf{p}}\right) $$. This sampling method ensures that the points are concentrated around tissue surfaces.

To estimate the absolute depth of endoscopic frames, we can optimize the network parameter $$ \theta $$ by supervising the rendered color images. To be more specific, the loss function utilized to train the network is defined as follows:

where $$ {\bf{p}} $$ represents the location of the pixel that $$ {\bf{r}}\left(s\right) $$ shoots toward, and $$ {\bf{I}}_i $$ corresponds to the input endoscopic image.

2.5 Volumetric reconstruction on fine depth

To further refine depth accuracy, we use the view synthesis results of NeRF to calculate the per-pixel error for the predicted structure. If the rendering at a specific pixel does not match the input endoscopic image well, a high error is assigned to the depth prediction of that pixel. The error map $$ {\bf{R}}_i\left({\bf{p}}\right) $$ for the pixel $$ {\bf{p}} $$ in the $$ i $$th view is expressed as:

The error map is then used to improve the estimated depth by a filter. We apply an off-the-shelf post-filtering approach ^[23] to obtain the fine output, which enhances absolute depth estimates, particularly in regions where the renderings are not accurate.

Afterward, these fine depth maps are fused to create a surface reconstruction. We use TSDF ^[24] to build a volumetric representation of the tissue surface. Since the predicted depth maps and the endoscope poses are scaled to the real world, all data are made scale-aware and -consistent before fusion. The surgical scene is represented by a discrete voxel grid, and for each of them, a weighted signed distance to the closest surface is recorded. The TSDF is updated in a straight manner, using sequential averaging for each voxel and the predicted depth for each pixel in every image. Finally, the whole 3D structure is reconstructed by the marching cubes method ^[25] from the volumetric representation.

3.1 Dataset and implementation details

We evaluate our scale-aware monocular reconstruction pipeline on the publicly available SCARED dataset ^[26]. This dataset consists of seven training datasets and two test datasets captured by a da Vinci Xi surgical robot. Each dataset is collected from a porcine model and contains four or five keyframes. Each keyframe includes a video with kinematic information about the endoscope. From each dataset, we randomly select one keyframe and extract a set of 40 to 80 images that cover the entire surgical scene. During the data collection process, the robot manipulates an endoscope to observe the interior scenes of the porcine abdominal anatomy. A projector is used to calculate high-quality depth maps for each frame. As a result, the dataset provides endoscopic videos with ground-truth depth maps and robot kinematics. Typical examples of the SCARED data are illustrated in Figure 2. In addition, the robot kinematics information is utilized to restore the scale.

Figure 2. Four typical examples of the SCARED data. For every row, when the robot manipulates the endoscope to move, diversified views and corresponding robot kinematics are recorded in sequence. SCARED: Stereo Correspondence And Reconstruction of Endoscopic Data.

In our implementation, we used the network architecture proposed in Mannequin Challenge ^[27] with pre-trained weights as the monocular depth network for coarse depth adaptation. Twenty fine-tuning epochs were used in the surgical scene-specific adaptation. We set $$ K=4 $$ for the geometric consistency check. For the NeRF-based optimization, we followed the settings in NeRF ^[15]. Specifically, we sampled 64 points in each ray and used a batch of 1, 024 rays during the training. We added random Gaussian noise with zero mean and unit variance to the density to regularize the network. Additionally, positional encoding was utilized to capture high-frequency details. Using Adam optimizer with an initial learning rate of 5e-4, which decayed exponentially to 5e-5, we trained our NeRF on each surgical scene for 200 $$ K $$ iterations. All experiments were conducted on a single RTX 2080 Ti.

3.2 Performance metrics

Table 1 lists the depth evaluation metrics ^[28] used in our experiments, where $$ d $$ and $$ d^* $$ denote the estimated depth value and the corresponding ground truth, respectively, $$ \textbf{D} $$ represents the estimated depth map, and $$ \upsilon \in \{1.25^1, 1.25^2\} $$. Additionally, since the comparison methods cannot accurately predict depth maps with an absolute scale from monocular images, we employ the ground truth median scaling method ^[29] to scale the predicted depth. The scaling is performed as follows:

where $$ {\bf{D}}_{sd} $$ denotes the scaled predicted depth, $$ s $$ represents the scale information calculated by the median scaling method, and $$ {\bf{G}} $$ is the ground truth depth.

3.3 Evaluation on scale-aware depth estimation

We compare the accuracy of depth estimation using the KV-EndoNeRF method with several other deep learning-based approaches and the SfM method, specifically COLMAP ^[7].

● COLMAP ^[7] is a general-purpose SfM pipeline used for reconstructing 3D point cloud reconstruction from ordered and unordered image collections. In our study, we apply it to monocular surgical scene reconstruction. The recovered points are then projected onto each image plane to obtain the sparse depth maps for evaluation.

● EndoSLAM ^[30] is an unsupervised relative monocular depth estimation method specifically designed for gastrointestinal tract organs. It combines residual networks with a spatial attention module to focus on highly textured tissue regions. We fine-tune the depth model using the SCARED data for comparison.

● AF-SfMLearner ^[10] is a novel self-supervised network for estimating monocular depth in endoscopic scenes. It is trained on the SCARED datasets, which contain severe brightness fluctuations induced by illumination variations, non-Lambertian reflections, and inter-reflections.

● DS-NeRF ^[17] is a general depth-supervised NeRF method that utilizes sparse reconstruction from the SfM to recover dense 3D structures. We apply DS-NeRF to estimate dense depth maps for each endoscopic image.

We present the quantitative depth comparison results on SCARED data in Table 2, which rescales the results using the ground truth median scaling method. In addition to standard depth evaluation metrics, we calculate the means and standard errors of the rescaling factors to demonstrate the scale-awareness ability. KV-EndoNeRF achieves the best up-to-scale performance with respect to five metrics and ranks the second best for the other two metrics. Notably, KV-EndoNeRF also achieves nearly perfect absolute scale estimation. These quantitative results show that our proposed method effectively extracts absolute scale information from kinematics and integrates it into NeRF for further depth optimization, resulting in accurate absolute depth estimation.

Furthermore, we select four representative images from the SCARED dataset for qualitative depth comparison. As shown in Figure 3, our method with NeRF-based optimization produces depth predictions with sharp boundaries and fine-grained details, outperforming other approaches in terms of absolute depth estimation. However, COLMAP could only recover sparse depth maps without the entire 3D geometry of the tissue surface. While EndoSLAM and AF-SfMLearner are capable of generating reasonable 3D structures of tissues, they lose many details in tissues with complex geometries and edges. Lastly, the estimated depth values from DS-NeRF contain significant noise, which could affect the surgeons' observations of complicated tissue surfaces.

Figure 3. Qualitative comparisons on SCARED. Our method outperforms COLMAP^[7], EndoSLAM^[30], AF-SfMLearner^[10], and DS-NeRF^[17] in terms of depth quality. A large depth value is encoded with yellow, while a small depth value is encoded with purple. SCARED: Stereo Correspondence And Reconstruction of Endoscopic Data; NeRF: neural radiance fields.

3.4 Comparison with state-of-the-art methods

We compare our method with state-of-the-art approaches in terms of 3D reconstruction and view synthesis. Firstly, we quantitatively assess the reconstruction results and compare them with ground truth 3D models calculated by a structure light camera ^[26]. Unlike other monocular scene reconstruction methods, we do not scale the structures during evaluation, thanks to our scale-aware depth estimation. KV-EndoNeRF achieves high accuracy in 3D reconstruction, with an average root mean square error (RMSE) error of $$ 1.259 \pm 0.257 $$ mm across all data. Figure 4A shows a qualitative comparison of SCARED data. As shown in the figure, the ground truth models in the third column, represented by gray points, indicate that these tissues have complex surfaces. The sparse point clouds recovered by COLMAP are presented in the first column of the figure. Due to the sparsity of the 3D points, it is difficult to observe the geometric structures and the textures of the tissue surfaces. In comparison, our reconstructed meshes shown in the second column present reasonable structures and rich details of the surface. Furthermore, we register the reconstruction results with the ground truth structures, and the registration results show that our 3D reconstruction matches well with the ground truth. In summary, our method can reconstruct smooth 3D structures from a monocular endoscope with accurate scale, high accuracy, and rich details of the surface texture. Moreover, in Figure 4B, we observe that the proposed method benefits the view synthesis quality of NeRF. With the coarse depth priors, our method improves the rendering quality for view synthesis. More 3D point cloud comparison results are illustrated in the Supplementary Materials.

Figure 4. (A) Qualitative reconstruction comparisons on SCARED; (B) Results on view synthesis. Better viewed when zoomed in. More results are shown in the Supplementary Materials. SCARED: Stereo Correspondence And Reconstruction of Endoscopic Data.

3.5 Ablation studies

We perform ablation studies to validate the effectiveness of the proposed pipeline in estimating fine absolute depth using robot kinematics and NeRF-based optimization. Results in Table 3 demonstrate that each module contributes to the final depth quality. Although the coarse depth estimation is not scaled, it provides a relatively accurate depth basis for the following NeRF-based optimization, as shown in the table. After computing the scale from kinematics data, we incorporate it into NeRF to optimize depth further. We observe that the NeRF improves the depth quality and retains the absolute scale information. Additionally, the refinement operation based on the view synthesis enhances absolute depth estimates, which is beneficial to the final scale-aware reconstruction.

Authors' contributions

Conceptualization, methodology, software, validation, formal analysis, investigation, data curation, visualization, writing - original draft: Wei R

Conceptualization, formal analysis, writing - review and editing: Guo J

Conceptualization, writing - review and editing: Lu Y, Zhong F

Conceptualization, resources, formal analysis, Writing review and editing, supervision: Liu Y, Sun D, Dou Q

Availability of data and materials

Stereo Correspondence And Reconstruction of Endoscopic Data (SCARED) is publicly available at https://endovissub2019-scared.grand-challenge.org.

Financial support and sponsorship

This research work was supported in part by Shenzhen Portion of Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone under HZQB-KCZYB-20200089, in part by Hong Kong Research Grants Council Project No. T42-409/18-R, and in part by National Natural Science Foundation of China Project No. 62322318.

Conflicts of interest

All authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

As our research does not deal with any patient data or broadly, we did not obtain an institutional review board (IRB) for this study.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2024.

Supplementary Materials