Gradient-Weighted Feature Back-Projection: A Fast Alternative to Feature Distillation in 3D Gaussian Splatting

Joji Joseph, Bharadwaj Amrutur, Shalabh Bhatnagar




(a) (b) (c)

Instantaneous 3D Segmentation (Prompt: Table, Negative Prompt: Vase, Other)

(a) Rendering without any segmentation. (b) The result after extracting the segmented regions. (c) The result after deleting the segmented regions.

Abstract

We introduce a training-free method for feature field rendering in Gaussian splatting. Our approach back-projects 2D features into pre-trained 3D Gaussians, using a weighted sum based on each Gaussian's influence in the final rendering. While most training-based feature field rendering methods excel at 2D segmentation but perform poorly at 3D segmentation without post-processing, our method achieves high-quality results in both 2D and 3D segmentation. Experimental results demonstrate that our approach is fast, scalable, and offers performance comparable to training-based methods.

Highlights

Feature Back-Projection

$$ \mathbf{f}_k = \frac{\sum_{(x,y,n)} \mathbf{F}_{2D}(x,y,n)\alpha_k(x,y,n)T_k(x,y,n)}{\sum_{(x,y,n)} \alpha_k(x,y,n)T_k(x,y,n)} $$ \begin{aligned} &\textbf{Terms Explanation:} \\ &\mathbf{f}_k: \text{Feature vector for the \( k \)-th Gaussian. Represents the aggregated feature assigned to the Gaussian in 3D.} \\ &\mathbf{F}_{2D}(x, y, n): \text{Feature vector at the 2D pixel coordinate \( (x, y) \) in the \( n \)-th view. Comes from input feature maps.} \\ &\alpha_k(x, y, n): \text{Opacity of the \( k \)-th Gaussian at pixel \( (x, y) \) in view \( n \).} \\ &T_k(x, y, n): \text{Transmittance of the \( k \)-th Gaussian at pixel \( (x, y) \) in view \( n \), accounting for occlusion effects.} \\ \end{aligned}

Gallery

2D Segmentation

Feature 3DGS Ours
Prompt: Table, Negative Prompt: Vase, Other
Prompt: Plant, Negative Prompt: Other
Prompt: Plant, Negative Prompt: Vase, Other
Prompt: Truck, Negative Prompt: Other

3D Segmentation

Extraction Deletion
Feature 3DGS Ours Feature 3DGS Ours
Prompt: Table, Negative Prompt: Vase, Other
Prompt: Plant, Negative Prompt: Other
Prompt: Plant, Negative Prompt: Vase, Other
Prompt: Truck, Negative Prompt: Other

Citation

If you find this paper or the code helpful for your work, please consider citing the following preprint: 


@misc{joseph2024gradientweightedfeaturebackprojectionfast,
    title={Gradient-Weighted Feature Back-Projection: A Fast Alternative to Feature Distillation in 3D Gaussian Splatting}, 
    author={Joji Joseph and Bharadwaj Amrutur and Shalabh Bhatnagar},
    year={2024},
    eprint={2411.15193},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2411.15193}, 
}