TriNeRFLet: A Wavelet Based Multiscale Triplane NeRF Representation
Multiscale 2D wavelet representation with high quality reconstruction and super-resolution
Tel Aviv University
(Coming Soon)
Overview
In this work, we propose TriNeRFLet, a 2D wavelet-based multiscale triplane representation for NeRF, which closes the 3D recovery performance gap and is competitive with current state-of-the-art methods. Building upon the triplane framework, we also propose a novel super-resolution (SR) technique that combines a diffusion model with TriNeRFLet for improving NeRF resolution.
Example of the multiscale property of the wavelet representation. On the right, we the same feautre plane at different resolutions. On the left we see the wavelet representation of the plane at the left, which is of the highest resolution. Note that its wavelet representation include the wavelet representations of the lower-resolution planes (the colors corresponds to the matching between each plane and its wavelet representation).
TriNeRFLet training scheme. First, wavelet features are transformed into the Triplane domain. Next, pixels are rendered using these features in order to fit them to their ground-truth values. The high frequencies channels of LH, HL and HH from all wavelet levels get regularized by L_1 loss.
SD-refine process . First, a HR image is rendered. To improve its quality, noise is added to it and then it is plugged into Stable Diffusion upscaler with its LR version for conditioning. The result, which is a refined HR image is added to the HR-set.
TriNeRFLet super-resolution. Low-resolution TriNeRFLet renders LR images using the LR version of the wavelet features, and fits them to the given low-resolution images. High-resolution TriNeRFLet renders HR images using all wavelet levels, and fits them to HR images from HR-set.