DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling

Linqi Zhou1, Andy Shih1, Chenlin Meng1,2, Stefano Ermon1
1Stanford University, 2Pika Labs
Paper arXiv Code

Abstract

Recent methods such as Score Distillation Sampling (SDS) and Variational Score Distillation (VSD) using 2D diffusion models for text-to-3D generation have demonstrated impressive generation quality. However, the long generation time of such algorithms significantly degrades the user experience. To tackle this problem, we propose DreamPropeller, a drop-in acceleration algorithm that can be wrapped around any existing text-to-3D generation pipeline based on score distillation. Our framework generalizes Picard iterations, a classical algorithm for parallel sampling an ODE path, and can account for non-ODE paths such as momentum-based gradient updates and changes in dimensions during the optimization process as in many cases of 3D generation. We show that our algorithm trades parallel compute for wallclock time and empirically achieves up to 4.7x speedup with a negligible drop in generation quality for all tested frameworks.

Accelerating Text-to-3D Generation

Our accelerator can be wrapped around all existing SDS/VSD-based 3D generation and achieves significant speedup without quality degradation. We do observe slight visual differences with DreamPropeller. This is likely due to randomness in GPU allocation and differences of each GPU's internal state, and with non-zero fixed-point error, momentum-based gradient update brings generation to different local minima. Samples within each comparison are run to the same number of total steps.

ProlificDreamer-Coarse +DreamPropeller ProlificDreamer-Coarse +DreamPropeller
ProlificDreamer-Geometry +DreamPropeller ProlificDreamer-Geometry +DreamPropeller
ProlificDreamer-Texture +DreamPropeller ProlificDreamer-Texture +DreamPropeller
"a detailed Victorian era house" "a pineapple"
DreamFusion +DreamPropeller DreamFusion +DreamPropeller
"an ice cream sundae" "a brightly colored mushroom growing on a log"
DreamGaussian +DreamPropeller DreamGaussian +DreamPropeller
"the leaning tower of Pisa, aerial view" "an ice cream"
Magic3D-Coarse +DreamPropeller Magic3D-Coarse +DreamPropeller
Magic3D-Refine +DreamPropeller Magic3D-Refine +DreamPropeller
"a beautiful dress made out of fruit, on a mannequin. Studio lighting, high quality, high resolution" "a DSLR photo of a blue tulip"
TextMesh +DreamPropeller TextMesh +DreamPropeller
"a wide angle DLSR photo of a squirrel in samurai armor wielding a katana" "an old vintage car"

Application to Image-to-3D Generation

Our accelerator can also speed up Image-to-3D generation guided by score distillation. We test on NeRF and 3D Gaussian Splatting, both of which are guided by Zero 1-to-3.

Source NeRF +DreamPropeller Source NeRF +DreamPropeller
Given image. Given image.
Source 3DGS +DreamPropeller Source 3DGS +DreamPropeller
Given image. Given image.

BibTeX

@article{zhou2023dreampropeller,
  title={DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling},
  author={Zhou, Linqi and Shih, Andy and Meng, Chenlin and Ermon, Stefano},
  journal={arXiv preprint arXiv:2311.17082},
  year={2023}
}

Work done at Pika Labs. For questions please direct to Linqi Zhou.