DreamFusion

DreamFusion is a text-to-3D generation method that utilizes a pretrained 2D text-to-image diffusion model. This approach addresses the challenges of requiring large-scale datasets and efficient architectures for 3D synthesis. Instead of training on 3D assets, DreamFusion uses a loss function based on probability density distillation to optimize a randomly-initialized 3D model, specifically a Neural Radiance Field (NeRF). This is achieved through a DeepDream-like procedure where the model's 2D renderings from different angles are optimized to achieve a low loss. The resulting 3D model can be viewed from various perspectives, relit with different illumination, and composited into 3D environments. DreamFusion does not require any 3D training data and can be used with a pretrained image diffusion model, showcasing the effectiveness of such models as priors for 3D synthesis. The method also allows for easy integration into 3D renderers or modeling software through mesh exports using the marching cubes algorithm. To generate 3D objects, DreamFusion utilizes Score Distillation Sampling (SDS), a technique that optimizes samples from a diffusion model using a differentiable loss function. This allows for optimization in an arbitrary