Multi-Modal Foundation · ADSKAILab
WaLa MVDream DM6
Generates six-view depth maps from text prompts as an intermediate representation for 3D shape generation.
What WaLa MVDream DM6 is
WaLa MVDream DM6 is a generative model developed by researchers at ADSKAILab to create multi-view depth maps directly from written descriptions. Designed for scientific research in engineering and materials science, it serves as an intermediate component within the Wavelet Latent Diffusion (WaLa) text-to-3D pipeline.
The model produces six-view depth maps that describe the surface distance of an object from multiple angles, ensuring comprehensive visual coverage. These generated depth maps are intended to be fed directly into downstream models, specifically WaLa-DM6-1B, to reconstruct complete three-dimensional geometry from text.
How it works
The model is built on the MVDream architecture and adapts the Stable Diffusion framework. Initialised from pretrained MVDream weights, it is fine-tuned to output six depth views instead of standard RGB images given a text prompt.
Fine-tuning was conducted on paired depth map and text data using NVIDIA H100 GPUs. Captions for the training dataset were produced using InternVL 2.0 based on four renderings and two distinct prompts per object, followed by augmentation via LLaMA 3.1 to enhance descriptive diversity.
Key facts
- Model architecture
- MVDream fine-tuned on Stable Diffusion
- Training hardware
- NVIDIA H100 GPUs
- Input format
- Text description
- Output format
- Six-view depth maps
What you can use it for
- Text-Conditioned Multi-View Depth Synthesis
- Synthesising coordinated six-angle depth representations from natural language descriptions for visual analysis.
- Intermediate Conditioning for 3D Geometry Generation
- Producing multi-view depth maps designed to drive 3D shape generation in downstream architectures such as WaLa-DM6-1B.
- Exploration of Wavelet Latent Diffusion Frameworks
- Conducting scientific research into multi-view diffusion mechanisms within the broader WaLa generative framework.
Run it here
We list this model but cannot run it yet: no library and no architectures — nothing can load it.
Everything on this page still applies. Running it needs a loader for this repository, not a bigger machine.
Strengths and limits
Good at
- Generates six distinct depth views simultaneously to provide comprehensive surface coverage of target objects.
- Leverages pretrained MVDream weights fine-tuned specifically for depth-map representation.
- Trained on captions enriched through multi-prompt vision-language captioning and large language model augmentation.
Watch out for
- May produce depth maps that lack consistency across the six views or fail to accurately depict the input text.
- Output depth quality directly affects the reconstruction fidelity of subsequent 3D shape generation stages.
- Performance can vary significantly depending on the specificity and complexity of the prompt.
- Inherits biases from the underlying text-image datasets used in pretraining and fine-tuning.
Inputs and outputs
You provide
Prompt
What to generate.
You get back
The returned image.
Licence
Check the licence before commercial use · other
The other licence carries terms worth reading before you depend on it.
Questions
Can I use WaLa MVDream DM6 commercially?
Check the licence before commercial use. The other licence carries terms worth reading before you depend on it.
Can this model produce 3D meshes directly?
No, it generates six-view depth maps. To obtain 3D shapes, its outputs must be processed by a downstream model such as WaLa-DM6-1B.
What inputs does the model take?
It takes natural language text descriptions as input.
How were the training captions prepared?
Captions were initially generated by InternVL 2.0 using four object renderings and two prompts, then augmented with LLaMA 3.1 to improve richness and diversity.
How to cite it
Aditya Sanghi, Aliasghar Khani, Pradyumna Reddy, Arianna Rampini, Derek Cheung, Kamal Rahimi Malekshan, Kanika Madan, Hooman Shayani. Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings. arXiv:2411.08017 [cs.CV], 2024.
Written from WaLa MVDream DM6's own documentation (model card). Statements citing a figure that does not appear there are removed before publication. Licence and running details come from our registry.