3D scenes & avatars
Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User’s Casual Sketches
arXiv:2408.04567 · Preprint · 2024
RESEARCH SUMMARY
Sketch2Scene converts casual sketches into interactive 3D game scenes by combining diffusion-based concept generation, scene interpretation and procedural construction in a game engine.
Read the supporting paper ↗
THE RESEARCH QUESTION
How can a rough sketch become a structured 3D environment?
A generated image can suggest a landscape without defining editable objects, terrain or spatial relationships. An interactive scene needs these explicit components. Sketch2Scene connects image generation with a structured construction pipeline. [Paper]
How the method works
- A sketch-guided diffusion stage produces an isometric concept image, reducing ambiguity about the camera viewpoint.
- Scene interpretation extracts layout and components such as terrain, trees and buildings from that image.
- Procedural content generation assembles the interpreted elements into an interactive 3D engine scene.
RESEARCH INTERPRETATION
What this contribution means
The contribution is the bridge from a visual design suggestion to explicit scene structure. Combining pretrained image generation with procedural tools avoids requiring a single model to learn every aspect of end-to-end 3D scene construction.
This section explains the contribution in context. For experimental protocols, numerical results and comparisons, consult the original paper.
Scope and limitations
The resulting scene follows the categories and construction rules supported by the pipeline. An interactive environment is not automatically a fully authored game with arbitrary mechanics, and errors in image interpretation can propagate into the scene. [Paper]
Questions about this research
Does Sketch2Scene output only a concept image?
No. The image is an intermediate representation used to build a structured, interactive 3D scene.
Sources, paper & code
Primary research sources support the summary above. Project resources provide demonstrations or implementations; Google Scholar is a bibliographic index.
- Primary sourcearXiv — paper record and versions ↗arxiv.org
- Primary sourceFull paper — PDF ↗arxiv.org
- Research resourceProject — demos and resources ↗xrvisionlabs.github.io
- Metadata indexGoogle Scholar — bibliographic listing ↗scholar.google.com