Abstract
As the resources for 3D shapes continue to grow, the demand and applications for 3D shapes are also increasing. However, 3D shape modeling remains a labor-intensive and time-consuming task, demanding years of expertise to master. Simultaneously, advancements in VR/AR technologies have made it possible to create 3D content directly through immersive interactions. To facilitate 3D shape modeling, in this thesis, we investigate the utilization of 3D sketches as an input modality and advocate a VR-scenario where 3D shape retrieval and generation are conducted. Our focus is on enabling a user-friendly sketching scenario where the sketches are composed of sparse lines and do not necessitate sketching skills, prior training, or time-consuming precision. Our overarching vision is to empower users to freely retrieve or generate 3D models through casual air-doodling within a VR setting.
Our study begins with the development of a VR interface to gather a dataset of VR sketches. To facilitate the training of retrieval methods, we introduce the first synthetic method for generating VR sketches from open shape datasets. We demonstrate that through training on synthetic sketches, we can achieve reasonable category-level shape retrieval accuracy for human VR sketches. Furthermore, we emphasize that in the context of 3D sketch to 3D shape retrieval, the point-cloud representation outperforms multi-view approaches. Aligning with the recent trend in fine-grained data analysis within the sketch community, we introduce the first fine-grained 3D VR sketch dataset containing 1,497 pairs of 3D VR sketches and 3D shapes within the chair category, showcasing a wide diversity of shapes. Utilizing this dataset, we delve into fine-grained 3D shape retrieval based on 3D VR sketches. We demonstrate that 3D sketches can significantly enhance the accuracy of instance-level shape retrieval when compared to 2D sketches. Furthermore, we enhance the structural similarity of retrieval results by establishing a novel connection between adaptive margin values and shape similarities. Expanding on our accomplishments in fine-grained 3D shape retrieval, we extend our research to the domain of 3D shape generation. We propose a 3D shape generation network conditioned on 3D VR sketches and introduce a dedicated loss function that encourages the generated 3D shapes to faithfully match the input sketch. In the final conclusion, we reflect on the limitations of our current work and outline potential directions for future research.