Sketches provide a sparse yet expressive modality for human–machine interaction, but their use in 3D scene understanding remains largely unexplored. While sketch-based image retrieval (SBIR) and text-driven 3D localisation have made significant progress, the problem of localising specific object instances in full 3D scenes from free-hand sketches has received comparatively little attention.

We introduce a framework for sketch-driven 3D object localisation in reconstructed indoor environments. Given a human sketch, our method retrieves candidate detections across a pose-aligned image bank, ranks them using a sketch-conditioned similarity model, and lifts the highest-ranked observations into 3D using calibrated camera geometry. Multi-view frustum aggregation over a structured primitive scene representation then produces a geometrically consistent object hypothesis.

Experiments across multiple apartment-scale environments demonstrate that sketches provide a powerful complement to text-based retrieval, substantially reducing localisation errors in scenes containing multiple visually similar object instances. These results establish sketch-driven localisation as a promising modality for resolving instance-level ambiguity in large-scale 3D environments.