Structured Outdoor Architecture Reconstruction by Exploration and Classification

Fuyang Zhang, Xiang Xu, Nelson Nauata and Yasutaka Furukawa


Figure 1. Our explore-and-classify reconstruction framework iterates between exploring new building structures by heuristic actions and classifying the correctness of geometries. The figure shows a particular path of exploration by our system. The red color highlights the major improvements against the previous step.


This paper presents an explore-and-classify framework for structured architectural reconstruction from an aerial image. Starting from a potentially imperfect building reconstruction by an existing algorithm, our approach 1) explores the space of building models by modifying the reconstruction via heuristic actions; 2) learns to classify the correctness of building models while generating classification labels based on the ground-truth; and 3) repeat. At test time, we iterate exploration and classification, seeking for a result with the best classification score. We evaluate the approach using initial reconstructions by two baselines and two state-of-the-art reconstruction algorithms. Qualitative and quantitative evaluations demonstrate that our approach consistently improves the reconstruction quality from every initial reconstruction.


Figure 2. The geometry exploration module starts from an initial model reconstructed by another algorithm, and produces a set of offspring reconstructions by heuristic actions. The geometry classification module ranks the generated reconstructions and keep the top samples for the next iteration (graphs with green color score on the right corner are the top samples in each iteration). The system dynamically generates training data for the classification module by picking generated reconstructions and creating classification labels based on the ground-truth. The classification module consists of three local primitive classifiers (i.e., junctions, edges, and regions). The first two classifiers are neural networks, which learns to classify if each primitive in the current reconstruction is correct or incorrect. The region classifier measures the consistency with the instance segmentation masks generated by Mask R-CNN. The overall classificationscore is the sum of junction, edge, and region scores.


[Arxiv] [Supp.] [Bibtex]


Check the code on this repo.


This research is partially supported by NSERC Discovery Grants with Accelerator Supplements and DND/NSERC Discovery Grant Supplement.