Structured Outdoor Architecture Reconstruction by Exploration and Classification

ICCV 2021

Fuyang Zhang, Xiang Xu, Nelson Nauata, Yasutaka Furukawa,

Paper Arxiv Code Supp.

This paper presents an explore-and-classify framework for structured architectural reconstruction from an aerial image. Starting from a potentially imperfect building reconstruction by an existing algorithm, our approach 1) explores the space of building models by modifying the reconstruction via heuristic actions; 2) learns to classify the correctness of building models while generating classification labels based on the ground-truth; and 3) repeat. At test time, we iterate exploration and classification, seeking for a result with the best classification score. We evaluate the approach using initial reconstructions by two baselines and two state-of-the-art reconstruction algorithms. Qualitative and quantitative evaluations demonstrate that our approach consistently improves the reconstruction quality from every initial reconstruction.



The geometry exploration module starts from an initial model reconstructed by another algorithm, and produces a set of offspring reconstructions by heuristic actions. The geometry classification module ranks the generated reconstructions and keep the top samples for the next iteration (graphs with green color score on the right corner are the top samples in each iteration). The system dynamically generates training data for the classification module by picking generated reconstructions and creating classification labels based on the ground-truth. The classification module consists of three local primitive classifiers (i.e., junctions, edges, and regions). The first two classifiers are neural networks, which learns to classify if each primitive in the current reconstruction is correct or incorrect. The region classifier measures the consistency with the instance segmentation masks generated by Mask R-CNN. The overall classificationscore is the sum of junction, edge, and region scores.

Related Projects

We propose a novel message passing neural (MPN) architecture Conv-MPN, which reconstructs an outdoor building as a planar graph from a single RGB image. Conv-MPN is specifically designed for cases where nodes of a graph have explicit spatial embedding.



@InProceedings{Zhang2021structured, author = {Zhang, Fuyang and Xu, Xiang and Nauata, Nelson and Furukawa, Yasutaka}, title = {Structured Outdoor Architecture Reconstruction by Exploration and Classification}, booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, month = {October}, year = {2021}, pages = {12427-12435} }