Dynamic Plane Convolutional Occupancy Networks

A WACV 2021 paper. An implicit 3D representation that learns where to project point-cloud features instead of using three fixed planes, improving surface reconstruction for objects and indoor scenes.

The problem

Convolutional Occupancy Networks reconstruct large 3D scenes by projecting point features onto three axis-aligned planes and running a CNN on them. Three fixed planes are a strong assumption: real objects are not always aligned with the axes, and their most informative views rarely are.

The idea

Learn the planes. A shallow network looks at the input point cloud and predicts the planes that best describe it, along with plane-specific features. Per-point features are projected onto these dynamic planes, processed by a U-Net with shared weights, and queried to predict whether any point in space is inside the surface.

Positional encoding of the point coordinates adds fine detail, and a similarity loss encourages the planes to spread over diverse directions.

Result

The method outperformed the state of the art for surface reconstruction from unoriented point clouds on ShapeNet and on an indoor scene dataset. Reconstruction quality improved progressively as planes were added, up to seven, and the model generalised better to inputs in unseen orientations.

Written with Stefan Lionar, Dusan Svilarkovic and Songyou Peng as a 3D Vision course project at ETH Zurich, and published at the Winter Conference on Applications of Computer Vision 2021.