Cross-Dimensional Refined Learning for Real-Time 3D Visual Perception from Monocular Video

Ziyang HongC. Patrick Yue

Mar 2023

摘要

We present a novel real-time capable learning method that jointly perceives a3D scene's geometry structure and semantic labels. Recent approaches toreal-time 3D scene reconstruction mostly adopt a volumetric scheme, where atruncated signed distance function (TSDF) is directly regressed. However, thesevolumetric approaches tend to focus on the global coherence of theirreconstructions, which leads to a lack of local geometrical detail. To overcomethis issue, we propose to leverage the latent geometrical prior knowledge in 2Dimage features by explicit depth prediction and anchored feature generation, torefine the occupancy learning in TSDF volume. Besides, we find that thiscross-dimensional feature refinement methodology can also be adopted for thesemantic segmentation task. Hence, we proposed an end-to-end cross-dimensionalrefinement neural network (CDRNet) to extract both 3D mesh and 3D semanticlabeling in real time. The experiment results show that the proposed methodachieves state-of-the-art 3D perception efficiency on multiple datasets, whichindicates the great potential of our method for industrial applications.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域