This website requires JavaScript.

Monocular Real-Time Volumetric Performance Capture

Ruilong LiYuliang XiuShunsuke SaitoZeng HuangKyle OlszewskiHao Li
Cornell University - arXiv
Jul 2020
摘要
We present the first approach to volumetric performance capture and novel-view rendering at real-time speed from monocular video, eliminating the need for expensive multi-view systems or cumbersome pre-acquisition of a personalized template model. Our system reconstructs a fully textured 3D human from each frame by leveraging Pixel-Aligned Implicit Function (PIFu). While PIFu achieves high-resolution reconstruction in a memory-efficient manner, its computationally expensive inference prevents us from deploying such a system for real-time applications. To this end, we propose a novel hierarchical surface localization algorithm and a direct rendering method without explicitly extracting surface meshes. By culling unnecessary regions for evaluation in a coarse-to-fine manner, we successfully accelerate the reconstruction by two orders of magnitude from the baseline without compromising the quality. Furthermore, we introduce an Online Hard Example Mining (OHEM) technique that effectively suppresses failure modes due to the rare occurrence of challenging examples. We adaptively update the sampling probability of the training data based on the current reconstruction accuracy, which effectively alleviates reconstruction artifacts. Our experiments and evaluations demonstrate the robustness of our system to various challenging angles, illuminations, poses, and clothing styles. We also show that our approach compares favorably with the state-of-the-art monocular performance capture. Our proposed approach removes the need for multi-view studio settings and enables a consumer-accessible solution for volumetric capture.
展开全部
图表提取

论文十问由沈向洋博士提出，鼓励大家带着这十个问题去阅读论文，用有用的信息构建认知模型。写出自己的十问回答，还有机会在当前页面展示哦。

  1. Q1
    论文试图解决什么问题？
    修宇亮 论文作者 2022/07/19

    用Implicit Function (IF) 做人体几何重建，没有任何加速策略的话，就需要对整个cube space的3D point做遍历，O(n^3)的复杂度，速度实在太慢。

  2. Q2
    这是否是一个新的问题？
    修宇亮 论文作者 2022/07/19

    从方法论上，不算一个新问题，有很多传统的数据结构都可以对其加速，比如Octree，但工程角度，确实还没有直接可以plug-and-play的实现。

  3. Q3
    这篇文章要验证一个什么科学假设？
    修宇亮 论文作者 2022/07/19

    Implicit Function出Mesh，可以做到实时速度。

  4. Q4
    有哪些相关研究？如何归类？谁是这一课题在领域内值得关注的研究员？
  5. Q5
    论文中提到的解决方案之关键是什么？
  6. Q6
    论文中的实验是如何设计的？
  7. Q7
    用于定量评估的数据集是什么？代码有没有开源？
  8. Q8
    论文中的实验及结果有没有很好地支持需要验证的科学假设？
  9. Q9
    这篇论文到底有什么贡献？
  10. Q10
    下一步呢？有什么工作可以继续深入？
3
被引用
38
笔记
0
问答
领域
· Computer Science· CV