Narrator: Towards Natural Control of Human-Scene Interaction Generation via Relationship Reasoning

Haibiao XuanXiongzheng LiJinsong ZhangHongwen ZhangYebin LiuKun Li

Mar 2023

摘要

Naturally controllable human-scene interaction (HSI) generation has animportant role in various fields, such as VR/AR content creation andhuman-centered AI. However, existing methods are unnatural and unintuitive intheir controllability, which heavily limits their application in practice.Therefore, we focus on a challenging task of naturally and controllablygenerating realistic and diverse HSIs from textual descriptions. From humancognition, the ideal generative model should correctly reason about spatialrelationships and interactive actions. To that end, we propose Narrator, anovel relationship reasoning-based generative approach using a conditionalvariation autoencoder for naturally controllable generation given a 3D sceneand a textual description. Also, we model global and local spatialrelationships in a 3D scene and a textual description respectively based on thescene graph, and introduce a partlevel action mechanism to representinteractions as atomic body part states. In particular, benefiting from ourrelationship reasoning, we further propose a simple yet effective multi-humangeneration strategy, which is the first exploration for controllablemulti-human scene interaction generation. Our extensive experiments andperceptual studies show that Narrator can controllably generate diverseinteractions and significantly outperform existing works. The code and datasetwill be available for research purposes.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域