[PDF] Goal-conditioned Offline Reinforcement Learning through State Space Partitioning-论文阅读讨论-ReadPaper

摘要

Offline reinforcement learning (RL) aims to infer sequential decisionpolicies using only offline datasets. This is a particularly difficult setup,especially when learning to achieve multiple different goals or outcomes undera given scenario with only sparse rewards. For offline learning ofgoal-conditioned policies via supervised learning, previous work has shown thatan advantage weighted log-likelihood loss guarantees monotonic policyimprovement. In this work we argue that, despite its benefits, this approach isstill insufficient to fully address the distribution shift and multi-modalityproblems. The latter is particularly severe in long-horizon tasks where findinga unique and optimal policy that goes from a state to the desired goal ischallenging as there may be multiple and potentially conflicting solutions. Totackle these challenges, we propose a complementary advantage-based weightingscheme that introduces an additional source of inductive bias: given avalue-based partitioning of the state space, the contribution of actionsexpected to lead to target regions that are easier to reach, compared to thefinal goal, is further increased. Empirically, we demonstrate that the proposedapproach, Dual-Advantage Weighted Offline Goal-conditioned RL (DAWOG),outperforms several competing offline algorithms in commonly used benchmarks.Analytically, we offer a guarantee that the learnt policy is never worse thanthe underlying behaviour policy.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答