PECAN

Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination
Xingzhou Lou · Jiaxian Guo · Junge Zhang · Jun Wang · Kaiqi Huang · Yali Du
CASIA · The University of Sydney · UCL · KCL

Abstract

Zero-shot human-AI coordination holds the promise of collaborating with humans without human data. Prevailing methods try to train the ego agent with a population of partners via self-play. However, this kind of method suffers from two problems: 1) The diversity of a population with finite partners is limited, thereby limiting the capacity of the trained ego agent to collaborate with a novel human; 2) Current methods only provide a common best response for every partner in the population, which may result in poor zero-shot coordination performance with a novel partner or humans. To address these issues, we first propose the policy ensemble method to increase the diversity of partners in the population, and then develop a context-aware method enabling the ego agent to analyze and identify the partner's potential policy primitives so that it can take different actions accordingly. In this way, the ego agent is able to learn more universal cooperative behaviors for collaborating with diverse partners. We conduct experiments on Overcooked environment, and evaluate the zero-shot human-AI coordination performance of our method with both behavior-cloned human proxies and real humans. The results demonstrate that our method significantly increases the diversity of partners and enables ego agents to learn more diverse behaviors than baselines, thus achieving state-of-the-art performance in all scenarios.
图 1:不同训练范式对比 (a) Self-play training (SP) — 与自身副本训练。
(b) Population-based training (PBT) — 与一组伙伴训练,每轮采样一个伙伴。
(c) PECAN — 每轮生成一个策略集成伙伴,ego agent 通过预训练的上下文感知模块在回合开始时识别伙伴的等级上下文。
对比传统自对弈与群体训练,PECAN 引入策略集成与上下文感知

Task

我们采用 Overcooked 中的五个布局(Cramped Room, Asymmetric Advantages, Coordination Ring, Forced Coordination, Counter Circuit)来评估 ego agent 与新颖伙伴的协作能力。

五个 Overcooked 布局 Cramped Room · Asymmetric Advantages · Coordination Ring · Forced Coordination · Counter Circuit
覆盖不同协作难度的任务场景

Procedure

1. 与人类代理模型

使用行为克隆代理模型(behavior-cloned human proxy)测试协作表现,并进行消融实验。 此外,设计实验验证策略集成提升伙伴多样性,以及 PECAN 学习到上下文感知策略。

2. 与真实人类

招募真实玩家评估 PECAN 的人机协作能力,要求玩家对智能体进行主观评分。 最后通过两个案例研究展示 PECAN 在人机协作中的自适应能力。

Results

📊 与人类代理模型的协作结果 PECAN 在五个布局上均超越基线。在 Asymmetric Advantages 中提升尤为显著(+26.8%)。
PECAN 有效提升协作能力,而非单纯完成任务
👥 真实人类协作表现 PECAN 在人类-AI 协作中优于基线。人类玩家对 PECAN 的自适应性和偏好评分更高。
PECAN 在真实人机协作中表现更优

Ablation Study

📉 消融实验结果 移除策略集成(e)或上下文编码器(c)后性能下降,部分布局甚至低于基线。
两个组件对性能均有重要贡献

下图展示了策略集成与上下文感知模块的效果:

🔍 组件效果分析 (a) t-SNE 显示策略集成提升了伙伴多样性。
(b) 当输入随机上下文或错误上下文时性能下降,表明 ego agent 的策略是上下文感知的。
策略集成增加多样性,上下文感知使 ego agent 灵活调整

Real Humans

📋 主观评分 人类玩家认为 PECAN 更具适应性,且更愿意与 PECAN 合作。
主观评价进一步验证 PECAN 的优势
🎮 案例研究 通过两个案例(Cramped Room 与 Asymmetric Advantages)展示 PECAN 的自适应行为。
详见 Demo Videos 部分

Demo 1: Cramped Room

人类玩家控制 蓝色 厨师, AI 控制 绿色 厨师。

▶️ MEP 演示
MEP:当人类阻挡时,AI 停滞等待
▶️ PECAN 演示
PECAN:人类阻挡时立即调整路线,从另一侧取洋葱
💡 人类有意阻挡 AI 前往洋葱的路线,PECAN 立即调整,而 MEP 则原地等待。

Demo 2: Asymmetric Advantages

人类玩家控制 蓝色 厨师, AI 控制 绿色 厨师。

▶️ MEP 演示
MEP:偏向从 1 号锅取菜,忽略 2 号锅
▶️ PECAN 演示
PECAN:从两个锅取菜,上菜更多,策略更多样
💡 该布局的技巧在于 AI 应从两个锅中取菜并上菜。MEP 强烈偏好 1 号锅,而 PECAN 无此偏好,上菜数量更多。