This website requires JavaScript.

Characterizing Manipulation from AI Systems

Micah CarrollAlan ChanHenry AshtonDavid Krueger
Mar 2023
Manipulation is a common concern in many domains, such as social media,advertising, and chatbots. As AI systems mediate more of our interactions withthe world, it is important to understand the degree to which AI systems mightmanipulate humans \textit{without the intent of the system designers}. Our workclarifies challenges in defining and measuring manipulation in the context ofAI systems. Firstly, we build upon prior literature on manipulation from otherfields and characterize the space of possible notions of manipulation, which wefind to depend upon the concepts of incentives, intent, harm, and covertness.We review proposals on how to operationalize each factor. Second, we propose adefinition of manipulation based on our characterization: a system is manipulative \textit{if it acts as if it were pursuing anincentive to change a human (or another agent) intentionally and covertly}.Third, we discuss the connections between manipulation and related concepts,such as deception and coercion. Finally, we contextualize ouroperationalization of manipulation in some applications. Our overall assessmentis that while some progress has been made in defining and measuringmanipulation from AI systems, many gaps remain. In the absence of a consensus definition and reliable toolsfor measurement, we cannot rule out the possibility that AI systems learn tomanipulate humans without the intent of the system designers. We argue that such manipulation poses a significant threat to human autonomy,suggesting that precautionary actions to mitigate it are warranted.