TOT: Topology-Aware Optimal Transport For Multimodal Hate Detection

Linhao ZhangLi JinXian SunGuangluan XuZequn ZhangXiaoyu LiNayu LiuShiyao YanQing Liu

Linhao ZhangLi JinXian Sun ...+5 Qing Liu

Feb 2023

摘要

Multimodal hate detection, which aims to identify harmful content online suchas memes, is crucial for building a wholesome internet environment. Previouswork has made enlightening exploration in detecting explicit hate remarks.However, most of their approaches neglect the analysis of implicit harm, whichis particularly challenging as explicit text markers and demographic visualcues are often twisted or missing. The leveraged cross-modal attentionmechanisms also suffer from the distributional modality gap and lack logicalinterpretability. To address these semantic gaps issues, we propose TOT: atopology-aware optimal transport framework to decipher the implicit harm inmemes scenario, which formulates the cross-modal aligning problem as solutionsfor optimal transportation plans. Specifically, we leverage an optimaltransport kernel method to capture complementary information from multiplemodalities. The kernel embedding provides a non-linear transformation abilityto reproduce a kernel Hilbert space (RKHS), which reflects significance foreliminating the distributional modality gap. Moreover, we perceive the topologyinformation based on aligned representations to conduct bipartite graph pathreasoning. The newly achieved state-of-the-art performance on two publiclyavailable benchmark datasets, together with further visual analysis,demonstrate the superiority of TOT in capturing implicit cross-modal alignment.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域