All4One: Symbiotic Neighbour Contrastive Learning via Self-Attention and Redundancy Reduction

Imanol G. EstepaIgnacio Saras\'uaBhalaji NagarajanPetia Radeva

Mar 2023

摘要

Nearest neighbour based methods have proved to be one of the most successfulself-supervised learning (SSL) approaches due to their high generalizationcapabilities. However, their computational efficiency decreases when more thanone neighbour is used. In this paper, we propose a novel contrastive SSLapproach, which we call All4One, that reduces the distance between neighbourrepresentations using ''centroids'' created through a self-attention mechanism.We use a Centroid Contrasting objective along with single Neighbour Contrastingand Feature Contrasting objectives. Centroids help in learning contextualinformation from multiple neighbours whereas the neighbour contrast enableslearning representations directly from the neighbours and the feature contrastallows learning representations unique to the features. This combinationenables All4One to outperform popular instance discrimination approaches bymore than 1% on linear classification evaluation for popular benchmark datasetsand obtains state-of-the-art (SoTA) results. Finally, we show that All4One isrobust towards embedding dimensionalities and augmentations, surpassing NNCLRand Barlow Twins by more than 5% on low dimensionality and weak augmentationsettings. The source code would be made available soon.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域