SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Amro AbbasKushal TirumalaD\'aniel SimigSurya GanguliAri S. Morcos

Mar 2023

摘要

Progress in machine learning has been driven in large part by massiveincreases in data. However, large web-scale datasets such as LAION are largelyuncurated beyond searches for exact duplicates, potentially leaving muchredundancy. Here, we introduce SemDeDup, a method which leverages embeddingsfrom pre-trained models to identify and remove semantic duplicates: data pairswhich are semantically similar, but not exactly identical. Removing semanticduplicates preserves performance and speeds up learning. Analyzing a subset ofLAION, we show that SemDeDup can remove 50% of the data with minimalperformance loss, effectively halving training time. Moreover, performanceincreases out of distribution. Also, analyzing language models trained on C4, apartially curated dataset, we show that SemDeDup improves over prior approacheswhile providing efficiency gains. SemDeDup provides an example of how simpleways of leveraging quality embeddings can be used to make models learn fasterwith less data.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域