Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval

Yi XieHuaidong ZhangXuemiao XuJianqing ZhuShengfeng He

Mar 2023

摘要

Previous Knowledge Distillation based efficient image retrieval methodsemploys a lightweight network as the student model for fast inference. However,the lightweight student model lacks adequate representation capacity foreffective knowledge imitation during the most critical early training period,causing final performance degeneration. To tackle this issue, we propose aCapacity Dynamic Distillation framework, which constructs a student model witheditable representation capacity. Specifically, the employed student model isinitially a heavy model to fruitfully learn distilled knowledge in the earlytraining epochs, and the student model is gradually compressed during thetraining. To dynamically adjust the model capacity, our dynamic frameworkinserts a learnable convolutional layer within each residual block in thestudent model as the channel importance indicator. The indicator is optimizedsimultaneously by the image retrieval loss and the compression loss, and aretrieval-guided gradient resetting mechanism is proposed to release thegradient conflict. Extensive experiments show that our method has superiorinference speed and accuracy, e.g., on the VeRi-776 dataset, given theResNet101 as a teacher, our method saves 67.13% model parameters and 65.67%FLOPs (around 24.13% and 21.94% higher than state-of-the-arts) withoutsacrificing accuracy (around 2.11% mAP higher than state-of-the-arts).

展开全部

图表提取