This website requires JavaScript.

Prototype-guided Cross-task Knowledge Distillation for Large-scale Models

Deng LiAming WuYahong HanQi Tian
Dec 2022
摘要
Recently, large-scale pre-trained models have shown their advantages in manytasks. However, due to the huge computational complexity and storagerequirements, it is challenging to apply the large-scale model to real scenes.A common solution is knowledge distillation which regards the large-scale modelas a teacher model and helps to train a small student model to obtain acompetitive performance. Cross-task Knowledge distillation expands theapplication scenarios of the large-scale pre-trained model. Existing knowledgedistillation works focus on directly mimicking the final prediction or theintermediate layers of the teacher model, which represent the global-levelcharacteristics and are task-specific. To alleviate the constraint of differentlabel spaces, capturing invariant intrinsic local object characteristics (suchas the shape characteristics of the leg and tail of the cattle and horse) playsa key role. Considering the complexity and variability of real scene tasks, wepropose a Prototype-guided Cross-task Knowledge Distillation (ProC-KD) approachto transfer the intrinsic local-level object knowledge of a large-scale teachernetwork to various task scenarios. First, to better transfer the generalizedknowledge in the teacher model in cross-task scenarios, we propose a prototypelearning module to learn from the essential feature representation of objectsin the teacher model. Secondly, for diverse downstream tasks, we propose atask-adaptive feature augmentation module to enhance the features of thestudent model with the learned generalization prototype features and guide thetraining of the student model to improve its generalization ability. Theexperimental results on various visual tasks demonstrate the effectiveness ofour approach for large-scale model cross-task knowledge distillation scenes.
展开全部
图表提取

暂无人提供速读十问回答

论文十问由沈向洋博士提出,鼓励大家带着这十个问题去阅读论文,用有用的信息构建认知模型。写出自己的十问回答,还有机会在当前页面展示哦。

Q1论文试图解决什么问题?
Q2这是否是一个新的问题?
Q3这篇文章要验证一个什么科学假设?
0
被引用
笔记
问答