SmartBERT: A Promotion of Dynamic Early Exiting Mechanism for Accelerating BERT Inference

Boren HuYun ZhuJiacheng LiSiliang Tang

Mar 2023

摘要

Dynamic early exiting has been proven to improve the inference speed of thepre-trained language model like BERT. However, all samples must go through allconsecutive layers before early exiting and more complex samples usually gothrough more layers, which still exists redundant computation. In this paper,we propose a novel dynamic early exiting combined with layer skipping for BERTinference named SmartBERT, which adds a skipping gate and an exiting operatorinto each layer of BERT. SmartBERT can adaptively skip some layers andadaptively choose whether to exit. Besides, we propose cross-layer contrastivelearning and combine it into our training phases to boost the intermediatelayers and classifiers which would be beneficial for early exiting. To keep theconsistent usage of skipping gates between training and inference phases, wepropose a hard weight mechanism during training phase. We conduct experimentson eight classification datasets of the GLUE benchmark. Experimental resultsshow that SmartBERT achieves 2-3x computation reduction with minimal accuracydrops compared with BERT and our method outperforms previous methods in bothefficiency and accuracy. Moreover, in some complex datasets like RTE and WNLI,we prove that the early exiting based on entropy hardly works, and the skippingmechanism is essential for reducing computation.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域