Controlled Descent Training

Viktor AnderssonBal\'azs VargaVincent SzolnokyAndreas Syr\'enRebecka J\"ornstenBal\'azs Kulcs\'ar

Mar 2023

摘要

In this work, a novel and model-based artificial neural network (ANN)training method is developed supported by optimal control theory. The methodaugments training labels in order to robustly guarantee training lossconvergence and improve training convergence rate. Dynamic label augmentationis proposed within the framework of gradient descent training where theconvergence of training loss is controlled. First, we capture the trainingbehavior with the help of empirical Neural Tangent Kernels (NTK) and borrowtools from systems and control theory to analyze both the local and globaltraining dynamics (e.g. stability, reachability). Second, we propose todynamically alter the gradient descent training mechanism via fictitious labelsas control inputs and an optimal state feedback policy. In this way, we enforcelocally $\mathcal{H}_2$ optimal and convergent training behavior. The novelalgorithm, \textit{Controlled Descent Training} (CDT), guarantees localconvergence. CDT unleashes new potentials in the analysis, interpretation, anddesign of ANN architectures. The applicability of the method is demonstrated onstandard regression and classification problems.

展开全部

图表提取

论文十问

参考文献

被引用

提问与回答

暂无人提供速读十问回答

被引用

笔记

问答

领域