RMSProp

RMSProp is an unpublished adaptive learning rate optimizer proposed by Geoff Hinton. The motivation is that the magnitude of gradients can differ for different weights, and can change during learning, making it hard to choose a single global learning rate. RMSProp tackles this by keeping a moving average of the squared gradient and adjusting the weight updates by this magnitude. The gradient updates are performed as:$$E\left[g^{2}\right]_{t} = \gamma E\left[g^{2}\right]_{t-1} + \left(1 - \gamma\right) g^{2}_{t}$$$$\theta_{t+1} = \theta_{t} - \frac{\eta}{\sqrt{E\left[g^{2}\right]_{t} + \epsilon}}g_{t}$$Hinton suggests $\gamma=0.9$, with a good default for $\eta$ as $0.001$.Image: Alec Radford

相关学科： AdaGradAdaDeltaAdaMaxNADAMAMSGradSGDAdaBoundAdamAdamWSGD with Momentum

学科讨论

暂无讨论内容，你可以

学科管理组

暂无学科课代表，你可以申请成为课代表

重要学者

Yoshua Bengio

429868 被引用，1063 篇论文

Jiawei Han

121361 被引用，1269 篇论文

Léon Bottou

98650 被引用，174 篇论文

Richard Socher

81897 被引用，249 篇论文

Francisco Herrera

79311 被引用，1072 篇论文

Quoc V. Le

73373 被引用，205 篇论文

Richard S. Sutton

61078 被引用，239 篇论文

Hua Wang

43448 被引用，637 篇论文

Noam Shazeer

37095 被引用，61 篇论文

Wei Liu

35988 被引用，1482 篇论文