This website requires JavaScript.

When Do Curricula Work in Federated Learning?

Saeed VahidianSreevatsank KadaveruWoonjoon Baek ...+4 Bill Lin
Dec 2022
摘要
An oft-cited open problem of federated learning is the existence of dataheterogeneity at the clients. One pathway to understanding the drastic accuracydrop in federated learning is by scrutinizing the behavior of the clients' deepmodels on data with different levels of "difficulty", which has been leftunaddressed. In this paper, we investigate a different and rarely studieddimension of FL: ordered learning. Specifically, we aim to investigate howordered learning principles can contribute to alleviating the heterogeneityeffects in FL. We present theoretical analysis and conduct extensive empiricalstudies on the efficacy of orderings spanning three kinds of learning:curriculum, anti-curriculum, and random curriculum. We find that curriculumlearning largely alleviates non-IIDness. Interestingly, the more disparate thedata distributions across clients the more they benefit from ordered learning.We provide analysis explaining this phenomenon, specifically indicating howcurriculum training appears to make the objective landscape progressively lessconvex, suggesting fast converging iterations at the beginning of the trainingprocedure. We derive quantitative results of convergence for both convex andnonconvex objectives by modeling the curriculum training on federated devicesas local SGD with locally biased stochastic gradients. Also, inspired byordered learning, we propose a novel client selection technique that benefitsfrom the real-world disparity in the clients. Our proposed approach to clientselection has a synergic effect when applied together with ordered learning inFL.
展开全部
图表提取

暂无人提供速读十问回答

论文十问由沈向洋博士提出,鼓励大家带着这十个问题去阅读论文,用有用的信息构建认知模型。写出自己的十问回答,还有机会在当前页面展示哦。

Q1论文试图解决什么问题?
Q2这是否是一个新的问题?
Q3这篇文章要验证一个什么科学假设?
0
被引用
笔记
问答