This website requires JavaScript.

FedEval: A Holistic Evaluation Framework for Federated Learning

Di ChaiLeye WangLiu YangJunxue ZhangKai ChenQiang Yang
Nov 2020
摘要
Federated Learning (FL) has been widely accepted as the solution forprivacy-preserving machine learning without collecting raw data. While newtechnologies proposed in the past few years do evolve the FL area,unfortunately, the evaluation results presented in these works fall short inintegrity and are hardly comparable because of the inconsistent evaluationmetrics and experimental settings. In this paper, we propose a holisticevaluation framework for FL called FedEval, and present a benchmarking study onseven state-of-the-art FL algorithms. Specifically, we first introduce the coreevaluation taxonomy model, called FedEval-Core, which covers four essentialevaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency,with various well-defined metrics and experimental settings. Based on theFedEval-Core, we further develop an FL evaluation platform with standardizedevaluation settings and easy-to-use interfaces. We then provide an in-depthbenchmarking study between the seven well-known FL algorithms, includingFedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensivelyanalyze the advantages and disadvantages of these algorithms and furtheridentify the suitable practical scenarios for different algorithms, which israrely done by prior work. Lastly, we excavate a set of take-away insights andfuture research directions, which are very helpful for researchers in the FLarea.
展开全部
图表提取

暂无人提供速读十问回答

论文十问由沈向洋博士提出,鼓励大家带着这十个问题去阅读论文,用有用的信息构建认知模型。写出自己的十问回答,还有机会在当前页面展示哦。

Q1论文试图解决什么问题?
Q2这是否是一个新的问题?
Q3这篇文章要验证一个什么科学假设?