This website requires JavaScript.

FedEval: A Holistic Evaluation Framework for Federated Learning

Di ChaiLeye WangLiu YangJunxue ZhangKai ChenQiang Yang
Nov 2020
Federated Learning (FL) has been widely accepted as the solution forprivacy-preserving machine learning without collecting raw data. While newtechnologies proposed in the past few years do evolve the FL area,unfortunately, the evaluation results presented in these works fall short inintegrity and are hardly comparable because of the inconsistent evaluationmetrics and experimental settings. In this paper, we propose a holisticevaluation framework for FL called FedEval, and present a benchmarking study onseven state-of-the-art FL algorithms. Specifically, we first introduce the coreevaluation taxonomy model, called FedEval-Core, which covers four essentialevaluation aspects for FL: Privacy, Robustness, Effectiveness, and Efficiency,with various well-defined metrics and experimental settings. Based on theFedEval-Core, we further develop an FL evaluation platform with standardizedevaluation settings and easy-to-use interfaces. We then provide an in-depthbenchmarking study between the seven well-known FL algorithms, includingFedSGD, FedAvg, FedProx, FedOpt, FedSTC, SecAgg, and HEAgg. We comprehensivelyanalyze the advantages and disadvantages of these algorithms and furtheridentify the suitable practical scenarios for different algorithms, which israrely done by prior work. Lastly, we excavate a set of take-away insights andfuture research directions, which are very helpful for researchers in the FLarea.