Federated ensemble model-based reinforcement learning in edge computing
Wang, J; Hu, J; Mills, J; et al.Min, G; Xia, M; Georgalas, N
Date: 5 April 2023
Article
Journal
IEEE Transactions on Parallel and Distributed Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Publisher DOI
Abstract
Federated learning (FL) is a privacy-preserving distributed machine learning paradigm that enables collaborative training
among geographically distributed and heterogeneous devices without gathering their data. Extending FL beyond the supervised learning
models, federated reinforcement learning (FRL) was proposed to handle sequential ...
Federated learning (FL) is a privacy-preserving distributed machine learning paradigm that enables collaborative training
among geographically distributed and heterogeneous devices without gathering their data. Extending FL beyond the supervised learning
models, federated reinforcement learning (FRL) was proposed to handle sequential decision-making problems in edge computing
systems. However, the existing FRL algorithms directly combine model-free RL with FL, thus often leading to high sample complexity
and lacking theoretical guarantees. To address the challenges, we propose a novel FRL algorithm that effectively incorporates modelbased RL and ensemble knowledge distillation into FL for the first time. Specifically, we utilise FL and knowledge distillation to create
an ensemble of dynamics models for clients, and then train the policy by solely using the ensemble model without interacting with
the environment. Furthermore, we theoretically prove that the monotonic improvement of the proposed algorithm is guaranteed. The
extensive experimental results demonstrate that our algorithm obtains much higher sample efficiency compared to classic model-free
FRL algorithms in the challenging continuous control benchmark environments under edge computing settings. The results also highlight
the significant impact of heterogeneous client data and local model update steps on the performance of FRL, validating the insights
obtained from our theoretical analysis.
Computer Science
Faculty of Environment, Science and Economy
Item views 0
Full item downloads 0
Except where otherwise noted, this item's licence is described as © 2023, IEEE. This version is made available under the CC-BY 4.0 license: https://creativecommons.org/licenses/by/4.0/