Federated ensemble model-based reinforcement learning in edge computing
dc.contributor.author | Wang, J | |
dc.contributor.author | Hu, J | |
dc.contributor.author | Mills, J | |
dc.contributor.author | Min, G | |
dc.contributor.author | Xia, M | |
dc.contributor.author | Georgalas, N | |
dc.date.accessioned | 2023-04-21T11:01:57Z | |
dc.date.issued | 2023-04-05 | |
dc.date.updated | 2023-04-21T08:58:41Z | |
dc.description.abstract | Federated learning (FL) is a privacy-preserving distributed machine learning paradigm that enables collaborative training among geographically distributed and heterogeneous devices without gathering their data. Extending FL beyond the supervised learning models, federated reinforcement learning (FRL) was proposed to handle sequential decision-making problems in edge computing systems. However, the existing FRL algorithms directly combine model-free RL with FL, thus often leading to high sample complexity and lacking theoretical guarantees. To address the challenges, we propose a novel FRL algorithm that effectively incorporates modelbased RL and ensemble knowledge distillation into FL for the first time. Specifically, we utilise FL and knowledge distillation to create an ensemble of dynamics models for clients, and then train the policy by solely using the ensemble model without interacting with the environment. Furthermore, we theoretically prove that the monotonic improvement of the proposed algorithm is guaranteed. The extensive experimental results demonstrate that our algorithm obtains much higher sample efficiency compared to classic model-free FRL algorithms in the challenging continuous control benchmark environments under edge computing settings. The results also highlight the significant impact of heterogeneous client data and local model update steps on the performance of FRL, validating the insights obtained from our theoretical analysis. | en_GB |
dc.description.sponsorship | European Union Horizon 2020 | en_GB |
dc.description.sponsorship | Royal Society | en_GB |
dc.description.sponsorship | Engineering and Physical Sciences Research Council (EPSRC) | en_GB |
dc.description.sponsorship | UK Research and Innovation | en_GB |
dc.identifier.citation | Vol. 34 (6), pp. 1848 - 1859 | en_GB |
dc.identifier.doi | https://doi.org/10.1109/tpds.2023.3264480 | |
dc.identifier.grantnumber | 101008297 | en_GB |
dc.identifier.grantnumber | IEC/NSFC/211460 | en_GB |
dc.identifier.grantnumber | EP/X019160/1 | en_GB |
dc.identifier.grantnumber | EP/X038866/1 | en_GB |
dc.identifier.uri | http://hdl.handle.net/10871/132961 | |
dc.identifier | ORCID: 0000-0001-5406-8420 (Hu, Jia) | |
dc.identifier | ORCID: 0000-0003-1395-7314 (Min, Geyong) | |
dc.language.iso | en | en_GB |
dc.publisher | Institute of Electrical and Electronics Engineers (IEEE) | en_GB |
dc.rights | © 2023, IEEE. This version is made available under the CC-BY 4.0 license: https://creativecommons.org/licenses/by/4.0/ | en_GB |
dc.subject | Edge Computing | en_GB |
dc.subject | distributed machine learning | en_GB |
dc.subject | federated learning | en_GB |
dc.subject | deep reinforcement learning | en_GB |
dc.title | Federated ensemble model-based reinforcement learning in edge computing | en_GB |
dc.type | Article | en_GB |
dc.date.available | 2023-04-21T11:01:57Z | |
dc.identifier.issn | 1045-9219 | |
dc.description | This is the author accepted manuscript. The final version is available from the IEEE via the DOI in this record | en_GB |
dc.identifier.eissn | 1558-2183 | |
dc.identifier.journal | IEEE Transactions on Parallel and Distributed Systems | en_GB |
dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | en_GB |
rioxxterms.version | AM | en_GB |
rioxxterms.licenseref.startdate | 2023-01-01 | |
rioxxterms.type | Journal Article/Review | en_GB |
refterms.dateFCD | 2023-04-21T10:57:43Z | |
refterms.versionFCD | AM | |
refterms.dateFOA | 2023-04-21T11:01:59Z | |
refterms.panel | B | en_GB |
refterms.dateFirstOnline | 2023-04-05 |
Files in this item
This item appears in the following Collection(s)
Except where otherwise noted, this item's licence is described as © 2023, IEEE. This version is made available under the CC-BY 4.0 license: https://creativecommons.org/licenses/by/4.0/