Collaborative Machine Learning for Detecting Network Anomalies on the Edge

Abstract

Massive uses of the Internet of Things (IoT) and mobile terminals have brought revolutionary changes to existing network applications and raised the importance of edge networks to a new level. Edge network environments can exhibit extreme diversity and flexibility, calling novel network security and data privacy requirements for resource-constrained edge terminals. Therefore, Machine Learning (ML) and Deep Learning (DL) based network intrusion detection could be one feasible solution to analyse network traffic and protect them from attacks. The existing approaches require central entities to collect, store, and analyse client network traffic flows. This approach can bring severe data privacy concerns and lack flexibility or scalability to fulfil security requirements on edge networks. Moreover, edge network traffic flows present higher heterogeneity compared to traditional network environments, and data are scattered on massive edge terminals. These features merge challenges and call for novel designs for network intrusion detection in a more distributed manner.

In recent years, Federated Learning (FL) has emerged as a promising technique for deploying ML in distributed systems. Instead of forwarding data to central nodes, FL clients can train their models locally and share model updates with other participants, enabling knowledge exchange while preserving data privacy. This collaborative machine learning approach allows clients to contribute to a model that delivers more comprehensive performance.

However, applying FL for edge network anomaly detection can still face significant challenges that need to be addressed. For example, edge network endpoints such as IoT devices are often designed for specific tasks with various embedded applications. The patterns of their network traffic flows can vary, and the training samples on these clients may exhibit class imbalance and non-Independent and Identically Distributed (non-IID) characteristics. Furthermore, deploying ML models for network anomaly detection on IoTs is often constrained by the availability of local resources. Including restricted samples for training, limited computing power, and insufficient storage capacity. Edge devices may be unable to support the cost of training and deploying sophisticated ML models, such as Deep Neural Networks (DNN). Moreover, traditional FL structures are centralized, assuming clients can connect to a central server for model exchanges. However, edge networks, which often involve multiple wireless communication techniques, present higher complexity and mobility. The flexibility and unpredictability of edge network connectivity necessitate innovative FL designs tailored to these challenges.

This thesis aims to address the aforementioned challenges and research gaps in collaborative machine learning for anomaly detection on edge network terminals. To coordinate model training among heterogeneous edge terminals with non-IID training samples, a novel FL framework is introduced, which incorporates clustering approaches during the model aggregation phase to enhance the performance of the aggregated model. Meanwhile, this thesis also introduces a hardware test platform to simulate real-world IoT environments with limited resources. To address the constraints of training data on IoT devices, a federated transfer learning approach was proposed, reusing instances from public datasets to assist localized model training. Furthermore, to meet the network security requirements of mobile edge terminals and unmanned vehicles, a novel decentralized FL architecture is proposed and implemented to detect wireless network intrusions in environments characterized by high flexibility and uncertainty. In summary, this thesis presents innovative solutions to enhance edge network security across multiple dimensions, making significant contributions to the development of more secure and reliable edge network systems.