On the Role of Information in Distributed Learning

Today, most data consumed by machine learning algorithms is generated by the enormous amount of sensors and embedded devices like smartphones, cars, drones, which are geographically distributed and potentially concerned about privacy guarantees. In response to this, machine learning systems are expected to transition from centralized to more distributed solutions, in which the training and/or the inference processes are brought closer to the source of the data. In such a setup, the natural learning and inference loops must consider the communication aspect between the involved entities which collaboratively train and run the learning models. In this regard, this work explores the way the exchanged information can be compressed, represented and conveyed through the communication networks; the impacts of unreliable and constrained communication channels on the system outputs; and the fundamental trade-off between the amount of exchanged information and the final performance. Specifically, the analyses first focus on standard federated learning, which is a very popular distributed training technique, and then switch to multi-agent reinforcement learning, in which the underlying learning problem is a decision process. In the end, examples of applications to the optimization of wireless communications networks are also provided.