Personalized Federated Learning for improving radar based precipitation nowcasting on heterogeneous areas
Authors:
Judith Sáinz-Pardo Díaz,
María Castrillo,
Juraj Bartok,
Ignacio Heredia Cachá,
Irina Malkin Ondík,
Ivan Martynovskyi,
Khadijeh Alibabaei,
Lisana Berberi,
Valentin Kozlov,
Álvaro López García
Abstract:
The increasing generation of data in different areas of life, such as the environment, highlights the need to explore new techniques for processing and exploiting data for useful purposes. In this context, artificial intelligence techniques, especially through deep learning models, are key tools to be used on the large amount of data that can be obtained, for example, from weather radars. In many…
▽ More
The increasing generation of data in different areas of life, such as the environment, highlights the need to explore new techniques for processing and exploiting data for useful purposes. In this context, artificial intelligence techniques, especially through deep learning models, are key tools to be used on the large amount of data that can be obtained, for example, from weather radars. In many cases, the information collected by these radars is not open, or belongs to different institutions, thus needing to deal with the distributed nature of this data. In this work, the applicability of a personalized federated learning architecture, which has been called adapFL, on distributed weather radar images is addressed. To this end, given a single available radar covering 400 km in diameter, the captured images are divided in such a way that they are disjointly distributed into four different federated clients. The results obtained with adapFL are analyzed in each zone, as well as in a central area covering part of the surface of each of the previously distributed areas. The ultimate goal of this work is to study the generalization capability of this type of learning technique for its extrapolation to use cases in which a representative number of radars is available, whose data can not be centralized due to technical, legal or administrative concerns. The results of this preliminary study indicate that the performance obtained in each zone with the adapFL approach allows improving the results of the federated learning approach, the individual deep learning models and the classical Continuity Tracking Radar Echoes by Correlation approach.
△ Less
Submitted 11 August, 2024;
originally announced August 2024.
Hazelcast Jet: Low-latency Stream Processing at the 99.99th Percentile
Authors:
Can Gencer,
Marko Topolnik,
Viliam Ďurina,
Emin Demirci,
Ensar B. Kahveci,
Ali Gürbüz Ondřej Lukáš,
József Bartók,
Grzegorz Gierlach,
František Hartman,
Ufuk Yılmaz,
Mehmet Doğan,
Mohamed Mandouh,
Marios Fragkoulis,
Asterios Katsifodimos
Abstract:
Jet is an open-source, high-performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet's purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast's In-memory Data Grid (IMDG): a set of high-performa…
▽ More
Jet is an open-source, high-performance, distributed stream processor built at Hazelcast during the last five years. Jet was engineered with millisecond latency on the 99.99th percentile as its primary design goal. Originally Jet's purpose was to be an execution engine that performs complex business logic on top of streams generated by Hazelcast's In-memory Data Grid (IMDG): a set of high-performance, in-memory, partitioned and replicated data structures. With time, Jet evolved into a full-fledged, scale-out stream processor that can handle out-of-order streams and exactly-once processing guarantees. Jet's end-to-end latency lies in the order of milliseconds, and its throughput in the order of millions of events per CPU-core. This paper presents main design decisions we made in order to maximize the performance per CPU-core, alongside lessons learned, and an empirical performance evaluation.
△ Less
Submitted 18 March, 2021;
originally announced March 2021.
