-
Guaranteed Approximation Bounds for Mixed-Precision Neural Operators
Authors:
Renbo Tu,
Colin White,
Jean Kossaifi,
Boris Bonev,
Nikola Kovachki,
Gennady Pekhimenko,
Kamyar Azizzadenesheli,
Anima Anandkumar
Abstract:
Neural operators, such as Fourier Neural Operators (FNO), form a principled approach for learning solution operators for PDEs and other mappings between function spaces. However, many real-world problems require high-resolution training data, and the training time and limited GPU memory pose big barriers. One solution is to train neural operators in mixed precision to reduce the memory requirement…
▽ More
Neural operators, such as Fourier Neural Operators (FNO), form a principled approach for learning solution operators for PDEs and other mappings between function spaces. However, many real-world problems require high-resolution training data, and the training time and limited GPU memory pose big barriers. One solution is to train neural operators in mixed precision to reduce the memory requirement and increase training speed. However, existing mixed-precision training techniques are designed for standard neural networks, and we find that their direct application to FNO leads to numerical overflow and poor memory efficiency. Further, at first glance, it may appear that mixed precision in FNO will lead to drastic accuracy degradation since reducing the precision of the Fourier transform yields poor results in classical numerical solvers. We show that this is not the case; in fact, we prove that reducing the precision in FNO still guarantees a good approximation bound, when done in a targeted manner. Specifically, we build on the intuition that neural operator learning inherently induces an approximation error, arising from discretizing the infinite-dimensional ground-truth input function, implying that training in full precision is not needed. We formalize this intuition by rigorously characterizing the approximation and precision errors of FNO and bounding these errors for general input functions. We prove that the precision error is asymptotically comparable to the approximation error. Based on this, we design a simple method to optimize the memory-intensive half-precision tensor contractions by greedily finding the optimal contraction order. Through extensive experiments on different state-of-the-art neural operators, datasets, and GPUs, we demonstrate that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
△ Less
Submitted 5 May, 2024; v1 submitted 27 July, 2023;
originally announced July 2023.
-
Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions
Authors:
Junyang Cai,
Khai-Nguyen Nguyen,
Nishant Shrestha,
Aidan Good,
Ruisen Tu,
Xin Yu,
Shandian Zhe,
Thiago Serra
Abstract:
One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the li…
▽ More
One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy. But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy. This drop plausibly reflects a loss in model complexity, which we aim to avoid. In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network, and consequently reduces the expected maximum number of linear regions based on the architecture. We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions and our proposed bound for the maximum number. Conversely, we find out that selecting the sparsity across layers to maximize our bound very often improves accuracy in comparison to pruning as much with the same sparsity in all layers, thereby providing us guidance on where to prune.
△ Less
Submitted 19 January, 2023;
originally announced January 2023.
-
Effective and Acceptable Eco-Driving Guidance for Human-Driving Vehicles: A Review
Authors:
Ran Tu,
Junshi Xu
Abstract:
Ecodriving guidance includes courses or suggestions for human drivers to improve driving behaviour, reducing energy use and emissions. This paper presents a systematic review of existing eco-driving guidance studies and identifies challenges to tackle in the future. A standard agreement on the guidance design has not been reached, leading to difficulties in designing and implementing eco-driving g…
▽ More
Ecodriving guidance includes courses or suggestions for human drivers to improve driving behaviour, reducing energy use and emissions. This paper presents a systematic review of existing eco-driving guidance studies and identifies challenges to tackle in the future. A standard agreement on the guidance design has not been reached, leading to difficulties in designing and implementing eco-driving guidance for human drivers. Both static and dynamic guidance systems have a great variety of guidance results. In addition, the influencing factors, such as the suggestion content, the displaying methods, and drivers socio-demographic characteristics, have opposite effects on the guidance result across studies, while the reason has not been revealed. Drivers motivation to practice eco behaviour, especially long-term, is overlooked. Besides, the relationship between users acceptance and system effectiveness is still unclear. Adaptive driving suggestions based on drivers habits can improve the effectiveness, while this field is under investigation.
△ Less
Submitted 27 March, 2022;
originally announced March 2022.
-
Multi-Objective Eco-Routing for Dynamic Control of Connected & Automated Vehicles
Authors:
Shadi Djavadian,
Ran Tu,
Bilal Farooq,
Marianne Hatzopoulou
Abstract:
The advent of intelligent vehicles that can communicate with infrastructure as well as automate the movement provides a range of new options to address key urban traffic issues such as congestion and pollution, without the need for centralized traffic control. Furthermore, the advances in the information, communication, and sensing technologies have provided access to real-time traffic and emissio…
▽ More
The advent of intelligent vehicles that can communicate with infrastructure as well as automate the movement provides a range of new options to address key urban traffic issues such as congestion and pollution, without the need for centralized traffic control. Furthermore, the advances in the information, communication, and sensing technologies have provided access to real-time traffic and emission data. Leveraging these advancements, a dynamic multi-objective eco-routing strategy for connected & automated vehicles (CAVs) is proposed and implemented in a distributed traffic management system. It is applied to the road network of downtown Toronto in an in-house agent-based traffic simulation platform. The performance of the proposed system is compared to various single-objective optimizations. Simulation results show the significance of incorporating real-time emission and traffic state into the dynamic routing, along with considering the expected delays at the downstream intersections. The proposed multi-objective eco-routing has the potential of reducing GHG and NOx emissions by 43% and 18.58%, respectively, while reducing average travel time by 40%.
△ Less
Submitted 8 October, 2020; v1 submitted 2 May, 2020;
originally announced May 2020.
-
Eigenvalue estimates for submanifolds in Hadamard manifolds and product manifolds $N\times\mathbb{R}$
Authors:
Jing Mao,
Rong-Qiang Tu,
Kai Zeng
Abstract:
In this paper, we investigate submanifolds with locally bounded mean curvature in Hadamard manifolds, product manifolds $N\times\mathbb{R}$, submanifolds with bounded $\varphi$-mean curvature in the hyperbolic space, and successfully give lower bounds for the weighted fundamental tone and the first eigenvalue of the $p$-Laplacian.
In this paper, we investigate submanifolds with locally bounded mean curvature in Hadamard manifolds, product manifolds $N\times\mathbb{R}$, submanifolds with bounded $\varphi$-mean curvature in the hyperbolic space, and successfully give lower bounds for the weighted fundamental tone and the first eigenvalue of the $p$-Laplacian.
△ Less
Submitted 4 September, 2019; v1 submitted 24 May, 2018;
originally announced May 2018.
