-
Spatio-Temporal Conformal Prediction for Power Outage Data
Authors:
Hanyang Jiang,
Yao Xie,
Feng Qiu
Abstract:
In recent years, increasingly unpredictable and severe global weather patterns have frequently caused long-lasting power outages. Building resilience, the ability to withstand, adapt to, and recover from major disruptions, has become crucial for the power industry. To enable rapid recovery, accurately predicting future outage numbers is essential. Rather than relying on simple point estimates, we…
▽ More
In recent years, increasingly unpredictable and severe global weather patterns have frequently caused long-lasting power outages. Building resilience, the ability to withstand, adapt to, and recover from major disruptions, has become crucial for the power industry. To enable rapid recovery, accurately predicting future outage numbers is essential. Rather than relying on simple point estimates, we analyze extensive quarter-hourly outage data and develop a graph conformal prediction method that delivers accurate prediction regions for outage numbers across the states for a time period. We demonstrate the effectiveness of this method through extensive numerical experiments in several states affected by extreme weather events that led to widespread outages.
△ Less
Submitted 25 November, 2024;
originally announced November 2024.
-
Hierarchical Spatio-Temporal Uncertainty Quantification for Distributed Energy Adoption
Authors:
Wenbin Zhou,
Shixiang Zhu,
Feng Qiu,
Xuan Wu
Abstract:
The rapid deployment of distributed energy resources (DER) has introduced significant spatio-temporal uncertainties in power grid management, necessitating accurate multilevel forecasting methods. However, existing approaches often produce overly conservative uncertainty intervals at individual spatial units and fail to properly capture uncertainties when aggregating predictions across different s…
▽ More
The rapid deployment of distributed energy resources (DER) has introduced significant spatio-temporal uncertainties in power grid management, necessitating accurate multilevel forecasting methods. However, existing approaches often produce overly conservative uncertainty intervals at individual spatial units and fail to properly capture uncertainties when aggregating predictions across different spatial scales. This paper presents a novel hierarchical spatio-temporal model based on the conformal prediction framework to address these challenges. Our approach generates circuit-level DER growth predictions and efficiently aggregates them to the substation level while maintaining statistical validity through a tailored non-conformity score. Applied to a decade of DER installation data from a local utility network, our method demonstrates superior performance over existing approaches, particularly in reducing prediction interval widths while maintaining coverage.
△ Less
Submitted 19 February, 2025; v1 submitted 18 November, 2024;
originally announced November 2024.
-
Characterization of point-source transient events with a rolling-shutter compressed sensing system
Authors:
Frank Qiu,
Joshua Michalenko,
Lilian K. Casias,
Cameron J. Radosevich,
Jon Slater,
Eric A. Shields
Abstract:
Point-source transient events (PSTEs) - optical events that are both extremely fast and extremely small - pose several challenges to an imaging system. Due to their speed, accurately characterizing such events often requires detectors with very high frame rates. Due to their size, accurately detecting such events requires maintaining coverage over an extended field-of-view, often through the use o…
▽ More
Point-source transient events (PSTEs) - optical events that are both extremely fast and extremely small - pose several challenges to an imaging system. Due to their speed, accurately characterizing such events often requires detectors with very high frame rates. Due to their size, accurately detecting such events requires maintaining coverage over an extended field-of-view, often through the use of imaging focal plane arrays (FPA) with a global shutter readout. Traditional imaging systems that meet these requirements are costly in terms of price, size, weight, power consumption, and data bandwidth, and there is a need for cheaper solutions with adequate temporal and spatial coverage. To address these issues, we develop a novel compressed sensing algorithm adapted to the rolling shutter readout of an imaging system. This approach enables reconstruction of a PSTE signature at the sampling rate of the rolling shutter, offering a 1-2 order of magnitude temporal speedup and a proportional reduction in data bandwidth. We present empirical results demonstrating accurate recovery of PSTEs using measurements that are spatially undersampled by a factor of 25, and our simulations show that, relative to other compressed sensing algorithms, our algorithm is both faster and yields higher quality reconstructions. We also present theoretical results characterizing our algorithm and corroborating simulations. The potential impact of our work includes the development of much faster, cheaper sensor solutions for PSTE detection and characterization.
△ Less
Submitted 11 February, 2025; v1 submitted 29 August, 2024;
originally announced August 2024.
-
Assessing Electricity Service Unfairness with Transfer Counterfactual Learning
Authors:
Song Wei,
Xiangrui Kong,
Alinson Santos Xavier,
Shixiang Zhu,
Yao Xie,
Feng Qiu
Abstract:
Energy justice is a growing area of interest in interdisciplinary energy research. However, identifying systematic biases in the energy sector remains challenging due to confounding variables, intricate heterogeneity in counterfactual effects, and limited data availability. First, this paper demonstrates how one can evaluate counterfactual unfairness in a power system by analyzing the average caus…
▽ More
Energy justice is a growing area of interest in interdisciplinary energy research. However, identifying systematic biases in the energy sector remains challenging due to confounding variables, intricate heterogeneity in counterfactual effects, and limited data availability. First, this paper demonstrates how one can evaluate counterfactual unfairness in a power system by analyzing the average causal effect of a specific protected attribute. Subsequently, we use subgroup analysis to handle model heterogeneity and introduce a novel method for estimating counterfactual unfairness based on transfer learning, which helps to alleviate the data scarcity in each subgroup. In our numerical analysis, we apply our method to a unique large-scale customer-level power outage data set and investigate the counterfactual effect of demographic factors, such as income and age of the population, on power outage durations. Our results indicate that low-income and elderly-populated areas consistently experience longer power outages under both daily and post-disaster operations, and such discrimination is exacerbated under severe conditions. These findings suggest a widespread, systematic issue of injustice in the power service systems and emphasize the necessity for focused interventions in disadvantaged communities.
△ Less
Submitted 24 January, 2024; v1 submitted 4 October, 2023;
originally announced October 2023.
-
Random Projections of Sparse Adjacency Matrices
Authors:
Frank Qiu
Abstract:
We analyze a random projection method for adjacency matrices, studying its utility in representing sparse graphs. We show that these random projections retain the functionality of their underlying adjacency matrices while having extra properties that make them attractive as dynamic graph representations. In particular, they can represent graphs of different sizes and vertex sets in the same space,…
▽ More
We analyze a random projection method for adjacency matrices, studying its utility in representing sparse graphs. We show that these random projections retain the functionality of their underlying adjacency matrices while having extra properties that make them attractive as dynamic graph representations. In particular, they can represent graphs of different sizes and vertex sets in the same space, allowing for the aggregation and manipulation of graphs in a unified manner. We also provide results on how the size of the projections need to scale in order to preserve accurate graph operations, showing that the size of the projections can scale linearly with the number of vertices while accurately retaining first-order graph information. We conclude by characterizing our random projection as a distance-preserving map of adjacency matrices analogous to the usual Johnson-Lindenstrauss map.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Tensor Products and Hyperdimensional Computing
Authors:
Frank Qiu
Abstract:
Following up on a previous analysis of graph embeddings, we generalize and expand some results to the general setting of vector symbolic architectures (VSA) and hyperdimensional computing (HDC). Importantly, we explore the mathematical relationship between superposition, orthogonality, and tensor product. We establish the tensor product representation as the central representation, with a suite of…
▽ More
Following up on a previous analysis of graph embeddings, we generalize and expand some results to the general setting of vector symbolic architectures (VSA) and hyperdimensional computing (HDC). Importantly, we explore the mathematical relationship between superposition, orthogonality, and tensor product. We establish the tensor product representation as the central representation, with a suite of unique properties. These include it being the most general and expressive representation, as well as being the most compressed representation that has errorrless unbinding and detection.
△ Less
Submitted 20 May, 2023; v1 submitted 17 May, 2023;
originally announced May 2023.
-
Convolutional Non-homogeneous Poisson Process with Application to Wildfire Risk Quantification for Power Delivery Networks
Authors:
Guanzhou Wei,
Feng Qiu,
Xiao Liu
Abstract:
The current projection shows that much of the continental U.S. will have significantly hotter and drier days in the following decades, leading to more wildfire hazards that threaten the safety of power grid. Unfortunately, the U.S. power industry is not well prepared and still predominantly relies on empirical fire indices which do not consider the full spectrum of dynamic environmental factors. T…
▽ More
The current projection shows that much of the continental U.S. will have significantly hotter and drier days in the following decades, leading to more wildfire hazards that threaten the safety of power grid. Unfortunately, the U.S. power industry is not well prepared and still predominantly relies on empirical fire indices which do not consider the full spectrum of dynamic environmental factors. This paper proposes a new spatio-temporal point process model, Convolutional Non-homogeneous Poisson Process (cNHPP), to quantify wildfire risks for power delivery networks. The proposed model captures both the current short-term and cumulative long-term effects of covariates on wildfire risks, and the spatio-temporal dependency among different segments of the power delivery network. The computation and interpretation of the intensity function are thoroughly investigated, and the connection between cNHPP and Recurrent Neural Network is also discussed. We apply the proposed approach to estimate wildfire risks on major transmission lines in California, utilizing historical fire data, meteorological and vegetation data obtained from the National Oceanic and Atmospheric Administration and National Aeronautics and Space Administration. Comparison studies are performed to show the applicability and predictive capability of the proposed approach. Useful insights are obtained that potentially enhance power grid resilience against wildfires.
△ Less
Submitted 30 December, 2022;
originally announced January 2023.
-
Commutativity and Disentanglement from the Manifold Perspective
Authors:
Frank Qiu
Abstract:
In this paper, we interpret disentanglement as the discovery of local charts of the data manifold and trace how this definition naturally leads to an equivalent condition for disentanglement: commutativity between factors of variation. We study the impact of this manifold framework to two classes of problems: learning matrix exponential operators and compressing data-generating models. In each pro…
▽ More
In this paper, we interpret disentanglement as the discovery of local charts of the data manifold and trace how this definition naturally leads to an equivalent condition for disentanglement: commutativity between factors of variation. We study the impact of this manifold framework to two classes of problems: learning matrix exponential operators and compressing data-generating models. In each problem, the manifold perspective yields interesting results about the feasibility and fruitful approaches their solutions. We also link our manifold framework to two other common disentanglement paradigms: group theoretic and probabilistic approaches to disentanglement. In each case, we show how these frameworks can be merged with our manifold perspective. Importantly, we recover commutativity as a central property in both alternative frameworks, further highlighting its importance in disentanglement.
△ Less
Submitted 16 December, 2023; v1 submitted 14 October, 2022;
originally announced October 2022.
-
Graph Embeddings via Tensor Products and Approximately Orthonormal Codes
Authors:
Frank Qiu
Abstract:
We propose a dynamic graph representation method, showcasing its rich representational capacity and establishing some of its theoretical properties. Our representation falls under the bind-and-sum approach in hyperdimensional computing (HDC), and we show that the tensor product is the most general binding operation that respects the superposition principle employed in HDC. We also establish some p…
▽ More
We propose a dynamic graph representation method, showcasing its rich representational capacity and establishing some of its theoretical properties. Our representation falls under the bind-and-sum approach in hyperdimensional computing (HDC), and we show that the tensor product is the most general binding operation that respects the superposition principle employed in HDC. We also establish some precise results characterizing the behavior of our method, including a memory vs. size analysis of how our representation's size must scale with the number of edges in order to retain accurate graph operations. True to its HDC roots, we also compare our graph representation to another typical HDC representation, the Hadamard-Rademacher scheme, showing that these two graph representations have the same memory-capacity scaling. We establish a link to adjacency matrices, showing that our method is a pseudo-orthogonal generalization of adjacency matrices. In light of this, we briefly discuss its applications toward a dynamic compressed representation of large sparse graphs.
△ Less
Submitted 3 June, 2023; v1 submitted 18 August, 2022;
originally announced August 2022.
-
Memory and Capacity of Graph Embedding Methods
Authors:
Frank Qiu
Abstract:
THIS PAPER IS NOW DEFUNCT: Check out "Graph Embeddings via Tensor Products and Approximately Orthonormal Codes", where it has been combined into one paper.
THIS PAPER IS NOW DEFUNCT: Check out "Graph Embeddings via Tensor Products and Approximately Orthonormal Codes", where it has been combined into one paper.
△ Less
Submitted 13 May, 2023; v1 submitted 18 August, 2022;
originally announced August 2022.
-
Spatio-Temporal Wildfire Prediction using Multi-Modal Data
Authors:
Chen Xu,
Yao Xie,
Daniel A. Zuniga Vazquez,
Rui Yao,
Feng Qiu
Abstract:
Due to severe societal and environmental impacts, wildfire prediction using multi-modal sensing data has become a highly sought-after data-analytical tool by various stakeholders (such as state governments and power utility companies) to achieve a more informed understanding of wildfire activities and plan preventive measures. A desirable algorithm should precisely predict fire risk and magnitude…
▽ More
Due to severe societal and environmental impacts, wildfire prediction using multi-modal sensing data has become a highly sought-after data-analytical tool by various stakeholders (such as state governments and power utility companies) to achieve a more informed understanding of wildfire activities and plan preventive measures. A desirable algorithm should precisely predict fire risk and magnitude for a location in real time. In this paper, we develop a flexible spatio-temporal wildfire prediction framework using multi-modal time series data. We first predict the wildfire risk (the chance of a wildfire event) in real-time, considering the historical events using discrete mutually exciting point process models. Then we further develop a wildfire magnitude prediction set method based on the flexible distribution-free time-series conformal prediction (CP) approach. Theoretically, we prove a risk model parameter recovery guarantee, as well as coverage and set size guarantees for the CP sets. Through extensive real-data experiments with wildfire data in California, we demonstrate the effectiveness of our methods, as well as their flexibility and scalability in large regions.
△ Less
Submitted 10 October, 2023; v1 submitted 26 July, 2022;
originally announced July 2022.
-
Quantifying Grid Resilience Against Extreme Weather Using Large-Scale Customer Power Outage Data
Authors:
Shixiang Zhu,
Rui Yao,
Yao Xie,
Feng Qiu,
Yueming,
Qiu,
Xuan Wu
Abstract:
In recent decades, the weather around the world has become more irregular and extreme, often causing large-scale extended power outages. Resilience -- the capability of withstanding, adapting to, and recovering from a large-scale disruption -- has become a top priority for the power sector. However, the understanding of power grid resilience still stays on the conceptual level mostly or focuses on…
▽ More
In recent decades, the weather around the world has become more irregular and extreme, often causing large-scale extended power outages. Resilience -- the capability of withstanding, adapting to, and recovering from a large-scale disruption -- has become a top priority for the power sector. However, the understanding of power grid resilience still stays on the conceptual level mostly or focuses on particular components, yielding no actionable results or revealing few insights on the system level. This study provides a quantitatively measurable definition of power grid resilience, using a statistical model inspired by patterns observed from data and domain knowledge. We analyze a large-scale quarter-hourly historical electricity customer outage data and the corresponding weather records, and draw connections between the model and industry resilience practice. We showcase the resilience analysis using three major service territories on the east coast of the United States. Our analysis suggests that cumulative weather effects play a key role in causing immediate, sustained outages, and these outages can propagate and cause secondary outages in neighboring areas. The proposed model also provides some interesting insights into grid resilience enhancement planning. For example, our simulation results indicate that enhancing the power infrastructure in a small number of critical locations can reduce nearly half of the number of customer power outages in Massachusetts. In addition, we have shown that our model achieves promising accuracy in predicting the progress of customer power outages throughout extreme weather events, which can be very valuable for system operators and federal agencies to prepare disaster response.
△ Less
Submitted 4 September, 2022; v1 submitted 20 September, 2021;
originally announced September 2021.
-
Solar Radiation Ramping Events Modeling Using Spatio-temporal Point Processes
Authors:
Minghe Zhang,
Chen Xu,
Andy Sun,
Feng Qiu,
Yao Xie
Abstract:
Modeling and predicting solar events, particularly the solar ramping event, is critical for improving situational awareness for solar power generation systems. It has been acknowledged that weather conditions such as temperature, humidity, and cloud density can significantly impact the emergence and position of solar ramping events. As a result, modeling these events with complex spatio-temporal c…
▽ More
Modeling and predicting solar events, particularly the solar ramping event, is critical for improving situational awareness for solar power generation systems. It has been acknowledged that weather conditions such as temperature, humidity, and cloud density can significantly impact the emergence and position of solar ramping events. As a result, modeling these events with complex spatio-temporal correlations is highly challenging. To tackle the question, we adopt a novel spatio-temporal categorical point process model, which intuitively and effectively addresses correlation and interaction among ramping events. We demonstrate the interpretability and predictive power of our model on extensive real-data experiments.
△ Less
Submitted 16 June, 2022; v1 submitted 26 January, 2021;
originally announced January 2021.
-
Deep Active Learning for Solvability Prediction in Power Systems
Authors:
Yichen Zhang,
Jianzhe Liu,
Feng Qiu,
Tianqi Hong,
Rui Yao
Abstract:
Traditional methods for solvability region analysis can only have inner approximations with inconclusive conservatism. Machine learning methods have been proposed to approach the real region. In this letter, we propose a deep active learning framework for power system solvability prediction. Compared with the passive learning methods where the training is performed after all instances are labeled,…
▽ More
Traditional methods for solvability region analysis can only have inner approximations with inconclusive conservatism. Machine learning methods have been proposed to approach the real region. In this letter, we propose a deep active learning framework for power system solvability prediction. Compared with the passive learning methods where the training is performed after all instances are labeled, the active learning selects most informative instances to be label and therefore significantly reduce the size of labeled dataset for training. In the active learning framework, the acquisition functions, which correspond to different sampling strategies, are defined in terms of the on-the-fly posterior probability from the classifier. The IEEE 39-bus system is employed to validate the proposed framework, where a two-dimensional case is illustrated to visualize the effectiveness of the sampling method followed by the full-dimensional numerical experiments.
△ Less
Submitted 22 December, 2020; v1 submitted 26 July, 2020;
originally announced July 2020.
-
Online detection of cascading change-points
Authors:
Rui Zhang,
Yao Xie,
Rui Yao,
Feng Qiu
Abstract:
We propose an online detection procedure for cascading failures in the network from sequential data, which can be modeled as multiple correlated change-points happening during a short period. We consider a temporal diffusion network model to capture the temporal dynamic structure of multiple change-points and develop a sequential Shewhart procedure based on the generalized likelihood ratio statist…
▽ More
We propose an online detection procedure for cascading failures in the network from sequential data, which can be modeled as multiple correlated change-points happening during a short period. We consider a temporal diffusion network model to capture the temporal dynamic structure of multiple change-points and develop a sequential Shewhart procedure based on the generalized likelihood ratio statistics based on the diffusion network model assuming unknown post-change distribution parameters. We also tackle the computational complexity posed by the unknown propagation. Numerical experiments demonstrate the good performance for detecting cascade failures.
△ Less
Submitted 5 February, 2021; v1 submitted 28 October, 2019;
originally announced November 2019.
-
Ensemble Learning Based Convex Approximation of Three-Phase Power Flow
Authors:
Ren Hu,
Qifeng Li,
Feng Qiu
Abstract:
Though the convex optimization has been widely used in power systems, it still cannot guarantee to yield a tight (accurate) solution to some problems. To mitigate this issue, this paper proposes an ensemble learning based convex approximation for AC power flow equations that differs from the existing convex relaxations. The proposed approach is based on quadratic power flow equations in rectangula…
▽ More
Though the convex optimization has been widely used in power systems, it still cannot guarantee to yield a tight (accurate) solution to some problems. To mitigate this issue, this paper proposes an ensemble learning based convex approximation for AC power flow equations that differs from the existing convex relaxations. The proposed approach is based on quadratic power flow equations in rectangular coordinates and it can be used in both balanced and unbalanced three-phase power networks. To develop this data-driven convex approximation of power flows, the polynomial regression (PR) is first deployed as a basic learner to fit convex relationships between the independent and dependent variables. Then, ensemble learning algorithms such as gradient boosting (GB) and bagging are introduced to combine learners to boost model performance. Based on the learned convex approximation of power flows, optimal power flow (OPF) is formulated as a convex quadratic programming problem. The simulation results on IEEE standard cases show that, in the context of solving OPF, the proposed data-driven convex approximation outperforms the conventional SDP relaxation in both accuracy and computational efficiency, especially in the cases that the conventional SDP relaxation fails.
△ Less
Submitted 15 May, 2020; v1 submitted 12 September, 2019;
originally announced September 2019.
-
Learning to Solve Large-Scale Security-Constrained Unit Commitment Problems
Authors:
Alinson S. Xavier,
Feng Qiu,
Shabbir Ahmed
Abstract:
Security-Constrained Unit Commitment (SCUC) is a fundamental problem in power systems and electricity markets. In practical settings, SCUC is repeatedly solved via Mixed-Integer Linear Programming, sometimes multiple times per day, with only minor changes in input data. In this work, we propose a number of machine learning (ML) techniques to effectively extract information from previously solved i…
▽ More
Security-Constrained Unit Commitment (SCUC) is a fundamental problem in power systems and electricity markets. In practical settings, SCUC is repeatedly solved via Mixed-Integer Linear Programming, sometimes multiple times per day, with only minor changes in input data. In this work, we propose a number of machine learning (ML) techniques to effectively extract information from previously solved instances in order to significantly improve the computational performance of MIP solvers when solving similar instances in the future. Based on statistical data, we predict redundant constraints in the formulation, good initial feasible solutions and affine subspaces where the optimal solution is likely to lie, leading to significant reduction in problem size. Computational results on a diverse set of realistic and large-scale instances show that, using the proposed techniques, SCUC can be solved on average 4.3x faster with optimality guarantees, and 10.2x faster without optimality guarantees, but with no observed reduction in solution quality. Out-of-distribution experiments provides evidence that the method is somewhat robust against dataset shift.
△ Less
Submitted 18 December, 2019; v1 submitted 4 February, 2019;
originally announced February 2019.
