Showing 1–20 of 20 results for author: Rajagopal, R

TemperatureGAN: Generative Modeling of Regional Atmospheric Temperatures

Authors: Emmanuel Balogun, Ram Rajagopal, Arun Majumdar

Abstract: Stochastic generators are useful for estimating climate impacts on various sectors. Projecting climate risk in various sectors, e.g. energy systems, requires generators that are accurate (statistical resemblance to ground-truth), reliable (do not produce erroneous examples), and efficient. Leveraging data from the North American Land Data Assimilation System, we introduce TemperatureGAN, a Generat… ▽ More Stochastic generators are useful for estimating climate impacts on various sectors. Projecting climate risk in various sectors, e.g. energy systems, requires generators that are accurate (statistical resemblance to ground-truth), reliable (do not produce erroneous examples), and efficient. Leveraging data from the North American Land Data Assimilation System, we introduce TemperatureGAN, a Generative Adversarial Network conditioned on months, locations, and time periods, to generate 2m above ground atmospheric temperatures at an hourly resolution. We propose evaluation methods and metrics to measure the quality of generated samples. We show that TemperatureGAN produces high-fidelity examples with good spatial representation and temporal dynamics consistent with known diurnal cycles. △ Less

Submitted 19 October, 2024; v1 submitted 29 June, 2023; originally announced June 2023.

Comments: Accepted for publication at Environmental Data Science Journal

NeuralProphet: Explainable Forecasting at Scale

Authors: Oskar Triebe, Hansika Hewamalage, Polina Pilyugina, Nikolay Laptev, Christoph Bergmeir, Ram Rajagopal

Abstract: We introduce NeuralProphet, a successor to Facebook Prophet, which set an industry standard for explainable, scalable, and user-friendly forecasting frameworks. With the proliferation of time series data, explainable forecasting remains a challenging task for business and operational decision making. Hybrid solutions are needed to bridge the gap between interpretable classical methods and scalable… ▽ More We introduce NeuralProphet, a successor to Facebook Prophet, which set an industry standard for explainable, scalable, and user-friendly forecasting frameworks. With the proliferation of time series data, explainable forecasting remains a challenging task for business and operational decision making. Hybrid solutions are needed to bridge the gap between interpretable classical methods and scalable deep learning models. We view Prophet as a precursor to such a solution. However, Prophet lacks local context, which is essential for forecasting the near-term future and is challenging to extend due to its Stan backend. NeuralProphet is a hybrid forecasting framework based on PyTorch and trained with standard deep learning methods, making it easy for developers to extend the framework. Local context is introduced with auto-regression and covariate modules, which can be configured as classical linear regression or as Neural Networks. Otherwise, NeuralProphet retains the design philosophy of Prophet and provides the same basic model components. Our results demonstrate that NeuralProphet produces interpretable forecast components of equivalent or superior quality to Prophet on a set of generated time series. NeuralProphet outperforms Prophet on a diverse collection of real-world datasets. For short to medium-term forecasts, NeuralProphet improves forecast accuracy by 55 to 92 percent. △ Less

Submitted 29 November, 2021; originally announced November 2021.

Comments: NeuralProphet can be installed with pip or from https://github.com/ourownstory/neural_prophet - Documentation is available at https://neuralprophet.com

Quick Line Outage Identification in Urban Distribution Grids via Smart Meters

Authors: Yizheng Liao, Yang Weng, Chin-woo Tan, Ram Rajagopal

Abstract: The growing integration of distributed energy resources (DERs) in distribution grids raises various reliability issues due to DER's uncertain and complex behaviors. With a large-scale DER penetration in distribution grids, traditional outage detection methods, which rely on customers report and smart meters' last gasp signals, will have poor performance, because the renewable generators and storag… ▽ More The growing integration of distributed energy resources (DERs) in distribution grids raises various reliability issues due to DER's uncertain and complex behaviors. With a large-scale DER penetration in distribution grids, traditional outage detection methods, which rely on customers report and smart meters' last gasp signals, will have poor performance, because the renewable generators and storages and the mesh structure in urban distribution grids can continue supplying power after line outages. To address these challenges, we propose a data-driven outage monitoring approach based on the stochastic time series analysis with a theoretical guarantee. Specifically, we prove via power flow analysis that the dependency of time-series voltage measurements exhibits significant statistical changes after line outages. This makes the theory on optimal change-point detection suitable to identify line outages. However, existing change point detection methods require post-outage voltage distribution, which is unknown in distribution systems. Therefore, we design a maximum likelihood estimator to directly learn the distribution parameters from voltage data. We prove that the estimated parameters-based detection also achieves the optimal performance, making it extremely useful for fast distribution grid outage identifications. Furthermore, since smart meters have been widely installed in distribution grids and advanced infrastructure (e.g., PMU) has not widely been available, our approach only requires voltage magnitude for quick outage identification. Simulation results show highly accurate outage identification in eight distribution grids with 14 configurations with and without DERs using smart meter data. △ Less

Submitted 1 April, 2021; originally announced April 2021.

Comments: 12 pages, 12 figures. arXiv admin note: substantial text overlap with arXiv:1811.05646

Short-Term Solar Irradiance Forecasting Using Calibrated Probabilistic Models

Authors: Eric Zelikman, Sharon Zhou, Jeremy Irvin, Cooper Raterink, Hao Sheng, Anand Avati, Jack Kelly, Ram Rajagopal, Andrew Y. Ng, David Gagne

Abstract: Advancing probabilistic solar forecasting methods is essential to supporting the integration of solar energy into the electricity grid. In this work, we develop a variety of state-of-the-art probabilistic models for forecasting solar irradiance. We investigate the use of post-hoc calibration techniques for ensuring well-calibrated probabilistic predictions. We train and evaluate the models using p… ▽ More Advancing probabilistic solar forecasting methods is essential to supporting the integration of solar energy into the electricity grid. In this work, we develop a variety of state-of-the-art probabilistic models for forecasting solar irradiance. We investigate the use of post-hoc calibration techniques for ensuring well-calibrated probabilistic predictions. We train and evaluate the models using public data from seven stations in the SURFRAD network, and demonstrate that the best model, NGBoost, achieves higher performance at an intra-hourly resolution than the best benchmark solar irradiance forecasting model across all stations. Further, we show that NGBoost with CRUDE post-hoc calibration achieves comparable performance to a numerical weather prediction model on hourly-resolution forecasting. △ Less

Submitted 14 October, 2020; v1 submitted 9 October, 2020; originally announced October 2020.

FedGAN: Federated Generative Adversarial Networks for Distributed Data

Authors: Mohammad Rasouli, Tao Sun, Ram Rajagopal

Abstract: We propose Federated Generative Adversarial Network (FedGAN) for training a GAN across distributed sources of non-independent-and-identically-distributed data sources subject to communication and privacy constraints. Our algorithm uses local generators and discriminators which are periodically synced via an intermediary that averages and broadcasts the generator and discriminator parameters. We th… ▽ More We propose Federated Generative Adversarial Network (FedGAN) for training a GAN across distributed sources of non-independent-and-identically-distributed data sources subject to communication and privacy constraints. Our algorithm uses local generators and discriminators which are periodically synced via an intermediary that averages and broadcasts the generator and discriminator parameters. We theoretically prove the convergence of FedGAN with both equal and two time-scale updates of generator and discriminator, under standard assumptions, using stochastic approximations and communication efficient stochastic gradient descents. We experiment FedGAN on toy examples (2D system, mixed Gaussian, and Swiss role), image datasets (MNIST, CIFAR-10, and CelebA), and time series datasets (household electricity consumption and electric vehicle charging sessions). We show FedGAN converges and has similar performance to general distributed GAN, while reduces communication complexity. We also show its robustness to reduced communications. △ Less

Submitted 15 June, 2020; v1 submitted 12 June, 2020; originally announced June 2020.

Comments: 23 pages, 10 figures

Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding

Authors: Zhecheng Wang, Haoyuan Li, Ram Rajagopal

Abstract: Understanding intrinsic patterns and predicting spatiotemporal characteristics of cities require a comprehensive representation of urban neighborhoods. Existing works relied on either inter- or intra-region connectivities to generate neighborhood representations but failed to fully utilize the informative yet heterogeneous data within neighborhoods. In this work, we propose Urban2Vec, an unsupervi… ▽ More Understanding intrinsic patterns and predicting spatiotemporal characteristics of cities require a comprehensive representation of urban neighborhoods. Existing works relied on either inter- or intra-region connectivities to generate neighborhood representations but failed to fully utilize the informative yet heterogeneous data within neighborhoods. In this work, we propose Urban2Vec, an unsupervised multi-modal framework which incorporates both street view imagery and point-of-interest (POI) data to learn neighborhood embeddings. Specifically, we use a convolutional neural network to extract visual features from street view images while preserving geospatial similarity. Furthermore, we model each POI as a bag-of-words containing its category, rating, and review information. Analog to document embedding in natural language processing, we establish the semantic similarity between neighborhood ("document") and the words from its surrounding POIs in the vector space. By jointly encoding visual, textual, and geospatial information into the neighborhood representation, Urban2Vec can achieve performances better than baseline models and comparable to fully-supervised methods in downstream prediction tasks. Extensive experiments on three U.S. metropolitan areas also demonstrate the model interpretability, generalization capability, and its value in neighborhood similarity analysis. △ Less

Submitted 29 January, 2020; originally announced January 2020.

Comments: To appear in Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)

AR-Net: A simple Auto-Regressive Neural Network for time-series

Authors: Oskar Triebe, Nikolay Laptev, Ram Rajagopal

Abstract: In this paper we present a new framework for time-series modeling that combines the best of traditional statistical models and neural networks. We focus on time-series with long-range dependencies, needed for monitoring fine granularity data (e.g. minutes, seconds, milliseconds), prevalent in operational use-cases. Traditional models, such as auto-regression fitted with least squares (Classic-AR… ▽ More In this paper we present a new framework for time-series modeling that combines the best of traditional statistical models and neural networks. We focus on time-series with long-range dependencies, needed for monitoring fine granularity data (e.g. minutes, seconds, milliseconds), prevalent in operational use-cases. Traditional models, such as auto-regression fitted with least squares (Classic-AR) can model time-series with a concise and interpretable model. When dealing with long-range dependencies, Classic-AR models can become intractably slow to fit for large data. Recently, sequence-to-sequence models, such as Recurrent Neural Networks, which were originally intended for natural language processing, have become popular for time-series. However, they can be overly complex for typical time-series data and lack interpretability. A scalable and interpretable model is needed to bridge the statistical and deep learning-based approaches. As a first step towards this goal, we propose modelling AR-process dynamics using a feed-forward neural network approach, termed AR-Net. We show that AR-Net is as interpretable as Classic-AR but also scales to long-range dependencies. Our results lead to three major conclusions: First, AR-Net learns identical AR-coefficients as Classic-AR, thus being equally interpretable. Second, the computational complexity with respect to the order of the AR process, is linear for AR-Net as compared to a quadratic for Classic-AR. This makes it possible to model long-range dependencies within fine granularity data. Third, by introducing regularization, AR-Net automatically selects and learns sparse AR-coefficients. This eliminates the need to know the exact order of the AR-process and allows to learn sparse weights for a model with long-range dependencies. △ Less

Submitted 27 November, 2019; originally announced November 2019.

Comments: Building a bridge between traditional statistical time-series models and deep-learning models. Main Topics: Time-Series, Auto-Regression, Neural Networks, Sparsity, Long-Range Dependencies

Distributed generation of privacy preserving data with user customization

Authors: Xiao Chen, Thomas Navidi, Stefano Ermon, Ram Rajagopal

Abstract: Distributed devices such as mobile phones can produce and store large amounts of data that can enhance machine learning models; however, this data may contain private information specific to the data owner that prevents the release of the data. We wish to reduce the correlation between user-specific private information and data while maintaining the useful information. Rather than learning a large… ▽ More Distributed devices such as mobile phones can produce and store large amounts of data that can enhance machine learning models; however, this data may contain private information specific to the data owner that prevents the release of the data. We wish to reduce the correlation between user-specific private information and data while maintaining the useful information. Rather than learning a large model to achieve privatization from end to end, we introduce a decoupling of the creation of a latent representation and the privatization of data that allows user-specific privatization to occur in a distributed setting with limited computation and minimal disturbance on the utility of the data. We leverage a Variational Autoencoder (VAE) to create a compact latent representation of the data; however, the VAE remains fixed for all devices and all possible private labels. We then train a small generative filter to perturb the latent representation based on individual preferences regarding the private and utility information. The small filter is trained by utilizing a GAN-type robust optimization that can take place on a distributed device. We conduct experiments on three popular datasets: MNIST, UCI-Adult, and CelebA, and give a thorough evaluation including visualizing the geometry of the latent embeddings and estimating the empirical mutual information to show the effectiveness of our approach. △ Less

Submitted 20 April, 2019; originally announced April 2019.

Comments: accepted in ICLR 2019 SafeML workshop

Sequential Multiple Structural Damage Detection and Localization: A Distributed Approach

Authors: Yizheng Liao, Ram Rajagopal

Abstract: As essential components of the modern urban system, the health conditions of civil structures are the foundation of urban system sustainability and need to be continuously monitored. In Structural Health Monitoring (SHM), many existing works will have limited performance in the sequential damage diagnosis process because 1) the damage events needs to be reported with short delay, 2) multiple damag… ▽ More As essential components of the modern urban system, the health conditions of civil structures are the foundation of urban system sustainability and need to be continuously monitored. In Structural Health Monitoring (SHM), many existing works will have limited performance in the sequential damage diagnosis process because 1) the damage events needs to be reported with short delay, 2) multiple damage locations have to be identified simultaneously, and 3) the computational complexity is intractable in large-scale wireless sensor networks (WSNs). To address these drawbacks, we propose a new damage identification approach that utilizes the time-series of damage sensitive features extracted from multiple sensors' measurements and the optimal change point detection theory to find damage occurrence time and identify the number of damage locations. As the existing change point detection methods require to centralize the sensor data, which is impracticable in many applications, we use the probabilistic graphical model to formulate WSNs and the targeting structure and propose a distributed algorithm for structural damage identification. Validation results show highly accurate damage identification in a shake table experiment and American Society of Civil Engineers benchmark structure. Also, we demonstrate that the detection delay is reduced significantly by utilizing multiple sensors' data. △ Less

Submitted 14 December, 2018; originally announced December 2018.

Comments: 38 pages, 16 figures

Structural Damage Detection and Localization with Unknown Post-Damage Feature Distribution Using Sequential Change-Point Detection Method

Authors: Yizheng Liao, Anne S. Kiremidjian, Ram Rajagopal, Chin-Hsuing Loh

Abstract: The high structural deficient rate poses serious risks to the operation of many bridges and buildings. To prevent critical damage and structural collapse, a quick structural health diagnosis tool is needed during normal operation or immediately after extreme events. In structural health monitoring (SHM), many existing works will have limited performance in the quick damage identification process b… ▽ More The high structural deficient rate poses serious risks to the operation of many bridges and buildings. To prevent critical damage and structural collapse, a quick structural health diagnosis tool is needed during normal operation or immediately after extreme events. In structural health monitoring (SHM), many existing works will have limited performance in the quick damage identification process because 1) the damage event needs to be identified with short delay and 2) the post-damage information is usually unavailable. To address these drawbacks, we propose a new damage detection and localization approach based on stochastic time series analysis. Specifically, the damage sensitive features are extracted from vibration signals and follow different distributions before and after a damage event. Hence, we use the optimal change point detection theory to find damage occurrence time. As the existing change point detectors require the post-damage feature distribution, which is unavailable in SHM, we propose a maximum likelihood method to learn the distribution parameters from the time-series data. The proposed damage detection using estimated parameters also achieves the optimal performance. Also, we utilize the detection results to find damage location without any further computation. Validation results show highly accurate damage identification in American Society of Civil Engineers benchmark structure and two shake table experiments. △ Less

Submitted 14 November, 2018; originally announced December 2018.

Comments: 20 pages

Fast Distribution Grid Line Outage Identification with $μ$PMU

Authors: Yizheng Liao, Yang Weng, Chin-Woo Tan, Ram Rajagopal

Abstract: The growing integration of distributed energy resources (DERs) in urban distribution grids raises various reliability issues due to DER's uncertain and complex behaviors. With a large-scale DER penetration, traditional outage detection methods, which rely on customers making phone calls and smart meters' "last gasp" signals, will have limited performance, because the renewable generators can suppl… ▽ More The growing integration of distributed energy resources (DERs) in urban distribution grids raises various reliability issues due to DER's uncertain and complex behaviors. With a large-scale DER penetration, traditional outage detection methods, which rely on customers making phone calls and smart meters' "last gasp" signals, will have limited performance, because the renewable generators can supply powers after line outages and many urban grids are mesh so line outages do not affect power supply. To address these drawbacks, we propose a data-driven outage monitoring approach based on the stochastic time series analysis from micro phasor measurement unit ($μ$PMU). Specifically, we prove via power flow analysis that the dependency of time-series voltage measurements exhibits significant statistical changes after line outages. This makes the theory on optimal change-point detection suitable to identify line outages via $μ$PMUs with fast and accurate sampling. However, existing change point detection methods require post-outage voltage distribution unknown in distribution systems. Therefore, we design a maximum likelihood-based method to directly learn the distribution parameters from $μ$PMU data. We prove that the estimated parameters-based detection still achieves the optimal performance, making it extremely useful for distribution grid outage identifications. Simulation results show highly accurate outage identification in eight distribution grids with 14 configurations with and without DERs using $μ$PMU data. △ Less

Submitted 14 November, 2018; originally announced November 2018.

Comments: 9 pages

Data-Driven Load Modeling and Forecasting of Residential Appliances

Authors: Yuting Ji, Elizabeth Buechler, Ram Rajagopal

Abstract: The expansion of residential demand response programs and increased deployment of controllable loads will require accurate appliance-level load modeling and forecasting. This paper proposes a conditional hidden semi-Markov model to describe the probabilistic nature of residential appliance demand, and an algorithm for short-term load forecasting. Model parameters are estimated directly from power… ▽ More The expansion of residential demand response programs and increased deployment of controllable loads will require accurate appliance-level load modeling and forecasting. This paper proposes a conditional hidden semi-Markov model to describe the probabilistic nature of residential appliance demand, and an algorithm for short-term load forecasting. Model parameters are estimated directly from power consumption data using scalable statistical learning methods. Case studies performed using sub-metered 1-minute power consumption data from several types of appliances demonstrate the effectiveness of the model for load forecasting and anomaly detection. △ Less

Submitted 8 October, 2018; originally announced October 2018.

Understanding Compressive Adversarial Privacy

Authors: Xiao Chen, Peter Kairouz, Ram Rajagopal

Abstract: Designing a data sharing mechanism without sacrificing too much privacy can be considered as a game between data holders and malicious attackers. This paper describes a compressive adversarial privacy framework that captures the trade-off between the data privacy and utility. We characterize the optimal data releasing mechanism through convex optimization when assuming that both the data holder an… ▽ More Designing a data sharing mechanism without sacrificing too much privacy can be considered as a game between data holders and malicious attackers. This paper describes a compressive adversarial privacy framework that captures the trade-off between the data privacy and utility. We characterize the optimal data releasing mechanism through convex optimization when assuming that both the data holder and attacker can only modify the data using linear transformations. We then build a more realistic data releasing mechanism that can rely on a nonlinear compression model while the attacker uses a neural network. We demonstrate in a series of empirical applications that this framework, consisting of compressive adversarial privacy, can preserve sensitive information. △ Less

Submitted 2 October, 2018; v1 submitted 21 September, 2018; originally announced September 2018.

Journal ref: 2018 IEEE Conference on Decision and Control (CDC)

Unbalanced Multi-Phase Distribution Grid Topology Estimation and Bus Phase Identification

Authors: Yizheng Liao, Yang Weng, Guangyi Liu, Zhongyang Zhao, Chin-woo Tan, Ram Rajagopal

Abstract: There is an increasing need for monitoring and controlling uncertainties brought by distributed energy resources in distribution grids. For such goal, accurate multi-phase topology is the basis for correlating measurements in unbalanced distribution networks. Unfortunately, such topology knowledge is often unavailable due to limited investment, especially for \revv{low-voltage} distribution grids.… ▽ More There is an increasing need for monitoring and controlling uncertainties brought by distributed energy resources in distribution grids. For such goal, accurate multi-phase topology is the basis for correlating measurements in unbalanced distribution networks. Unfortunately, such topology knowledge is often unavailable due to limited investment, especially for \revv{low-voltage} distribution grids. Also, the bus phase labeling information is inaccurate due to human errors or outdated records. For this challenge, this paper utilizes smart meter data for an information-theoretic approach to learn the topology of distribution grids. Specifically, multi-phase unbalanced systems are converted into symmetrical components, namely positive, negative, and zero sequences. Then, this paper proves that the Chow-Liu algorithm finds the topology by utilizing power flow equations and the conditional independence relationships implied by the radial multi-phase structure of distribution grids with the presence of incorrect bus phase labels. At last, by utilizing Carson's equation, this paper proves that the bus phase connection can be correctly identified using voltage measurements. For validation, IEEE systems are simulated using three real data sets. The simulation results demonstrate that the algorithm is highly accurate for finding multi-phase topology even with strong load unbalancing condition and DERs. This ensures close monitoring and controlling DERs in distribution grids. △ Less

Submitted 12 July, 2019; v1 submitted 18 September, 2018; originally announced September 2018.

Comments: 17 pages, 18 figures

Generative Adversarial Privacy

Authors: Chong Huang, Peter Kairouz, Xiao Chen, Lalitha Sankar, Ram Rajagopal

Abstract: We present a data-driven framework called generative adversarial privacy (GAP). Inspired by recent advancements in generative adversarial networks (GANs), GAP allows the data holder to learn the privatization mechanism directly from the data. Under GAP, finding the optimal privacy mechanism is formulated as a constrained minimax game between a privatizer and an adversary. We show that for appropri… ▽ More We present a data-driven framework called generative adversarial privacy (GAP). Inspired by recent advancements in generative adversarial networks (GANs), GAP allows the data holder to learn the privatization mechanism directly from the data. Under GAP, finding the optimal privacy mechanism is formulated as a constrained minimax game between a privatizer and an adversary. We show that for appropriately chosen adversarial loss functions, GAP provides privacy guarantees against strong information-theoretic adversaries. We also evaluate GAP's performance on the GENKI face database. △ Less

Submitted 26 June, 2019; v1 submitted 13 July, 2018; originally announced July 2018.

Comments: Talk presentation at Privacy in Machine Learning and Artificial Intelligence (PiMLAI) Workshop, ICML 2018

Urban MV and LV Distribution Grid Topology Estimation via Group Lasso

Authors: Yizheng Liao, Yang Weng, Guangyi Liu, Ram Rajagopal

Abstract: The increasing penetration of distributed energy resources poses numerous reliability issues to the urban distribution grid. The topology estimation is a critical step to ensure the robustness of distribution grid operation. However, the bus connectivity and grid topology estimation are usually hard in distribution grids. For example, it is technically challenging and costly to monitor the bus con… ▽ More The increasing penetration of distributed energy resources poses numerous reliability issues to the urban distribution grid. The topology estimation is a critical step to ensure the robustness of distribution grid operation. However, the bus connectivity and grid topology estimation are usually hard in distribution grids. For example, it is technically challenging and costly to monitor the bus connectivity in urban grids, e.g., underground lines. It is also inappropriate to use the radial topology assumption exclusively because the grids of metropolitan cities and regions with dense loads could be with many mesh structures. To resolve these drawbacks, we propose a data-driven topology estimation method for MV and LV distribution grids by only utilizing the historical smart meter measurements. Particularly, a probabilistic graphical model is utilized to capture the statistical dependencies amongst bus voltages. We prove that the bus connectivity and grid topology estimation problems, in radial and mesh structures, can be formulated as a linear regression with a least absolute shrinkage regularization on grouped variables (\textit{group lasso}). Simulations show highly accurate results in eight MV and LV distribution networks at different sizes and 22 topology configurations using PG\&E residential smart meter data. △ Less

Submitted 13 September, 2018; v1 submitted 6 November, 2016; originally announced November 2016.

Comments: 15 pages, 16 figures

Journal ref: IEEE transactions on power systems 2018

A Sparse Linear Model and Significance Test for Individual Consumption Prediction

Authors: Pan Li, Baosen Zhang, Yang Weng, Ram Rajagopal

Abstract: Accurate prediction of user consumption is a key part not only in understanding consumer flexibility and behavior patterns, but in the design of robust and efficient energy saving programs as well. Existing prediction methods usually have high relative errors that can be larger than 30% and have difficulties accounting for heterogeneity between individual users. In this paper, we propose a method… ▽ More Accurate prediction of user consumption is a key part not only in understanding consumer flexibility and behavior patterns, but in the design of robust and efficient energy saving programs as well. Existing prediction methods usually have high relative errors that can be larger than 30% and have difficulties accounting for heterogeneity between individual users. In this paper, we propose a method to improve prediction accuracy of individual users by adaptively exploring sparsity in historical data and leveraging predictive relationship between different users. Sparsity is captured by popular least absolute shrinkage and selection estimator, while user selection is formulated as an optimal hypothesis testing problem and solved via a covariance test. Using real world data from PG&E, we provide extensive simulation validation of the proposed method against well-known techniques such as support vector machine, principle component analysis combined with linear regression, and random forest. The results demonstrate that our proposed methods are operationally efficient because of linear nature, and achieve optimal prediction performance. △ Less

Submitted 21 February, 2017; v1 submitted 5 November, 2015; originally announced November 2015.

Comments: 36 pages, 7 figures

Targeting Customers for Demand Response Based on Big Data

Authors: Jungsuk Kwac, Ram Rajagopal

Abstract: Selecting customers for demand response programs is challenging and existing methodologies are hard to scale and poor in performance. The existing methods were limited by lack of temporal consumption information at the individual customer level. We propose a scalable methodology for demand response targeting utilizing novel data available from smart meters. The approach relies on formulating the p… ▽ More Selecting customers for demand response programs is challenging and existing methodologies are hard to scale and poor in performance. The existing methods were limited by lack of temporal consumption information at the individual customer level. We propose a scalable methodology for demand response targeting utilizing novel data available from smart meters. The approach relies on formulating the problem as a stochastic integer program involving predicted customer responses. A novel approximation is developed algorithm so it can scale to problems involving millions of customers. The methodology is tested experimentally using real utility data. △ Less

Submitted 14 September, 2014; originally announced September 2014.

Distribution System Load and Forecast Model

Authors: Raffi Sevlian, Siddarth Patel, Ram Rajagopal

Abstract: This short document provides experimental evidence for the set of assumptions on the mean load and forecast errors made in \cite{Sevlian2014A_Outage} and \cite{Sevlian2014B_Outage}. We show that the mean load at any given node is distributed normally, where we compute the mean and variance. We then present an aggregation-error curve for a single day ahead forecaster. Residual analysis shows that b… ▽ More This short document provides experimental evidence for the set of assumptions on the mean load and forecast errors made in \cite{Sevlian2014A_Outage} and \cite{Sevlian2014B_Outage}. We show that the mean load at any given node is distributed normally, where we compute the mean and variance. We then present an aggregation-error curve for a single day ahead forecaster. Residual analysis shows that beyond 500 customers, gaussian residuals is a reasonable model. We then show the forecaster has uncorrelated errors. △ Less

Submitted 11 July, 2014; originally announced July 2014.

Short Term Electricity Load Forecasting on Varying Levels of Aggregation

Authors: Raffi Sevlian, Ram Rajagopal

Abstract: We propose a simple empirical scaling law that describes load forecasting accuracy at different levels of aggregation. The model is justified based on a simple decomposition of individual consumption patterns. We show that for different forecasting methods and horizons, aggregating more customers improves the relative forecasting performance up to specific point. Beyond this point, no more improve… ▽ More We propose a simple empirical scaling law that describes load forecasting accuracy at different levels of aggregation. The model is justified based on a simple decomposition of individual consumption patterns. We show that for different forecasting methods and horizons, aggregating more customers improves the relative forecasting performance up to specific point. Beyond this point, no more improvement in relative performance can be obtained. △ Less

Submitted 30 August, 2017; v1 submitted 31 March, 2014; originally announced April 2014.

Comments: Significant changes from previous version. Extension to full day ahead forecasting, added appendix of methodologies and scaling equality (not previous upper bound). Under review International Journal of Power and Energy Systems