1,253
Views
1
CrossRef citations to date
0
Altmetric
Article

Role of Variable Renewable Energy Penetration on Electricity Price and its Volatility across Independent System Operators in the United States

, , , , , , ORCID Icon, & show all
Article: 2158145 | Received 22 Dec 2021, Accepted 29 Nov 2022, Published online: 22 Feb 2023

Abstract

The US electrical grid has undergone substantial transformation with increased penetration of wind and solar forms of variable renewable energy (VRE). Despite the benefits of VRE for decarbonization, it has garnered some controversy for inducing unwanted effects in regional electricity markets. In this study, the role of VRE penetration is examined on the system electricity price and price volatility based on hourly, real-time, historical data from six independent system operators (ISOs) in the US using quantile and skew t-distribution regressions. After correcting for temporal effects, we found an increase in VRE penetration is associated with a decrease in system electricity price in all ISOs studied. The increase in VRE penetration is associated with a decrease in temporal price volatility in five out of six ISOs studied. The relationships are nonlinear. These results are consistent with the modern portfolio theory where diverse volatile assets may lead to more stable and less risky portfolios.

1. Introduction

The US electrical grid is undergoing a transformation with respect to the diversity of assets in its energy portfolio with substantial integration of renewable energy technologies, particularly variable renewable energy (VRE) technologies—non-dispatchable technologies with changing output dependent on renewable resource availability, such as solar and wind power. Although the majority of energy generation comes from fossil fuels, total energy generated from renewable resources (hydroelectric power included) accounted for 18% of total generation in the United States in 2020 with variable sources—wind and solar energy—accounting for 10.7% of the total generation (U.S. Energy Information Administration Citation2020). This is owing to increased build-out of renewable energy projects facilitated by the ebbing cost of installation and favorable policy initiatives (Bipartisan Policy Center Citation2020). By 2021, an estimated 27.6 GW of renewables is planned to come online with 12.2 GW from wind and 15.4 GW from utility-scale solar (U.S. Energy Information Administration Citation2021). While the addition of renewable energy will help accelerate the decarbonization of the power sector, its optimal integration will require substantial changes in power system design and market regulations to accommodate the inherent variability of these resources on the electric grid (U.S. Energy Information Administration Citation2021). To do this, it is important to elucidate an understanding of the role of VRE on electricity price and price volatility (both with respect to time and VRE penetration) because these impact the entire electricity value chain from the energy producers to the consumers.

Several articles have explored the relationship between electricity price and VRE penetration of wind (Woo et al. Citation2011, Citation2013; Brancucci Martinez-Anido et al. Citation2016; Zarnikau et al. Citation2019; Gil and Lin Citation2013; Tsai and Eryilmaz Citation2018; Quint and Dahlke Citation2019; Haratyk Citation2017; Zarnikau et al. Citation2020), solar (Deetjen et al. Citation2016; Craig et al. Citation2018), or both (Bushnell and Novan Citation2018; Wiser et al. Citation2017; Barbose et al. Citation2016; Woo et al. Citation2016, Citation2014). Some of these articles have used simulated data (Haratyk Citation2017; Deetjen et al. Citation2016; Barbose et al. Citation2016), while others have used historical data (Zarnikau et al. Citation2020; Bushnell and Novan Citation2018; Quint and Dahlke Citation2019; Tsai and Eryilmaz Citation2018; Craig et al. Citation2018; Gil and Lin Citation2013; Zarnikau et al. Citation2019; Woo et al. Citation2011, Citation2013, Citation2014, Citation2016). A few have combined simulated data with historical data validation (Brancucci Martinez-Anido et al. Citation2016; Wiser et al. Citation2017; Mills et al. Citation2020, Citation2021). The methods used in these studies have primarily been multivariate linear regressions (Zarnikau et al. Citation2020; Bushnell and Novan Citation2018; Haratyk Citation2017; Wiser et al. Citation2017; Quint and Dahlke Citation2019; Tsai and Eryilmaz Citation2018; Craig et al. Citation2018; Zarnikau et al. Citation2019; Woo et al. Citation2011, Citation2013, Citation2014, Citation2016; Mwampashi et al. Citation2020; Gelabert et al. Citation2011) or visualizations with descriptive statistics (Blazquez et al. Citation2018; Cutler et al. Citation2011). It is worth noting that the optimization methods in production-cost models for most of the scenario-type/simulation-based analyses (Barbose et al. Citation2016; Brancucci Martinez-Anido et al. Citation2016; Deetjen et al. Citation2016; Seel et al. Citation2018) are often based on linear assumptions. These methods are, however, limited in that they assume a linear relationship between electricity price and VRE with constant variance, but there are inherent distributional changes in the electricity price with respect to VRE requiring more robust modeling choices. One alternative approach has been applied to a case study of PJM (see for acronym definitions) using the Robust Linear Weighted Regression (Gil and Lin Citation2013); however, the scope is limited to a single independent system operator (ISO) and one year of data. In general, increasing penetration of VRE is associated with a lower average wholesale price of electricity. This behavior has been found in both domestic (Zarnikau et al. Citation2020; Bushnell and Novan Citation2018; Haratyk Citation2017; Wiser et al. Citation2017; Quint and Dahlke Citation2019; Tsai and Eryilmaz Citation2018; Craig et al. Citation2018; Barbose et al. Citation2016; Gil and Lin Citation2013; Zarnikau et al. Citation2020; Brancucci Martinez-Anido et al. Citation2016; Deetjen et al. Citation2016; Woo et al. Citation2011, Citation2013, Citation2014, Citation2016; Seel et al. Citation2018) and some international (Blazquez et al. Citation2018; Cutler et al. Citation2011; Mwampashi et al. Citation2020; Gelabert et al. Citation2011; Jónsson et al. Citation2010; Sáenz de Miera et al. Citation2008) electricity markets. A summary of the domestic studies can be found in . Using the robust methods in this study, the results not only corroborate the existing literature on increased VRE penetration leading to a reduction in electricity price, but also illustrate that this effect is nonlinear, and the greatest impact can be seen in the reductions in extremely high system electricity prices with increased VRE.

Table 1. Temporal range, resolution, and VRE composition for ISO data.

Table 2. Literature summary for US studies (domestic) on VRE impact on price. The specified VRE, solar or wind, is given in the rightmost column.

Negative system electricity price is also a characteristic of the modern electricity market. Negative prices occur as a result of generation-demand imbalance resulting in high supply during times of low demand. Several factors contribute, including a substantial decrease in demand, limited flexibility in power-plant operations (e.g. slow or expensive ramping, limited energy storage), and limited transmission capacities. Studies of some electricity markets have attributed higher frequency of negative prices to high penetrations of VRE (Cutler et al. Citation2011; De Vos Citation2015). One study found that extreme negative prices—the result of electricity oversupply—are highly correlated with periods of high wind penetration (Cutler et al. Citation2011). However, there was no relationship observed between the frequency of negative electricity prices and VRE, with inconsistent behavior observed across the ISOs studied.

While the literature on the average behavior of electricity price and VRE penetration has been consistent, its price volatility with respect to time (temporal price volatility) effects have been a topic of controversy. Some studies find evidence of an increase in temporal price volatility as VRE penetration increases (Mwampashi et al. Citation2020; Astaneh and Chen Citation2013; Woo et al. Citation2011; Seel et al. Citation2018), while others are unable to show significant evidence that increasing the share of VRE leads to high temporal price volatility (Rai and Nunn Citation2020; Mulder and Scholtens Citation2013). A case study of Germany and Denmark found that increasing penetration of VRE could either increase or decrease the temporal price volatility (Rintamäki et al. Citation2017). While studies have often found that high wind-energy penetration contributes to high temporal price volatility (Woo et al. Citation2011; Brancucci Martinez-Anido et al. Citation2016; Mwampashi et al. Citation2020), a study in South Korea showed that temporal price-volatility decreases as wind penetration increases up to 10% because the wind profile matches the demand patterns at this penetration (Shcherbakova et al. Citation2014). There are also instances where solar-energy penetration has abated temporal price volatility (Pereira da Silva and Horta Citation2018; Rintamäki et al. Citation2017). The evidence from literature suggests that the relationship between temporal price volatility and VRE penetration can vary widely as a result of a confluence of several factors, including patterns of demand (Shcherbakova et al. Citation2014), weather (Mwampashi et al. Citation2020), and the availability of flexible generation (Rintamäki et al. Citation2017). In five of the six ISOs studied, the system temporal price volatility decreased as the penetration of VRE increased across all quantiles. The results are consistent with the modern portfolio theory that posits that the portfolio of diverse and uncorrelated (or low correlated) assets leads to price-volatility reduction (Markowitz Citation1952). Similar to the finance field, we show that adding diverse and volatile assets can lead to a less risky portfolio of such energy assets. For example, adding solar and wind power to more traditional power generation sources, such as coal, natural gas, and nuclear, leads to a more diverse portfolio. Adding different technologies to the energy portfolio is beneficial and, as a result, an increase in VRE penetration leads to a reduction of temporal price volatility.

In this study, robust methods—quantile regression and skew-t distribution regression—are used to evaluate the impact of VRE penetration on the conditional electricity price and price volatility. The analysis is done conditionally on the seasonality in order to assess effects on the residual price, i.e. we analyze the behavior of the price after removing what is expected given the time of day and year. Volatility is defined both with respect to time (temporal price volatility) and VRE penetration. Quantile regression and skew-t distribution regression have been chosen since they are well suited for handling outliers, nonlinearity, and skewness. By quantifying the skewness of the conditional price distribution, we aim to evaluate the frequency of above- and below-average conditional prices at varying levels of VRE. This analysis examines six of the seven ISO regions in the United States shown in (excluding the Electricity Reliability Council of Texas, ERCOT). The ERCOT region has been excluded from this analysis due to data constraints that make a comparable analysis infeasible. Despite this, to the best of our knowledge this is the most comprehensive study of the role of VRE on system electricity price and price volatility at the national scale for the United States using robust, nonlinear methods.

Figure 1. Map of independent system operators (ISOs) in the United States. ISOs are responsible for coordinating, controlling, and monitoring the electricity grid within their region of control. There are seven ISOs in the United States, and the regions covered by these ISOs account for 67% of the total electricity demand in the country. Adapted from Climate-XChange (Citation2021).

Figure 1. Map of independent system operators (ISOs) in the United States. ISOs are responsible for coordinating, controlling, and monitoring the electricity grid within their region of control. There are seven ISOs in the United States, and the regions covered by these ISOs account for 67% of the total electricity demand in the country. Adapted from Climate-XChange (Citation2021).

Our robust, nonlinear methods for exploring the relationship between VRE and system electricity price and price volatility in several ISOs in the United States could be applied in other markets and has applicability in the broader international context. For example, international studies have explored this relationship in Italy (Blazquez et al. Citation2018), Germany (Blazquez et al. Citation2018; Mwampashi et al. Citation2020), South Australia (Cutler et al. Citation2011), Spain (Blazquez et al. Citation2018; Sáenz de Miera et al. Citation2008), Denmark (Mwampashi et al. Citation2020; Jónsson et al. Citation2010), and South Korea (Shcherbakova et al. Citation2014), and could be enriched by our chosen methods. The approach captures nonlinearity and distributional effects and can be applied to other electricity markets. Not only the methods but also the findings have a broader context in relation to modern portfolio theory with respect to the addition of a highly volatile asset, VRE, and a corresponding reduction in electricity price and volatility. However, it is important to note that though our methodological approach can be extended to other electricity markets to explore the relationship between VRE and system electricity price or price volatility, the actual trend and relationship may inherently differ in each context owing to several factors, such as the amount of VRE penetration, and the policies and regulations that govern the local electricity market.

This article aims to (i) apply robust, nonlinear methods to the study of electricity price and price volatility, (ii) elucidate more generalizable conclusions by scaling the analysis to explore multiple ISOs, and (iii) contextualize the results based on modern portfolio theory.

2. Methods

2.1. Data Collection and Processing

Each independent system operator (ISO) collects data on the system price of electricity and the amount of electricity that was generated from various energy sources (e.g. solar, wind, natural gas, coal, nuclear). These data are available on each ISO’s respective website in various formats. For example, some ISOs release files containing full years’ worth of data, while others provide daily updates through widgets. Therefore, a web scraper was built to collect the data for each ISO. Note the different operators collect data at different temporal frequencies, but the highest resolution that the operator provides is always used. Furthermore, the time periods covered for each operator do not match. As a result, there are more data for some operators than others; however, this difference is not expected to impact the analysis because each analysis is done independently (See for the temporal resolution and coverage period for each ISO). From this price and supply information, we calculate two variables at each time point: (i) the percentage of energy generated using VRE and (ii) the system price per megawatt-hour of electricity.

2.1.1. Percentage of VRE

In calculating the percentage of VRE used throughout the analysis, only electricity generated from solar and wind is considered. However, each ISO provides slightly different granularity on the energy sources in their electricity-generation mix. For example, if the relative generation from solar is low, then this source may be bundled with other renewable energy sources. Because of the variability of solar adoption and the differences in data reporting among ISOs, solar electricity-generation data was not specifically reported in the generation mix for NYISO and MISO, in which case only the wind electricity-generation data was used to represent the VRE percentage (). Note, the effect of these differences is small and should not impact the analysis. The percentage of VRE is calculated as the sum of the generation from solar and wind sources divided by the total amount of generation from all sources.

2.1.2. System Price

Each ISO reports information on the price paid at specific locations or hubs in their network. Due to transmission and congestion costs, the price at different hubs can be different. To ensure that the results are interpretable and comparable, the transmission and traffic/congestion costs are removed to compute a system price for each ISO.

For each ISO, this system price reflects the cost of generating one megawatt-hour of electricity. Again, since each ISO has different reporting standards, there are slight differences in how the system price is computed for each ISO. Full details for the relevant procedure for each ISO are given in Appendix A1. As before, these differences are not expected to impact the analysis.

2.2. Seasonal and Diurnal Adjustment

Changes in energy price are driven to a large extent by changes in consumer demand. For example, energy prices are higher during the day (when consumers are awake) than at night and higher during the summer (when consumers use air conditioning) than spring. To properly analyze the impact of VRE penetration on energy prices, this expected pattern is taken into account and used to detrend the price. Before detrending prices at the temporal scales, it is not clear whether the estimated associations between energy price and VRE penetration are due to expected seasonal fluctuations or changes in VRE penetration. By removing the expected pattern, we can analyze the effect of VRE penetration on the changes in energy price compared to what is expected.

We define what is expected as a linear model with categorical variables used to estimate these temporal effects. For the hourly data, categorical variables for hour of day, season, and weekend are included. The model allows for interaction effects between the hour of day and season, so that each season has its own diurnal pattern. More formally, for the hourly data the price at time t is given by (1) Price(t)=β1hour(t)+β2season(t)+β3hour(t)×season(t)+β4weekend(t)+ϵt,(1) where hour(t), season(t) give the hour of the day and season at time t respectively, weekend(t) is a binary variable indicating whether time t occurs on a weekend and ϵt is a residual term. There are four seasons which correspond to the following calendar month groupings: winter = 12, 1, 2, spring = 3, 4, 5, summer = 6, 7, 8, and autumn = 9, 10, 11. Note, in a standard linear model the residual term is noise. This is not the case in this setting because the price (and temporal price volatility) is expected to be driven by varying levels of VRE. Fitted values for the six locations are shown in . The estimated diurnal pattern for CAISO (separated by season) is shown in . A more detailed figure is available in Appendix A2.

Figure 2. The energy price data for each ISO is detrended by the average price for hour, season, and weekend trends. The observed price data, shown with black dots, is plotted by day of the year. The red line indicates the estimated trend. The resulting detrended price is the observed value minus the average. Note there are repeated observations for multiple years and the y-axis is clipped to the range $10–150.

Figure 2. The energy price data for each ISO is detrended by the average price for hour, season, and weekend trends. The observed price data, shown with black dots, is plotted by day of the year. The red line indicates the estimated trend. The resulting detrended price is the observed value minus the average. Note there are repeated observations for multiple years and the y-axis is clipped to the range $10–150.

Figure 3. CAISO raw system prices by seasons and hour of day. The plot shows the estimated diurnal pattern (red line) by season. For plotting purposes, prices greater than 150 are excluded. Orange dots depict negative prices and teal dots depict positive prices (all in USD).

Figure 3. CAISO raw system prices by seasons and hour of day. The plot shows the estimated diurnal pattern (red line) by season. For plotting purposes, prices greater than 150 are excluded. Orange dots depict negative prices and teal dots depict positive prices (all in USD).

The parameters were estimated using the base lm function within the R programming language. The residual terms, ϵt, are used to explore the relationship between VRE and price, and is referred to as the detrended system electricity price. The following analyses will make use of this detrended system price to calculate temporal volatility as well as examine the relationship with VRE penetration.

2.3. Temporal Price Volatility

We calculated the temporal volatility of detrended system electricity price as a function of time (temporal price volatility), using the exponential weighted moving standard deviation (EWMSD). Detrended price volatility is the instantaneous standard deviation of the hourly detrended electricity price. The exponentially weighted moving standard deviation improves on the sample standard deviation by assigning weights to the periodic price. Specifically, EWMSD gives more weight to the recent price and decreases the weights exponentially as historical data points increases. The equation is usually given as: (2) EWMSDi=(1λ)Ui2+(1λ)λ2Ui12++(1λ)λnUin2,(2) where Ui2 is the price variance in the current hour i and λ is the weight. We have used a weight of 0.94 because this a standard weighting factor used in financial risk/volatility analysis (Longerstaey and Spencer Citation1996). This method of calculating volatility is based on the operation-stage volatility (realized effects) and should not be confused with planning-stage volatility that attempts to capture differences between the predicted and observed real-time price variations (predicted effects; Astaneh and Chen Citation2013).

2.4. Nonparametric Quantile Regression

A nonparametric quantile regression model was fitted based on B-splines basis function expansion of percentage VRE to examine the possible nonlinear effects of the percentage of VRE on the hourly electricity price and temporal price volatility. This approach does not require us to assume a specified distribution for the response and is robust with heavy-tailed distributions as compared to linear regression. By fitting quantile regression at different quantile levels, a more comprehensive characterization of the effect of the percentage of renewables on price and price volatility can be obtained.

The τth (0<τ<1) quantile of a random variable Y is QY(τ)=inf{t:FY(t)τ} where FY is the cumulative distribution function of Y. Quantiles in the context of regression had been first introduced in Koenker and Bassett (Citation1978). Given a dependent variable Y and independent variable matrix X, linear regression estimates the conditional mean of Y given X by assuming that the dependence relation Y=Xβ and that the cumulative distribution of Y given X, FY|x, is a Gaussian distribution, among other assumptions. Instead, quantile regression assumes that the τth (0<τ<1) conditional quantile of Y given X = x is defined as QY(τ|x)=inf{t:FY|x(t)τ}=Xβ, without any further assumption on FY|x. The conditional median corresponds to QY(0.5|x). Interpretation of the conditional quantile is straightforward. For example, given the predictor value X = x and τ=0.9, 90% of observations of Y with associated x fall below QY|X(0.9). Given τ=0.9, an increase in X by one unit would increase the 0.9 quantile of Y by β.

In general, assuming the linear functional form Xβ of the quantile QY(τ|x) may lead to undesirable results when the conditional quantile of Y given X, QY(τ|x), is not line QY(τ|x) ar in X and hence mis-specified. Splines can be used as a curve-fitting technique for using piecewise polynomials of X. Knots are values of the independent variable which are used to partition the curve into pieces to fit a final piecewise polynomial and the degree of each polynomial defines the degree of the spline. The nonparametric quantile regression models the conditional quantile using a spline approximation instead of a specific functional form of X. The B-splines (Schoenberg Citation1988; Curry and Schoenberg Citation1988) approximation is known to be flexible and computationally efficient. Specifically, let π(t)=(b1(t),,bkn+l+1(t)) denote a vector of normalized B-spline basis functions of order l + 1 and degree l. Then QY(τ|x) is approximated by β0+π(x)β, where β0 is the intercept and β are a vector of l + 1 coefficients to be estimated from the data by minimizing a quantile loss function. The quantile regression is implemented using the Quantreg package in R which can incorporate splines. Splines can be calculated using the Splines package in R.

When analyzing the hourly price, the nonparametric quantile regression is applied to the detrended hourly price (dependent variable) conditional on B-splines of percentage of VRE (independent variables), as previously described. When analyzing the hourly temporal price volatility, nonparametric quantile regression is applied to the exponentially weighted moving standard deviation (EWMSD) estimator of hourly volatility (2)(dependent variable), computed using the detrended price (1) with B-splines of percentage of VRE as the independent variables. B-splines of degree 3 are used with number of knots depending on visual inspection of the data scatterplot of price or price volatility vs. VRE such that the splines do not overfit at the boundaries.

We evaluate the subsequent nonlinear effects induced by the B-spline basis by calculating an approximate derivative. The approximate derivative of the spline basis functions, π(t)=(b1(t),,bkn+l+1(t)), can be calculated using the JM package in R. The derivative of QY(τ|x) is then approximated by (π(x))β.

2.4.1. Skew t-Distribution Regression

It was clear from plots of price against VRE penetration () that many features of the distribution of price were changing as a function of VRE penetration. As an alternative to the nonparametric quantile regression, we chose to additionally model the distribution with the parameteric skew t-distribution. The skew t-distribution analysis provides complementary interpretation of generalized linear effects on the conditional distribution of detrended system price given VRE penetration.

Figure 4. Quantiles of skew t-distribution based on the estimated pointwise means of the parameters listed in . Points correspond to seasonally detrended system price and the y-axis is trimmed to the inner 99% of detrended price; red points correspond to a negative price before seasonal detrending.

Figure 4. Quantiles of skew t-distribution based on the estimated pointwise means of the parameters listed in Table A1. Points correspond to seasonally detrended system price and the y-axis is trimmed to the inner 99% of detrended price; red points correspond to a negative price before seasonal detrending.

A generalized linear model was fitted for location, shape, and scale of the conditional distribution of detrended price, ϵt, given percentage VRE. We used the four parameter skew t-distribution (ST) parameterized as in Fernández and Steel (Citation1998) with parameters <μ<,σ>0,ν>0, and τ>0. The centrality and skewness (shape) parameters, μ and ν respectively, varied with percent VRE (xt) via a generalized linear model: μt=β0μ+β1μxt/100,log(νt)=β0ν+β1νxt/100, where the log transformation ensures positivity of νt and the scale and tail index parameters, σ and τ respectively, were assumed to be constant: log(σ)=β0σ,log(τ)=β0τ.

Therefore, the following conditional distribution is obtained for the residual price: ϵt|xtST(μt,νt,σ,τ). All parameters were estimated by maximum likelihood with the RS algorithm as implemented in the R package Gamlss (Stasinopoulos and Rigby Citation2007). For a given level of VRE, xt, we evaluate the estimated distribution by quantile summaries as the complete distribution is defined.

3. Vre Penetration and System Electricity Price

Renewable energy resources and the adoption of utility-scale VRE technologies varies substantially throughout the United States, as seen in the corresponding VRE generation shares across ISOs in . In order to accurately assess the role of VRE on system electricity price, a comparative analysis across multiple ISOs was performed using regional data from 2014 to 2020 when available. Details on the regional data used for each ISO can be found in . The system electricity price is based on the real-time wholesale price of electricity excluding the congestion costs and transmission losses. Corrections were also made for the diurnal, seasonal, and weekend effects that may influence demand and resource availability and ultimately impact price (shown in ). Descriptive statistics for the electricity price and temporal price volatility after applying these corrections, can be found in . For each ISO studied, the mean system electricity price is substantially larger than the median price, indicating skewness and price outliers, suggesting that the ordinary least squares approach applied in the majority of the previous studies using multivariate linear regression (Zarnikau et al. Citation2020; Bushnell and Novan Citation2018; Haratyk Citation2017; Wiser et al. Citation2017; Quint and Dahlke Citation2019; Tsai and Eryilmaz Citation2018; Craig et al. Citation2018; Zarnikau et al. Citation2019; Woo et al. Citation2011, Citation2013, Citation2014, Citation2016; Mwampashi et al. Citation2020; Gelabert et al. Citation2011) may not have adequately captured the dynamics of the existing relationship between VRE penetration and electricity price.

Figure 5. VRE (solar and wind) penetration. Shown here is the percentage of VRE generation to total generation in each of the six ISOs considered. Southwest Power Pool (SPP) achieved the highest VRE penetration based on the data used and for the study period considered, followed by CAISO, MISO, ISONE, NYISO, and PJM.

Figure 5. VRE (solar and wind) penetration. Shown here is the percentage of VRE generation to total generation in each of the six ISOs considered. Southwest Power Pool (SPP) achieved the highest VRE penetration based on the data used and for the study period considered, followed by CAISO, MISO, ISONE, NYISO, and PJM.

Table 3. Descriptive Statistics Summary. This shows the minimum, median, maximum, and mean statistics for the detrended system price (in USD) and temporal price volatility (in USD).

The quantile regression results in show that an increase in the penetration of VRE is associated with a decrease in detrended system electricity price. While this trend is generally true across all ISOs and quantiles (0.25th, 0.50th, and 0.75th), the relative system price reduction is often greatest in the extreme prices, i.e. the 0.90th quantile. Additionally, the change in the system price is not constant across the percentage of VRE nor the quantiles and, once again, this is typically more apparent at higher quantiles. For example, in PJM, , when the VRE percentage changes from 0 to 1%, the median detrended system price decreases by $1.6 USD in average absolute values and the 0.90th quantile of the detrended system price decrease by $10 in average absolute values but, when the VRE percentage changes from 5 to 6%, the median detrended system price decreases by $0.2 in average absolute values and the 0.90th quantile detrended price decreases by $0.8 in average absolute values. The derivative of the detrended system price with respect to the percentage of VRE is depicted in for PJM. It is evident that increasing penetration of VRE lowers the frequency of occurrence of higher extremes of system prices because suggest the value of the 0.90th quantile is decreasing with increasing VRE.

Figure 6. VRE penetration (in %) and detrended electricity price (in $). The six panels show the relationship between VRE penetration and detrended electricity price for NYISO, MISO, PJM, CAISO, ISONE, and SPP, respectively. The continuous lines show the quantiles with the colors yellow, green, blue, and purple, representing the 0.25th, 0.50th (median), 0.75th, and the 0.90th quantiles, respectively. Each panel’s axis limits are adjusted separately in order to clearly show the quantiles’ trend direction.

Figure 6. VRE penetration (in %) and detrended electricity price (in $). The six panels show the relationship between VRE penetration and detrended electricity price for NYISO, MISO, PJM, CAISO, ISONE, and SPP, respectively. The continuous lines show the quantiles with the colors yellow, green, blue, and purple, representing the 0.25th, 0.50th (median), 0.75th, and the 0.90th quantiles, respectively. Each panel’s axis limits are adjusted separately in order to clearly show the quantiles’ trend direction.

Figure 7. The derivative of the detrended system price (in USD) and detrended system price volatility (temporal volatility in USD) with respect to the percentage of VRE for the PJM independent system operator.

Figure 7. The derivative of the detrended system price (in USD) and detrended system price volatility (temporal volatility in USD) with respect to the percentage of VRE for the PJM independent system operator.

The results from the skew-t distribution regression in also show the distributional effects of increasing VRE on price. A decreasing trend in skewness with VRE is estimated for all ISOs. The skewness approaches symmetry (ν = 1) with increasing VRE for four of the ISOs, suggesting that as VRE is increased there is a lower frequency of large price extremes above the average price. Large price extremes often happen when expensive energy generators need to be dispatched, typically when low-cost electricity generators, such as solar and wind, are unavailable or unable to meet higher than expected demand. The magnitude and duration of these price spikes are particularly harmful to electricity retailers who cannot pass on price risk to customers (Anderson et al. Citation2007). The main contribution to the pattern toward symmetry is the decrease in large, positive fluctuations from the average price. A decrease in skewness to negative is found for SPP and CAISO, which implies that as VRE increases in these systems it is more likely to have deviations of price below average. It is worth noting that these two regions (SPP and CAISO) also have the highest levels of VRE penetration.

One policy mechanism that has been used to reduce price spikes is to implement price caps. Details of the price cap, including the capped value, when the price cap is enforced, and which technologies are restricted by the cap, vary with time and across ISOs. Temporary price caps have been implemented in times of extreme weather events or major distortion in electricity markets (Sirin and Erten Citation2022). In 2016, the Federal Energy Regulatory Commission issued Order 831, which generally created a price cap of $1,000/MWh to be applied across all regional transmission organizations and ISOs (U.S. Federal Energy Regulatory Commission Citation2016). Please note that there are exceptions indicated in the order that reflect a resource’s verified cost-based incremental energy offer. To assess the impact of price caps on our analysis, we calculated the fraction of electricity prices at or above $1,000/MWh for each ISO. We found that for all ISOs, the relative frequencies were all around or less than 0.01%. Since the price cap has been set well above the marginal and opportunity costs of all or nearly all resources, this policy mechanism has rarely prevented voluntary market clearing that would otherwise occur (Wilson Citation2000). Therefore, the results found in our study are unlikely to be a result of price caps.

Figure 8. Estimated effects of VRE percentage on the skewness (ν) parameter for a skew t-distribution for detrended price using a log link. Gray bands indicate the 95% confidence intervals. The red reference line at 1 denotes symmetry.

Figure 8. Estimated effects of VRE percentage on the skewness (ν) parameter for a skew t-distribution for detrended price using a log link. Gray bands indicate the 95% confidence intervals. The red reference line at 1 denotes symmetry.

The results for the system price reduction are consistent with the merit-order effect of renewables where conventional generation assets are unprofitable in the market when there is a high penetration of VRE. This occurs because VREs tend to have lower marginal operating costs and are, therefore, prioritized in satisfying demand. This effect has been well studied and observed in electricity markets in several countries (Sensfuß et al. Citation2008; Cludius et al. Citation2014; Würzburg et al. Citation2013; Cutler et al. Citation2011; Böckers et al. Citation2013; Forrest and MacGill Citation2013; Amor et al. Citation2014; Huisman et al. Citation2013; Swinand and O’Mahoney Citation2015; Klinge Jacobsen and Zvingilaite Citation2010; Shcherbakova et al. Citation2014; Winkler et al. Citation2016). The lowering of electricity prices as VRE penetration increases is advantageous for consumers but not necessarily for the generators/suppliers because this causes a reduction in the income from electricity sold in the market. However, the lower cost electricity can attract energy-intensive industries, increasing electricity demand within the ISO, allowing generators/suppliers to increase revenues from increased production. Additionally, not all generators are affected equally and some, particularly those with flexible generation assets, may benefit from increased VRE penetration since they can contribute flexible reserves to manage VRE variability (International Renewable Energy Agency Citation2019; Akrami et al. Citation2019). Apart from attaining a sustainable price level (Winkler et al. Citation2016), which is good for end users, the merit-order effect can also help to promote flexibility in the market to incorporate flexible generation assets that can ramp up and down in response to increasing VRE penetration. It can also incentivize storage and demand-response technologies. Security of supply is another concern because the merit-order effect may lead to the shutdown of baseload conventional plants; however, this risk can be reduced with the introduction of capacity markets (Winkler et al. Citation2016). Lowering the system electricity price may increase the cost of some sustainable energy policies, such as feed-in tariffs (Huisman et al. Citation2013), and may lead to lower cost flexible natural gas plants replacing nuclear plants and, consequently, producing more emissions (Rothwell Citation2000). Careful policy design will be needed to mitigate these effects.

Negative prices are the extreme end-products of oversupply of electricity. It is most common in markets with large amounts of nuclear, hydroelectric, and wind generation (U.S. Energy Information Administration Citation2012). However, conventional resources also play a role in driving down prices (at least in the short term) because generators from these inflexible assets often prefer not to shut down or reduce their output when there are high amounts of VRE due to technical and economic costs related to output reduction and regulation costs (in the case of hydroelectric power, for compliance with environmental regulations for water flow to maintain fish population; U.S. Energy Information Administration Citation2012). The trend of negative system price (based on the original raw data without correction for temporal effects) with increasing VRE penetration varies for different ISOs (shown in the red-colored points in and ). However, the frequency of negative detrended system price has no clear trend with percentage of VRE; although it seems to be concentrated at high levels of VRE penetration for SPP and CAISO, the two ISOs with the highest range of VRE penetration.

This observation might suggest that at the current levels of VRE penetration, among other factors, increased VRE penetration is not solely responsible for increased frequency of negative system electricity prices but rather due to a combination of other factors such as seasonal, daily, and weekend temporal effects as well as geographic, demographic, climatic, and behavioral factors, which are inherently different across each of the ISOs studied. In general, at very high penetration of VRE, there needs to be increased flexibility of the electrical grid in terms of availability of grid-scale energy storage systems that can act as control reserves in times of excess supply of VRE.

4. VRE Penetration and System Electricity Price Volatility

The system electricity price varies as a function of both time and VRE penetration. First, to calculate the volatility of the detrended system electricity price as a function of time (temporal price volatility), the exponential weighted moving average (EWMSD) volatility is employed. Second, to assess the variability of the detrended system electricity price as a function of VRE penetration, the distributional characteristics of the skew-t model are evaluated.

The results for the quantile regression on EWMSD price volatility (temporal volatility), shown in , indicate that there is a reduction in temporal price volatility as the penetration of VRE increases across all quantiles (0.50th, 0.75th and 0.90th) for 5 out of the 6 ISOs studied (ISONE, MISO, PJM, CAISO, and NYISO). In the upper quantile (0.90th quantile), it is evident that at higher quantiles there is a larger reduction in temporal price volatility as VRE penetration increases, indicating that there is a significant drop in higher extremes of temporal price volatility with VRE penetration. For SPP, an increase in VRE percentage up to about 40% is associated with decreasing the temporal price volatility; however, an increase in temporal price volatility is observed as VRE penetration increases beyond 40%. The difference in the effect of VRE penetration on temporal price volatility for SPP can be attributed to times of high penetration of wind when demand is low, causing oversupply of electricity and, thus, increasing the frequency of negative prices. This effect observed in SPP corroborates some of the literature on price volatility where high penetration of wind can cause an increase in price volatility (Rintamäki et al. Citation2017).

Figure 9. VRE penetration (in %) and temporal price volatility (in USD). The panels show the relationship between VRE penetration and temporal detrended system price volatility for NYISO, MISO, PJM, CAISO, ISONE, and SPP, respectively. The continuous lines show the quantiles with the colors yellow, green, and purple, representing the 0.50th (median), 0.75th and the 0.90th quantiles.

Figure 9. VRE penetration (in %) and temporal price volatility (in USD). The panels show the relationship between VRE penetration and temporal detrended system price volatility for NYISO, MISO, PJM, CAISO, ISONE, and SPP, respectively. The continuous lines show the quantiles with the colors yellow, green, and purple, representing the 0.50th (median), 0.75th and the 0.90th quantiles.

The change in the system temporal price volatility is not constant across the percentage of VRE or the quantiles and, once again, this is typically more apparent at higher quantiles. For example, in PJM, when the VRE percentage changes from 0 to 1%, the median system temporal price volatility decreases by $0.8 USD in average absolute values and the 0.90th quantile of the system temporal price volatility decreases by $5 in average absolute values; but, when the VRE percentage changes from 5 to 6%, the median system temporal price volatility decreases by about $0.2 in average absolute values and the 0.90th quantile temporal price volatility decreases by about $0.4 in average absolute values. The derivative (measure of change) of the system temporal price volatility with respect to the percentage of VRE for PJM is depicted in .

The result of the skew-t regression elucidates the variability of detrended system electricity price by estimating the energy price distribution as a function of VRE penetration. We evaluated the changes in the shape of the distribution as VRE penetration increases by investigating the differences among various quantiles. The difference between the 0.75th and 0.25th quantile measures variability about the median of detrended energy price and, as it decreases, we expect less deviation from the median. The difference between the 0.75th and 0.50th quantile and between the 0.90th and 0.50th both show changes in above-average prices, whereas the latter describes the frequency of more “extreme” prices.

In , the range of the estimated skew-t distribution quantiles for detrended system price decreases as percentage of VRE increases. The tightening of the distribution around the median detrended price as the percentage of VRE increases is a result of the decrease in estimated skewness parameter as VRE increases. To illustrate more clearly, select differences in the detrended system price quantiles are shown in . In all ISOs studied, the detrended system electricity price difference between the 0.75th and 0.50th quantile (teal) and between the 0.90th and 0.50th quantile (purple) decrease as the percentage of VRE increases. In all but two ISOs (SPP and CAISO), a reduction in the detrended system price difference between the 0.75th and 0.25th quantile (yellow) is observed as the percentage of VRE increases. From , high VRE penetration leads to negative skewness in SPP and CAISO. This negative skewness effect can also be seen in for the 0.25th quantile (yellow) with negative deviation from the median at high VRE penetrations in SPP and CAISO. The effect is most apparent in SPP where there is high frequency of negative prices at VRE penetrations above 40%.

Figure 10. Difference between 0.75th and 0.25th quantile (yellow), 0.75th and 0.50th (teal), and 0.90th and 0.50th (purple) as plotted with approximate 95% credible intervals. It is important to note that for several of the ISOs, a decrease in the spread of the distribution (volatility with respect to VRE) is observed in the quantiles as measured by all three differences, but SPP and CAISO have curvilinear trends in the difference between 0.75th and 0.25th, which is caused by the skewness changing from positive to negative with increasing VRE penetration.

Figure 10. Difference between 0.75th and 0.25th quantile (yellow), 0.75th and 0.50th (teal), and 0.90th and 0.50th (purple) as plotted with approximate 95% credible intervals. It is important to note that for several of the ISOs, a decrease in the spread of the distribution (volatility with respect to VRE) is observed in the quantiles as measured by all three differences, but SPP and CAISO have curvilinear trends in the difference between 0.75th and 0.25th, which is caused by the skewness changing from positive to negative with increasing VRE penetration.

Overall, the reductions in these multiple measures of price volatility with respect to VRE indicate that generally increased VRE penetration reduces detrended system electricity price variability, particularly through reducing extremely high electricity prices. This trend is observed in all ISOs studied for detrended system prices above the median. The exception to this trend occurs in SPP and CAISO at high penetration of VRE for detrended system prices below the median, where there is an increased frequency of negative system prices.

5. Conclusion

In this study, the relationship between detrended system electricity price, price volatility, and VRE penetration was analyzed using a robust approach to account for non-linearities and skewness. While several studies have investigated the relationship between VRE and electricity price using multivariate linear regressions, these are insufficient at adequately capturing the underlying relationships in the highly skewed data. The results not only corroborate the existing literature on the merit-order effect of VRE, which causes a reduction in electricity price with increased VRE penetration, but also illustrates that the merit order effect is nonlinear, and the greatest effect can be seen in the reductions in extremely high detrended system electricity prices (0.90th quantile) with increased VRE. In all ISOs studied, the spread in extreme high system prices, as measured by the price difference between the 0.90th and 0.50th quantiles and the 0.75th and 0.50th quantiles, reduces as the percentage of VRE increases. In all but one ISO studied, system temporal price volatility (calculated based on the time-dependent EWMA price volatility) decreased as the penetration of VRE increased across all quantiles (0.50th, 0.75th, and 0.90th).

While this study has examined the relationship between VRE penetration and the detrended system electricity price and volatility using robust techniques, it is important to comment on the bounds of validity of the study findings. The results presented in this work are valid within the studied temporal range (shown in ) and VRE penetration levels (shown in ). Given global and regional ambitious renewable-energy targets, future VRE penetration is likely to change to match these targets and therefore the relationship between VRE penetration may differ from the levels observed in the study result. Also, there are geographical, policy, and demographic factors at play within each of the ISOs studied. Direct application of the study findings should consider these interacting factors. However, the current study provides a robust holistic understanding of the relationship between VRE penetration and the system electricity price and volatility at a national scope (six out of seven US ISOs) and based on historical data, which can be adapted to future analyses of this relationship and potentially serve to inform electricity-market policy and decision making.

The results are consistent with the modern portfolio theory that shows that adding diverse and uncorrelated (or low-correlated) assets to a portfolio is associated with a reduction of total price volatility. This is a particularly preferable outcome when individual assets are highly risky (i.e. individual volatilities are large), resulting in more stable and less risky portfolios of such assets. “Diversity” is used here as a way to connect the study finding with a theoretical framework in finance, where diversity is synonymous to the increase in penetration of VRE. Also, it is important to note that the bounds of validity in relationship to diversity are based on the current VRE penetration levels and should not be extrapolated to future data. Even at that, we find that some ISOs (SPP and CAISO) show an increasing volatility at high penetration levels of VRE (relative to other regions) depending on the observed quantiles. This underscores the importance of grid-scale energy storage systems which may help to seamlessly integrate high levels of VRE, thus improving grid flexibility while allowing for reserves that can balance supply in times of high availability of stochastically variable VRE resources.

With increasing generation from VRE on the electrical grid, the use of a robust approach helps to expose the nonlinear relationships that are useful in developing an accurate understanding of behavior of system electricity price and its volatility. However, it is important to recognize that the power grid is highly dynamic and, therefore, this study should not be construed as an argument for reaching a threshold VRE penetration. The technologies, policies, and markets associated with the grid are rapidly changing. Most regions have ambitious VRE targets that will need adequate energy-storage capabilities to reduce energy curtailment and negative electricity prices, innovative markets to appropriately value demand response and auxiliary services, and mindful policies to ensure that long-term infrastructure building meets global sustainability goals.

Author Contributions

Conceptualization: MGS, DAS.

Methodology: SER, OOO, TLJS, MGS, SER, GES, SS, LW, DSM, DAS.

Investigation: OOO.

Visualization: OOO, TLJS, SER, GES, SS.

Data Curation: OOO, SER.

Formal analysis: OOO, TLJS, MGS, SER, GES, SS, LW, DAS.

Funding acquisition: MGS, LW, DSM, DAS.

Project administration: OOO, MGS, DAS.

Supervision: TLJS, MGS, LW, DSM, DAS.

Writing—original draft: OOO, TLJS, MGS, SER, SS, LW, DAS.

Writing—review & editing: OOO, TLJS, MGS, GES, SS, LW, DSM, DAS.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Raw data and code can be found here: https://github.com/Qunlexie/VRE-Impact-on-Price-and-Volatility.

Additional information

Funding

NSF Harnessing the Data Revolution (HDR) program, “Collaborative Research: Predictive Risk Investigation SysteM (PRISM) for Multi-layer Dynamic Interconnection Analysis” (#1940176, 1940223, 1940276, 2023755). U.S. DOE, “Wave Energy Technology Assessment for Optimal Grid Integration and Blue Economy Advancement” (DE-EE9443).

References

Appendix 1.

ISO-Specific Procedures

From each ISO, we collected historical data on the following information:

  1. Electricity prices: this includes locational marginal price, congestion price and losses

  2. Generation-mix data: this includes electricity generation, separated by energy source, used to satisfy demand.

This data-collection process was different for each of the ISOs because the ISOs differ in their methods of reporting market data on energy prices and operational data on their generation mix. A step-by-step of the data collection procedure for each of the ISOs is as follows:

  1. ISONE: The ISONE data was obtained from the ISONE API web services API v.1.1 ISONE (Citation2021) that gives a range of public market and energy data. The generation data was obtained for the entire system in resolutions ranging from 5 to 15 minutes and queried using [/genfuelmix] and was then aggregated to hourly resolution. The locational marginal price data was obtained for the single hub location in hourly resolutions alongside the losses and congestion cost and queried using [/hourlylmp/rt/final/day/daylocation/locationId]. The system price, referred to here as the marginal energy cost, is then obtained as: (3) MEC=LMPMCCMLC,(3) where: LMP = locational marginal price

    MEC = marginal energy costMCC = marginal congestion costMLC = marginal loss cost

  2. NYISO: NYISO provides a repository for market report and information via its public management-information system NYISO (Citation2020). The scraper was built to obtain CSV files for the 5-minute, real-time system generation in NYISO and queried using [/csv/rtfuelmix/{date}rtfuelmix_csv.zip]. The location-based marginal price, congestion cost, and loss cost were also obtained for each of the 11 hub locations in 5-minute resolutions and queried using [/csv/realtime/{date}realtime_zone_csv.zip]. We found that the four interface locations in the NYISO have significant price variations from the internal locations and were thus excluded. We then obtain the system price, referred to here as the marginal energy cost, as: (4) MEC=LBMP+MCCMLC,(4) where: LBMP = Locational Based Marginal Price

    It is important to note that price calculations in New York ISO (Citation2020) is slightly different from all the other ISOs. The 5-minute resolution data was then mean aggregated to an hourly resolution which was used in the analysis.

  3. PJM: PJM has an interactive data platform, Data Miner 2 PJM (Citation2021), where market and operational data is stored. The platform provides hourly locational marginal price, marginal loss price, and congestion price, which can be queried [/feed/rt_da_monthly_lmps]. The system real-time price was also provided directly but can also be obtained using (EquationEq. 1) above. The generation-mix data was obtained in hourly resolution for the entire PJM ISO and queried using [/feed/gen_by_fuel].

  4. MISO: MISO provides a market and operational report repository MISO (Citation2021) which we directly scraped. The generation-mix data was obtained from this repository in hourly resolution and queried using [/marketreports/{day}_sr_gfm.xls]. The locational marginal, congestion, and loss prices were also obtained for each of the hub and queried using [/marketreports/{day}_rt_lmp_final_csv.zip]. Similarly, a unique system price was calculated using (EquationEq. 1) above.

  5. SPP: SPP provides an integrated marketplace SPP (Citation2021) in the form of a file browser API platform for downloading market reports. The scraper was then built to download the provided 5-minute resolution generation data and queried using [/generation-mix-historical]. The same procedure was used to obtain the 5-minute locational marginal prices, congestion price, and the loss price and queried using [rtbm-lmp-by-location] for specified hub locations, after which a unique system price was obtained (using EquationEq. 1).

  6. CAISO: The scraper was built to obtain data from the California ISO Open Access Same-Time Information System (OASIS) API via schema California ISO (Citation2017) that use structured query parameters to fetch data from the system. The data collected include: 5-minute, real-time renewables (solar and wind) forecast by trading hub; 5-minute, real-time total generation by the entire CAISO area; and 5-minute interval local marginal price, marginal loss price, and the congestion price by trading hub. Similarly, a unique system price was calculated using (EquationEq. 1) above. The renewable-generation data were then sum-aggregated for all hub locations in order to obtain a representation for the total CAISO region.

Appendix 2.

Detrending Example

Figure A1. The three panels show an illustration of the detrending procedure for a snapshot of data from CAISO in the second half of July 2019. In (A) the raw system price is shown (dots) along with the estimated seasonal trend (red line). We detrend the raw system price by subtracting the trend to obtain the detrended system price (B). The detrended system price represents how much the system price differs from the seasonal average; note that most of the time it is near 0. From (B), we calculated the exponentially weighted moving average standard deviation (C) that is a temporal measure of the variability of the detrended system price- in other words, How much does the detrended price vary from the detrended prices in the recent past?

Figure A1. The three panels show an illustration of the detrending procedure for a snapshot of data from CAISO in the second half of July 2019. In (A) the raw system price is shown (dots) along with the estimated seasonal trend (red line). We detrend the raw system price by subtracting the trend to obtain the detrended system price (B). The detrended system price represents how much the system price differs from the seasonal average; note that most of the time it is near 0. From (B), we calculated the exponentially weighted moving average standard deviation (C) that is a temporal measure of the variability of the detrended system price- in other words, How much does the detrended price vary from the detrended prices in the recent past?

Table A1. Parameter values for the generalized skew t-distribution hourly detrended price. An example of interpretation of the effect of VRE on the centrality and skewness of the distribution of hourly price is an increase in VRE decreases both centrality and skewness of hourly price in CAISO (β1μ<0 and β1ν<0). The interpretation also holds for ISONE, NYISO and SPP.