AI-based forecasting for optimised solar energy management and smart grid efficiency

Abstract

This paper considers two pertinent research inquiries: ‘Can an AI-based predictive framework be utilised for the optimisation of solar energy management?’ and ‘What are the ways in which the AI-based predictive framework can be integrated within the Smart Grid infrastructure to improve grid reliability and efficiency?’ The study deploys a Deep Learning model based on Long Short-Term Memory techniques, leading to refined accuracy in solar electricity generation forecasts. Such an AI-supported methodology aids power grid operators in comprehensive planning, thereby ensuring a robust electricity supply. The effectiveness of this framework is tested using performance metrics such as MAE, RMSE, nMAE, nRMSE, and $R^2$. A persistent model is utilised as a reference for comparison. Despite a slight decrease in predictive precision with the expansion of the forecast horizon, the proposed AI-based framework consistently surpasses the persistent model, particularly for horizons beyond two hours. Therefore, this research underscores the potential of AI-based prediction in fostering efficient solar energy management and enhancing Smart Grid reliability and efficiency.

Keywords:

• Artificial intelligence
• shortage economy
• energy operations
• deep learning
• smart grids
• renewable energy

1. Introduction

A year following the onset of the War in Ukraine, the worldwide energy context has undergone significant alterations, with soaring energy prices impacting consumers globally amidst a geopolitical scenario where energy security is paramount. The immediate surge in energy prices and heightened energy security issues have further spotlighted the necessity for a transition towards more sustainable and geopolitically stable energy sources (IEA 2022). The economic upheaval sparked by the conflict has heightened calls for an expedited transition to sustainable energy. Such a transition would entail a shift from highly polluting fuels, which are typically supplied by a few large-scale producers, towards low-carbon and more decentralised energy alternatives, including renewables (Lorente et al. 2023; Umar, Riaz, and Yousaf 2022). This transition is particularly pertinent in Europe, which has acutely experienced the reverberations of the conflict and where gas has traditionally been a mainstay of imports. Although a mild winter of 2022–2023, there was a serious threat that the European Union could encounter a deficit of nearly 30 billion cubic meters of natural gas (IEA 2023). Amidst these security concerns, the transition to clean energy has gained momentum. The Commission's strategy intends to diversify supplies of 101.5 billion cubic meters of gas. The plan implies that nearly 60% of this reduction could be achieved by boosting imports to Europe from other nations. However, an additional 33% could be realised through new renewable energy generation and conservation initiatives (Tollefson 2022).

Even before the war, the occurrence of energy shortages was surprisingly frequent, underscoring the inherent fragility of the global energy infrastructure (Ivanov and Dolgui 2022; Ivanov et al. 2023; Ivanov and Keskin 2023). This vulnerability was demonstrated dramatically in the power outage incidents of February 2021 in Texas (Sugg et al. 2023) and later in the year in the provinces of Heilongjiang, Jilin, and Liaoning in China (Jingyi and Yin 2021). These energy disruptions had significant repercussions on societal functionality and supply chain robustness. For instance, the Texas blackout alone resulted in an estimated economic loss of $130 billion, reflecting the costs of lost output and collateral damages (Busby et al. 2021). Moreover, these energy deficits, coupled with an accelerated price surge, triggered a pivotal need for supply chains to reorient and mitigate the impending risks of hyperinflation (IMF 2022). A further exacerbation of the crisis has been the extraordinary shortage of transportation capacities experienced globally between 2020 and 2022. This scarcity has led to an escalation in freight prices, product scarcities, and overall destabilisation of supply chains, all of which profoundly affect the energy sector, where the transportation of energy resources forms a critical component of the supply chain (Ivanov and Dolgui 2022). These empirical realities, when intersected with the growing threats of climate change (Obersteiner et al. 2001; Rising et al. 2022; Stern and Taylor 2007), ongoing war in Ukraine (Tollefson 2022), and risks of hyperinflation (IMF 2022), underscore the imperative for novel solutions that have potential to promote the efficient utilisation of energy resources.

The path towards a solution necessitates a substantial enhancement in energy production capacity from renewable sources. This transition involves constructing and implementing new wind and solar farms, hydroelectric power stations, and nuclear plants, as well as developing innovative models and algorithms for superior energy management. While our study doesn't delve into energy generation capabilities, it is essential to underscore the conclusions of recent research (S. Wang et al. 2023). This paper indicates that the projected demands for power sector generation materials across a broad spectrum of climate-energy scenarios necessitate a considerable augmentation in the production of specific commodities. However, it is reassuring that geological reserves are anticipated to suffice to meet future needs, even for the most critical and scarce components such as neodymium, dysprosium, tellurium, fiberglass, and solar-grade polysilicon. Notably, the estimated environmental impacts associated with the extraction and processing of these commodities are projected to be minimal (S. Wang et al. 2023). Focusing our lens more narrowly, our study places particular emphasis on the management of solar energy, envisaged as a leading energy source in the future energy mix (Adeh et al. 2019; Mercure et al. 2021). In this context, Artificial Intelligence (AI) in general and deep learning, in particular, emerge as a promising technology with significant potential to revolutionise solar energy management, primarily through the provision of accurate forecasts (Alam et al. 2022; Rai et al. 2021). In this regard, we postulate the following research questions. RQ1: ‘How can an AI-based forecasting framework be leveraged to optimise solar energy management?’. RQ2: ‘How can the AI-based forecasting framework be effectively implemented within the Smart Grid ecosystem to enhance grid reliability and efficiency?’.

The primary contribution of our paper is an exploration of an AI-based forecasting framework for enhanced solar power integration into grids. Using deep learning, we offer an approach for accurate solar electricity forecasting, enabling effective planning and stable electricity supply, thereby reducing reliance on non-renewable energy. This work illuminates the utility of AI in advancing sustainable energy and a greener future. Moreover, our paper probes the practical application of this framework within the Smart Grid, crucial for boosting grid reliability and efficiency. Our work bridges the theory-practice divide, offering insights on real-world AI application in grid systems.

In addition to the findings presented in this paper, we also make the following tangible contributions that would benefit the wider scientific and research community. We provide an extensive, cleaned dataset that contains a total of 17,297,280 measurements. The dataset encapsulates the Global Horizontal Irradiance, measured with a pyranometer at a sampling frequency of 10 seconds from January 1, 2016, until November 1, 2021. This comprehensive dataset, filtered out of noise and inconsistencies, presents a valuable resource for future research on solar radiation, climate modelling, and renewable energy forecasting. The cleaned dataset, along with its original, unprocessed version, is openly available. The cleaned dataset can be accessed via our GitHub repository.¹ This repository also contains the source code of our deep learning models and pipelines, which are used to analyse the given dataset and arrive at the conclusions presented in the paper. Providing the community with the source code enables other researchers to reproduce our study's results, validate our findings, and extend our work for future studies. Besides, the raw data can be accessed on the GitHub repository of the Distributed Electrical Systems Laboratory – Power Systems group at the Swiss Federal Institute of Technology Lausanne (EPFL).² These contributions aim at enhancing reproducibility and collaboration within the research community, aligning with the principles of open science. This fact ensures the transparency of our study and provides a testbed for future studies to build upon our work. We hope that the dataset, along with the deep learning models and pipelines, will be valuable tools for researchers, academics, and professionals in similar and intersecting domains.

At this point, it is essential to define such terms as AI, deep learning and Smart Grid because they are pivotal for our research and crucial for further reading. These definitions are preliminary and will be further formally unfolded in the following sections. It is essential to emphasise that in this study, we take an instrumentalist view of AI (Bostrom 2014), and do not speculate on AI cognition or the philosophical aspects. The definition of AI proposed by Arthur Samuel suits well in the context of our study. ‘AI is the field of study that gives computers the ability to learn by using sampled data without being explicitly programmed.’ (Samuel 1959). Deep learning, in turn, is a branch of AI that refers to computational models designed to learn and represent data through the implementation of layered structures of algorithms, commonly known as ‘deep neural networks’. These multilayered structures enable the model to learn and abstract high-level features from raw input data (Goodfellow, Bengio, and Courville 2016). A Smart Grid ecosystem refers to the interconnected network of various components involved in the generation, distribution, and consumption of electricity, which utilise advanced communication, automation, and IT systems to improve the efficiency, reliability, and sustainability of electricity services (Dileep 2020).

The remainder of the paper is organised as follows. In Section 2, we review the literature and position our paper among the related works. In Section 3, we describe the technologies behind the proposed forecasting framework and specify the deep learning architecture. Section 4 benchmarks the experimental results. Section 5 delves deeper into the critical facets of the AI-based framework for solar forecasting, and its potential use cases and implementation within the Smart Grid ecosystem. Section 6 provides the summary of important insights as well as outlines the directions for future research.

2. Literature review

This section reviews the solar forecasting literature, particularly global horizontal irradiance (GHI) at multiple horizons ranging from minutes ahead to days ahead, and focuses on the recent research motivated by the integration of solar panels in Smart Grids using a data-driven approach. This section also introduces the concept of a Smart Grid ecosystem and explains all the critical players involved.

2.1. Approaches to solar energy forecasting

Antonanzas et al. (2016) reviewed photovoltaic power forecasting to determine which techniques obtain better results based on the temporal and spatial horizons. Table 1 summarises and extends the proposed categories for forecasting techniques, namely: statistical, sky and satellite image, and Numerical Weather Prediction (NWP). Statistical methods include regressive approaches such as the Auto Regressive Moving Average (ARMA) model and the Auto-Regressive Integrated Moving Average model (ARIMA) (Pedro and Coimbra 2012) based on time-series of solar irradiance (Chu et al. 2014). Another example of statistical methods is models based on AI, such as the studies on solar forecasting using Support Vector Regression Machines (Rana, Koprinska, and Agelidis 2015) and Artificial Neural Networks models (Pang, Niu, and O'Neill 2020).

Table 1. Summary of techniques for solar forecasting and their use-cases.

Forecasting horizon	Electricity market	Captured behaviour	Forecasting models
Minutes-ahead (15mn to 1h)	Intra-day	Short-term fluctuations	Statistical
			Sky imagery
			Satellite imagery
Hours-ahead (1h to 24h)	Intra-day	Daily seasonality	Statistical
			Satellite imagery
			NWP
Day-ahead (1 day to 7 days)	Day-ahead	Trend	Statistical
			NWP

Satellite and sky images are image-based techniques that compute Cloud Motion Vectors (CMVs) to model the displacements of clouds. The main difference lies in the origin of the images: Satellite images come from space, whereas sky images are taken from the ground. Satellite images allow longer horizon forecasts (30min-10h), whereas ground images allow forecasts from (1mn to 45min). Cloud motion vectors allow the creation of Atmospheric Motion Vectors (AMVs), which include data about cloud movements, their type, and density. AMV can forecast GHI at a given location using multiple approaches. Some examples are Support Vector Machines (SVMs) (Jang et al. 2016), deep convolutional neural networks (CNNs) (Feng and Zhang 2020), and probabilistic approaches (Jang et al. 2016).

Numerical weather prediction (NWP) is appropriate for forecasting horizons between 1 hour and 3 days. It can be customised to consider specific parameters and has been used in multiple pieces of research. The reference NWP model is the Weather Research and Forecasting (WRF) model, which was developed and released in 2000 by the National Center for Atmospheric Research (NCAR) in the USA to model atmospheric phenomena (Powers et al. 2017). Its accuracy was evaluated for GHI forecasts made 1-day, 2-days, and 3-days ahead in Andalusia and reached a mean error of 30% for horizons between 1–3 days (Lara-Fanego et al. 2012). In 2019, the WRF was modified to fit urban mesoscale areas and named the urban weather research and forecasting (uWRF) model (Gamarro, Gonzalez, and Ortiz 2019). It has two specific parametrizations: a multilayer urban parametrization (Martilli, Clappier, and Rotach 2002) and a building energy model parametrization (Salamanca et al. 2009). The uWRF was tested to forecast the GHI 24 hours ahead in five areas in New York City and compared with five other nonurban areas in New Jersey. The results were better in urban areas with clear sky conditions and mean error overall conditions of 45%.

2.2. Machine learning models for solar energy forecasting

Solar electricity production is volatile, dependent on the weather, and mainly connected at the regional level, making it less predictable for Independent System Operators (ISOs). Recent works have shown that Artificial Intelligence and Machine Learning (ML) algorithms can provide a promising solution for accurate solar forecasting. Solar forecasting using ML has already been employed by ISOs and has resulted in significant improvements of up to 30% more accuracy (National Grid ESO 2019). Since energy markets worldwide are considering how to align with net-zero carbon emissions objectives, having accurate forecasting techniques in various time horizons plays an essential role in the transition to a sustainable electricity system (ESO 2021).

There are also a number of review studies that have focused on different AI models and techniques in the area of energy conservation and renewable energy, especially solar and hybrid systems. Al-falahi, Jayasinghe, and Enshaei (2017) provide a review on size optimisation methodologies for standalone solar and wind hybrid renewable energy systems. Akhter et al. (2019) review research studies addressing the forecasting of photovoltaic power generation based on ML and metaheuristic techniques. To expand on this review Alsadi and Khatib (2018) focus on a variety of criteria, constrains, models, techniques, and software tools with respect to the research status of the optimisation of photovoltaic power systems. Other review studies (e.g. Akhter et al. 2019; Alsadi and Khatib 2018; Das et al. 2018; Khan, Pal, and Saeed 2018; Sobri, Koohi-Kamali, and Rahim 2018) have tended to complement or overlap with previous work on photovoltaic and/or hybrid power system optimisation. From a general perspective, according to Nishant, Kennedy, and Corbett (2020), only between 2015 and 2019, 250 studies addressed the application of AI in the area of energy conservation and renewable energy. In a more recent study on AI applications for smarter eco-cities in the field of environmental sustainability, synthesise a range of empirical and theoretical evidence in this area, where ML algorithms (e.g. ANN, SVM, Decision Trees (DT), Evolutionary Computing (EC), and Batch-Normalisation (BN) are mainly used for energy forecasting and optimisation. Other algorithms for decision support (e.g. Fuzzy Logic (FL), Adaptive Neuro-Fuzzy Inference System (ANFIS), Expert Systems (ES), hybrid models) have also been used together with ML algorithms for energy production, distribution, operation, maintenance, and planning

In this paper, we focused on using ML algorithms (including Deep Learning) for forecasting GHI time-series. These algorithms have been at the heart of many types of solar forecasting research over the past 15 years, since they make it possible to replace classic theoretical models, such as NWP, with empirical models based on past data. The most common ML models are (1) GPR (Gaussian Process Regression) which performs a regression using multiple Gaussian distributions; (2) SVR (Support Vector Regression) which performs a regression by maximising support vector distances to find the most accurate regression fit; and (3) ANN (Artificial Neural Networks) which are part of Deep Learning and rely on neurons to perform regressions. In this study, we focused on using a sub-field of ML called Deep Learning and structures called long short term memory (LSTM) which have shown promising result (Gbémou et al. 2021).

Table 2 compares our paper to other works presented in the review by Gbémou et al. (2021) in terms of forecast horizons, forecasting methods, data timestep, performance criteria, input variables, and original database.

Table 2. Summary of recent ML works and their technical characteristic for GHI forecasting.

Auhors	Forecast Horizons*	Input Variables	Data	Data Time Step**	Performance metrics	Forecasting models
Sharifzadeh, Sikinioti-Lock, and Shah (2019)	Hours-ahead (1h – 6 h)	GHI, DNI, DH	3 years	1 h	RMSE, nRMSE, MAE, nMAE	Scaled persistence, MLP, Random forest
Chandola et al. (2020)	Hours-ahead (3h – 24 h)	DHI, DNI, dew point, temperature, pressure, relative humidity, wind direction, wind speed	4 years	3 h	RMSE, MAPE	LSTM
Tolba et al. (2020)	Minutes-ahead Hours-ahead (30min – 48h)	Time	45 days	30 min	nRMSE	Persistence, GPR
Gbémou et al. (2021)	Minutes-ahead Hours-ahead (10min – 4h)	GHI	2 years	10 min	nRMSE, DMAE, CWC	Persistence, GPR
Our AI-Based Forecasting Framework	Minutes-ahead Hours-ahead Days-ahead (15mn – 7days)	GHI, GHIcs, kcs 1st, 2nd, 3rd derivatives Seasonal components history, polynomials	6 years	10s	RMSE, nRMSE, MAE, nMAE 95% confidence interval, $R^2$	Persistence, LSTM

${}^{\ast }$The duration of forecast horizons is provided in the parenthesis.${}^{\ast \ast }$Data time step, or measurement frequency, ranging from seconds to hours

2.3. Smart grid ecosystem

Electric power, generated from various energy sources, is conveyed to end-users via an intricate network of power plants and high-voltage transmission lines, commonly known as the power grid. Historically, this system has been dominated by two principal actors – the producers and consumers. However, the advent and proliferation of photovoltaic (PV) installations, driven by technological advancements, have given birth to a new paradigm of the Smart Grid ecosystem. This new framework introduces ‘prosumers,’ who simultaneously produce, buy, and sell energy, thus altering the traditional dynamics of the power grid (Gautier, Jacqmin, and Poudou 2018). Such a Smart Grid ecosystem, encompassing both transmission and distribution networks, is an ever-evolving, complex, and dynamic entity, continually adapting to emerging technologies and the entry of new market participants. The Smart Grid facilitates real-time interaction between producers and consumers, promoting a more efficient and reliable electricity system. Within this ecosystem, an array of devices interconnects producers and consumers, enabling data collection and dissemination. This data is instrumental for producers in electricity generation and for consumers in energy management, regulating usage across lighting, appliances, and industrial equipment. A simplified representation of the smart power grid ecosystem is presented in Figure 1. The subsequent subsections elaborate on the primary components and participants of the Smart Grid ecosystem. For a comprehensive review of the electric power industry's current operational and policy challenges, refer to Parker, Tan, and Kazan (2019).

Figure 1. Simplified smart power grid ecosystem.

Simplified representation of the smart power grid ecosystem and its critical elements.

2.3.1. Electricity production by PV panels

The photovoltaic effect, a physical and chemical phenomenon, causes certain materials to generate electricity upon exposure to light, a principle that forms the basis of photovoltaic (PV) panel operation. PV panels consist of cells that convert the energy of photons, or irradiance, from the sun into electricity. A crucial determinant of PV panel performance is its efficiency, which signifies the ratio of sunlight that is transformed into electricity (Palz 2010).

Although the U.S. Department of Energy's National Renewable Energy Laboratory researchers have recently crafted a solar cell boasting a record efficiency of 39.5% (France et al. 2022), the practical peak efficiency generally remains around 20% (Green et al. 2022). This is largely due to the fact that the bulk of the received irradiance is either reflected or turned into heat, with only a minor fraction being converted into electricity (Elibol et al. 2017). The electricity generated by a PV panel, typically expressed in watts per square meter (W/m2), denotes the amount of electricity produced per unit area of the panel. This quantity depends on two primary factors: 1) the panel's efficiency, which is influenced by operating conditions such as angle, humidity, and temperature; and 2) the amount of sunlight striking the panel, determined by factors such as weather, geographic location, and time of day (Palz 2010).

Ramos-Hernanz et al. (2020) have provided an extensive review of PV panels' characteristic curve and proposed an optimal efficiency curve. However, the amount of sunlight that reaches the PV panels remains a variable factor, significantly impacting the overall electricity production.

2.3.2. Prosumers

A prosumer is an individual or organisation (industry or company) that is both a producer and a consumer of goods and services. The term was coined by Toffler (1980), to describe how technological advancement is blurring the lines between production and consumption, turning consumers into consumer-producers. In the Smart Grid context, a prosumer is a consumer who produces and generates their own electricity, typically through rooftop PV panels or wind power, and then injects and sells any excess back to the Smart Grid.

Prosumers act as independent power plants in a smart power grid ecosystem, and provide several advantages. First, prosumers help to increase the overall amount of renewable energy being used, since they are more likely to use green energy sources. Second, prosumers help to create a more resilient grid, since they can provide backup power in case of a grid outage. Finally, they can help reduce energy costs for everyone involved, since they can sell their surplus power back to the grid for less than the retail price.

However, having prosumers in the smart grid ecosystem can create new challenges for managing electrical power grids. Prosumers' electricity production may vary depending on the weather, location, and time of day. This production variability and information distortion can create instability and potentially generate a bullwhip effect (Lee, Padmanabhan, and Whang 2004), in the power grid. Thus, the importance of accurately forecasting prosumers' electricity production is paramount.

2.3.3. Electricity market

In the electricity market (the shaded area of Figure 1), electricity is traded via bilateral contracts (Over The Counter, or OTC), power exchange, and electricity pool. It is a wholesale market, in which electricity is bought and sold by producers, consumers, and prosumers (Kettunen, Nematollahi, and Zinchenko 2022).

The electricity market is a unique commodity market with strict rules for the quantity exchanged and a specific delivery time. In Europe, the European Power Exchange (EPEX), the largest electricity exchange platform, operates across multiple countries in two main markets: the intra-day and the day-ahead markets.

Auctions are one type of transaction used to buy and sell electricity in a short period of time. They are typically used in intra-day and day-ahead markets, where electricity is traded for use on the same day or the next day. There are multiple country-specific intra-day auctions to facilitate trading with intermittent renewable resources. These auctions have a shorter timescale to optimise the short-term market (for example, 15 minutes in Germany); allow players with renewable assets (for example, solar and wind) to improve their position based on their most recent forecast.

Market players are responsible for their final position regarding the volume of energy sold or bought. Take the example from the Subsection 2.3.2 to better understand the settlement process. The prosumer was supposed to deliver 1kWh and only delivered 0.7kWh, so they have to settle the discrepancy of 0.3kWh. This discrepancy is the gap between what the prosumer has contracted to generate or consume and what they did generate or consume. Thus, the prosumer has to pay the energy imbalance price to settle this discrepancy (Elexon 2019).

2.3.4. Independent system operators

An Independent System Operator (Figure 1) is a non-profit organisation responsible for coordinating and controlling the operation of the smart power grid ecosystem and managing the electric power market. ISOs are at the heart of the transition to a sustainable electricity ecosystem and play a crucial role in maintaining the security of the whole network and balancing supply and demand second by second every day of the year.

As technology advances and there is a move toward net-zero carbon emissions targets, more prosumers are joining the Smart Grid ecosystem, and more and more PV panels are contributing to the electricity market. This makes forecasting the electricity production and supply of prosumers more challenging. To overcome this challenge and ensure the electricity system runs smoothly, dedicated teams within ISOs make supply and demand forecasts and use multiple scheduling and planning processes, from real time to a year ahead. ISOs produce solar forecasts at different time horizons to securely plan and operate the grid: (1) minute(s) ahead; (2) Hour(s) ahead; and (3) Day(s) ahead. Inaccurate forecasts result in increased requirements for balancing services and costs. Inaccuracy in the supply and demand forecast directly impacts consumers' and prosumers' bills as well as the incentives that ISOs receive from regulators to decrease the balancing costs.

2.3.5. Current challenges

The rising significance of solar electricity in the global energy landscape necessitates reliable forecasting of its production and supply. Such forecasts are invaluable to policymakers, utilities, and other stakeholders as they shape the trajectory of solar energy (Parker, Tan, and Kazan 2019). Moreover, understanding the variability of solar power through these forecasts is critical to managing its impact on the grid.

However, given the intermittent and weather-dependent nature of solar electricity, combined with its regionalised connections, forecasting future output presents a formidable challenge for ISOs and prosumers alike. This challenge directly links to the first research question (RQ1). In the context of global energy markets undergoing revisions to align with net-zero carbon emission goals, precise forecasting techniques across various time horizons are instrumental for efficient power grid operation and the transition to a sustainable electricity system (ESO 2021). This fact introduces the relevance of the second research question (RQ2).

3. Methodology

This section explores the research methodology, beginning with the Supply Forecast Framework for energy prediction. We then outline the dataset's origins, content, and preprocessing steps. Next, we describe the application of Deep Learning for Supply Forecasting via the LSTM model and its architecture. The feature selection process refining model input is detailed, concluding with model validation assessing forecast accuracy.

3.1. Energy supply forecasting

Figure 2 illustrates the proposed supply forecasting framework. As depicted in step 1, our goal was to provide energy supply forecasting measured as Global Horizontal Irradiance (GHI) for PV panels. Moreover, we used an extensive range of forecasting horizons ranging from minutes-ahead (15 min to 1 hour) to hours ahead (1 to 24 hours) to days ahead (1 to 7 days). Considering an extensive range allows decision makers such as prosumers, service providers/aggregators, and producers to better forecast intra-day and day-ahead bidding and trading on the electricity market.

Figure 2. Proposed Supply Forecasting Framework.

Simplified representation of the proposed framework for solar energy forecasting.

The GHI data was obtained from a pyranometer installed on the PV panels and stored on a server as raw data. As shown in step 2, the measurement frequency was 10 seconds. In step 3, the raw data was pre-processed, put in our Deep Learning models, and post-processed to output GHI forecasts with horizons ranging from 15 minutes to 7 days ahead. In step 4, our model displays the forecasted mean irradiance during the next daylight hours of the given horizon. For example, if we look at a horizon of 4 days ahead, the model predicts the mean irradiance from sunrise to sunset for the next 4 days. As the last step, we used time-series cross-validation and performance metrics to evaluate and measure the accuracy of the proposed Deep Learning model.

3.2. Data

We used data measurements from January 1, 2016 until November 1, 2021 on the roof of the Swiss Federal Institute of Technology in Lausanne (EPFL) at an altitude of 400 meters (46.518${}^{\circ }$N, 6.565${}^{\circ }$E). The GHI was measured with a pyranometer with a time resolution of 10 seconds (Apogee Instruments, Inc 2021). Measurements of almost six years of GHI data with a sampling frequency of 10 seconds provided us with a total dataset of 17,297,280 measurements.

3.2.1. Global horizontal irradiance data

The usual metric for measuring irradiance on PV panels is Global Horizontal Irradiance Data (GHI). It measures the irradiance received by a horizontal plane in watts per square meter (W/m${}^2$). Although the electricity production of PV panels is influenced by many factors such as solar irradiance, cell temperature and incidence angle (Tolba et al. 2020). The solar irradiance, or GHI, hitting the PV panel is the essential component to consider for solar forecasting (Dinçer and Mera 2010) since every PV system (set of PV panels) is unique and depends on the number of PV panels installed, brand, and location. Moreover, due to the universality of GHI, it can be used for any type of PV panels, regardless of their characteristics (Engerer and Mills 2014), and it can be integrated into models for electricity supply forecasting (Ibrahim et al. 2015).

3.2.2. Data cleaning and wrangling

The stored raw data, obtained by the pyranometer, cannot be processed straight away. It often contains missing data which are recorded as not available numbers (NaNs) and outliers. Missing values may arise from maintenance operations on the server, power outages, or server errors. Outliers of GHI measurements may come from different sources such as leaves falling on the sensor and/or operators performing maintenance on the cell while the pyranometer is capturing data. To clean and wrangle the data, we took the following eight steps:

• Step 1 – Removal of missing values: There are two types of missing values within the dataset. The first type are isolated and nonconsecutive missing values which occur due to errors in the registration of measurements in the database. We treated this type of missing values by replacing them with an interpolation of the previous and following values. The second type of missing values are a high density of consecutive missing values, which occur due to maintenance and sensor adjustments on the roof. We eliminated the measurement entries from the dataset when we observed consecutive missing values.

• Step 2 – Identification and removal of outliers: The upper and lower limits of GHI data were set to 1000 W/m${}^2$ (maximum terrestrial normal surface irradiance) and 0W/$m^2$ respectively (Boxwell 2012). We removed any value outside this range and considered it as an outlier.

• Step 3 – Clear sky global horizontal irradiance ($\mathrm{GH}{I}_{\mathrm{cs}}$): We added $\mathrm{GH}{I}_{\mathrm{cs}}$ to our dataset to enhance the accuracy of the supply forecasting. $\mathrm{GH}{I}_{\mathrm{cs}}$ is the irradiance on a horizontal surface in clear sky conditions (i.e.without cloud cover) and can be calculated by the Ineichen and Perez model (Perez et al. 1990). We calculate $\mathrm{GH}{I}_{\mathrm{cs}}$ with Python and the pvlib library (Holmgren, Hansen, and Mikofski 2018).

• Step 4 – Removal of night measurements: We removed GHI data measured at night from the dataset since no GHI hits PV panels then and there is no power production. Moreover, having many constant values as part of the training set and test set could affect the performance of our model by creating a bias.

• Step 5 – Clear sky index ($k_{\mathrm{cs}}$): GHI has very high variations due to deterministic components (e.g. rotation of the earth) and stochastic components (e.g. cloud movement). Thus, we introduced the clear sky index $k_{\mathrm{cs}}$ to eliminate the deterministic components and isolate the stochastic components (Paulescu, Paulescu, and Badescu 2021). $k_{\mathrm{cs}}$ is the ratio between GHI and $\mathrm{GH}{I}_{\mathrm{cs}}$ (Equation 1(1) $k_{\mathrm{cs}}=\frac{\mathrm{GHI}}{\mathrm{GH}{I}_{\mathrm{cs}}}$(1) ). (1) $k_{\mathrm{cs}}=\frac{\mathrm{GHI}}{\mathrm{GH}{I}_{\mathrm{cs}}}$(1)

• Step 6 – Calculation of finite-difference: We introduced derivatives of GHI, $\mathrm{GH}{I}_{\mathrm{cs}}$ and $k_{\mathrm{cs}}$ to capture their fluctuations and their pace of fluctuations (F. Wang et al. 2012). Thus, the first-order, second-order, and third-order derivatives of each parameter are calculated using backward finite differences (Olver 2014).

• Step 7 – Adding seasonality: GHI data has seasonal and cyclic patterns. For example, peak GHI is reached in the middle of a sunny day, and GHI data has higher peaks on summer days than in winter. To integrate seasonality, we included the following features in the dataset: (1) hours and minutes, (2) the number of the day within a year, and (3) the number of the month. We then modelled these parameters using half sinusoid to capture GHI seasonality and cyclic behaviours.

• Step 8 – Sampling:Since we have 10 second measurements for GHI, we used downsampling (Pfenninger 2017) to calculate the mean value over the given forecasting horizon. Thus, we downsampled the dataset to cover three forecasting horizons, ranging from minutes ahead (15 minutes to 1 hour) to hours ahead (1 to 24 hours) to days ahead (1 to 7 days). Therefore, we obtained 15 sub-datasets to train and test the Deep Learning model.

3.2.3. Input and output datasets

After performing the data cleaning and wrangling process, we obtained input and output datasets. The input dataset was sampled every 10 seconds. As listed below, each measurement has five feature groups, including three features. Thus, each measurement of the input dataset has overall 15 features.

• Group 1 – First derivative $\frac{d^{(1)}}{\mathrm{dt}}$: $\dot{\mathrm{GHI}}$, $\dot{\mathrm{GH}{I}_{\mathrm{cs}}}$, ${\dot{k}}_{\mathrm{cs}}$

• Group 2 – Second derivative $\frac{d^{(2)}}{\mathrm{dt}}$: $\ddot{\mathrm{GHI}}$, $\ddot{\mathrm{GH}{I}_{\mathrm{cs}}}$, ${\ddot{k}}_{\mathrm{cs}}$

• Group 3 – Third derivative: $\frac{d^{(3)}}{\mathrm{dt}}$: $\stackrel{⃛}{\mathrm{GHI}}$, $\stackrel{⃛}{\mathrm{GH}{I}_{\mathrm{cs}}}$, ${\stackrel{⃛}{k}}_{\mathrm{cs}}$

• Group 4 – Basic input: GHI, $\mathrm{GH}{I}_{\mathrm{cs}}$, $k_{\mathrm{cs}}$

• Group 5 -Seasonal features: $C_{\mathrm{hour}}$, $C_{\mathrm{day}}$, $C_{\mathrm{year}}$

The output datasets, or labels, of the Deep Learning model, are solely made of measurement of GHI over forecasting horizons ranging from minutes ahead (15mn to 1 hour), to hours ahead (1 to 24 hours), and to days ahead (1 to 7 days).

3.3. Validation

We measure model performance using time-series cross-validation (Kohavi 1995) to ensure model robustness across different forecasting horizons. More specifically, we trained the Deep Learning model ten times using the feature sets presented in Subsection 3. We evaluated the Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and $R^2$ for each cross-validation run of the forecasting horizon. Also, we monitored standard deviations of these performance metrics over the cross-validation runs to ensure there was no significant variation.

Table 3. Feature Selection Based on Grid Search.

	Minutes ahead				Hours ahead					Days ahead
	15mn	30mn	45mn	1h	2h	4h	6h	12h	24h	2d	3d	4d	5d	6d	7d
Group 1 – $d^1$								✓	✓		✓	✓	✓
Group 2 – $d^2$									✓		✓
Group 3 – $d^3$									✓
Group 4 – GHI	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Group 5 – Season		✓	✓		✓	✓	✓	✓	✓	✓		✓	✓	✓	✓
Polynomial Order	5	3	5	5	4	5	5	1	2	4	1	1	3	4	4
History (h)	5	5	5	5	10	5	5	5	5	5	15	5	10	10	10

3.4. Deep learning for energy supply forecasting

This section presents the three main steps taken to develop our proposed Deep Learning model. Subsection 3.4.1 provides details about pre-processing, a necessary step to prepare and put the dataset in the correct format for the Deep Learning model. Subsection 3.4.2 explains what LSTM networks are and how we leverage them for supply forecasting.

3.4.1. Pre-processing

The preprocessing shapes the data to optimise the training and forecast efficiency of the Deep Learning model. The following steps are performed:

• Step 1 – Feature selection: Each forecasting horizon has specific features associated with it to provide the best possible result. This step keeps only the features of interest and discards the others. The feature selection process is detailed in Subsection 3.5 and a summary of features per horizon is shown in Table 3.

• Step 2 – Polynomial feature augmentation: A common practice in Deep Learning approaches is to create new features through polynomial augmentation. This consists of creating new non-linear feature spaces based on a basic set of features to enhance the Deep Learning process. This step was performed on the whole dataset with the PolynomialFeatures function from scikit-learn (Pedregosa et al. 2011). The choice of polynomial order for each horizon is detailed in Subsection 3.5.

• Step 3 – Standardisation and scaling: The input dataset is standardised and scaled between [0,1] to optimise computational complexity and performance. We used the MinMaxScaler function from sci-kit learn (Pedregosa et al. 2011) to perform this procedure.

• Step 4 – History: The LSTM architecture uses past data to learn from it and forecast to the next timestep $T_{t+1}$. Therefore, data from $T_{t-h}$ to $T_t$ are concatenated together. Subsection 3.5 provides the rationale behind the choice of history, horizon h, in this paper.

• Step 5 – Deleting of missing values: As explained in step 4, we used history made of h previous measurements and concatenated them to the input. Thus, the h oldest entries are incomplete and contain missing values. Therefore, we removed all of these entries.

3.4.2. Deep learning model architecture

LSTMs are a type of recurrent neural network (RNN) architecture that are especially useful for dealing with long-term dependencies in sequential data like time series. They were introduced by Hochreiter and Schmidhuber (1997) to solve the vanishing gradient problem, where the contribution of information decays geometrically over time, making learning long-term dependencies practically impossible. LSTM networks are an optimised version of RNNs for long-time sequences. LSTM is a type of RNN architecture that includes memory neurons in addition to the classic architecture (Fleuret 2021). Memory neurons serve as an internal memory that stores information from previous calculations to build long-term temporal dependencies. A single LSTM cell can be represented in compact form using just six equations (see the Equations below). (2) $\begin{array}{rl}{f}_t& ={\sigma }_g(W_f{x}_t+U_f{h}_{t-1}+b_f)\end{array}$(2) (3) $\begin{array}{rl}{i}_t& ={\sigma }_g(W_i{x}_t+U_i{h}_{t-1}+b_i)\end{array}$(3) (4) $\begin{array}{rl}{o}_t& ={\sigma }_g(W_o{x}_t+U_o{h}_{t-1}+b_o)\end{array}$(4) (5) $\begin{array}{rl}{\tilde{c}}_t& ={\sigma }_c(W_c{x}_t+U_c{h}_{t-1}+b_c)\end{array}$(5) (6) $\begin{array}{rl}{c}_t& =f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t\end{array}$(6) (7) $\begin{array}{rl}{h}_t& =o_t\odot {\sigma }_h(c_t)\end{array}$(7) where $x_t\in {\mathbb{R}}^d$ is an input vector to the LSTM unit. $f_t\in {(0,1)}^h$ denotes the forget gate's activation vector. $i_t\in {(0,1)}^h$ is respectively the input gate's activation vector. $o_t\in {(0,1)}^h$ is the output gate's activation vector. $h_t\in {(-1,1)}^h$ is a hidden state vector also known as output vector of the LSTM unit. ${\tilde{c}}_t\in {(-1,1)}^h$ corresponds to cell input activation vector. $c_t\in {\mathbb{R}}^h$ is cell state vector. $W\in {\mathbb{R}}^{h\times d}$, $U\in {\mathbb{R}}^{h\times h}$ and $b\in {\mathbb{R}}^h$ are weight matrices and bias vector parameters which have to be learned during the training phase. The superscripts d and h refer to the number of input features and hidden units and the subscript t indicates the time step. ${\sigma }_g$ is a sigmoid activation function. ${\sigma }_c$ and ${\sigma }_h$ denote hyperbolic tangent activation functions (Gers, Schmidhuber, and Cummins 2000). The initial values are set to $c_0=0$ and $h_0=0$ and the operator ⊙ corresponds to the Hadamard product (element-wise product).

Figure 3 illustrates the underlying relations between the elements of the LSTM cell. LSTM cell makes decisions about what to forget from the previous cell state, what to update from the current input, and what to output as the hidden state, making them capable of learning long-term dependencies.

Figure 3. LSTM Cell and its core components. Adapted from Graves, Jaitly, and Mohamed (2013).

LSTM Cell and its core components including forget gate, input gate, cell gate, and output gate.

1. Forget Gate $f_t$ decides what information should be discarded from the cell state. It uses the sigmoid function to squash values between 0 and 1, indicating the amount of information to be discarded. The forget gate looks at the current input and the previous hidden state (output from the last time step).

2. Input Gate $i_t$ decides which values from the input should be updated in the cell state. It also uses the sigmoid function. The candidate values are a set of new cell state values that could be added to the cell state, calculated using a hyperbolic tangent activation function, which outputs values between −1 and 1.

3. Cell state $c_t$ is the ‘memory’ of the LSTM cell. It's updated based on the decisions made by the forget gate and the input gate. First, the cell state is multiplied by the forget gate's output to forget certain information. Then, the input gate's output is element-wise multiplied by the candidate values, and the result is added to the cell state.

4. Output gate $o_t$ decides what the next hidden state should be. This hidden state is used for predictions and is also passed to the next LSTM cell. The output gate takes the current input and the previous hidden state passes them through a sigmoid function and then multiplies the output by the cell state to compute the next hidden state.

A Deep Learning model based on LSTMs involves stacking multiple layers of these LSTM cells, creating a ‘depth’ of hidden layers (Liu et al. 2022). This is done to extract higher levels of abstraction from sequential data, making the model capable of understanding more complex patterns. The input is passed through each layer sequentially, with each layer potentially learning to recognise different features. While LSTM cells form the basic unit of memory, an LSTM-based Deep Learning model represents an entire architecture employing these units (Z. Li et al. 2020).

It is beneficial for modelling and predicting sequential data where the output depends on the input from previous steps. Since our dataset has high temporal dependencies, LSTM is a suitable deep-learning model for our supply forecasting framework. This paper proposes a deep-learning model with three LSTM layers, each consisting of 300 units. This three-layer architecture is followed by a dense layer made of one node to output the GHI forecast. The described architecture is implemented using Keras, (Chollet 2015).

3.5. Feature selection

Selecting the ideal set of features isn't a one-size-fits-all scenario for every forecasting horizon. It's crucial to remember that incorporating irrelevant features into a Deep Learning model can have an adverse impact on its performance. Therefore, the selection of pertinent features should align with a specific forecasting horizon. This principle extends to the size of the historical data incorporated into the Deep Learning model, as well as the degree of polynomial feature augmentation. To optimise input selection, we utilised a grid search approach (LaValle, Branicky, and Lindemann 2004). This technique probes each possible combination of inputs with historical data ranging from 5 to 25 and polynomial degrees set between 1 and 5.

The training procedure also incorporates an early stopping method (Prechelt 2012). This strategy monitors the model's error value on the validation set, ensuring it diminishes during model training. The implemented method has a patience threshold set at 10 epochs, meaning if the performance doesn't improve within 10 training epochs, the best previous model is restored, and training is terminated (Chollet 2015). We evaluate the features based on their $R^2$ score (refer to Subsection 4.1), aiming for the highest possible $R^2$. If two feature sets yield the same $R^2$ score (to the second decimal), preference is given to the feature set that presents the lowest mean absolute error (MAE). Table 3 delineates the optimal feature set for each forecasting horizon.

4. Results

This section starts with the report of the evaluation metrics used to measure the effectiveness of our models. Then, we introduce the persistent model, which serves as our reference benchmark. Finally, we assess the proposed Deep Learning model, demonstrating its predictive capabilities and overall performance.

4.1. Evaluation metrics

To evaluate our AI-based Forecasting Framework, we used quality metrics including scale-dependent metrics such as the mean absolute error (MAE) and the root mean square error (RMSE). MAE and RMSE each provide a different view on the quality of the result in [W/m${}^2$]. Thus, the RMSE measures the error quadratically which penalises big outliers more compared to MAE (Pal 2017). For comparison with other studies, we included normalised quality metrics: the normalised mean absolute error (nMAE) and the normalised root mean square error (nRMSE). The normalisation process uses the mean value of the measurements in the test dataset. Moreover, we used statistical quality metrics to evaluate the robustness of our results. We used the MAE confidence intervals at 95% level as well as the $R^2$ coefficient of determination. Hyndman and Koehler (2006) provide a thorough review of forecast accuracy measures.

4.2. Benchmark reference model

We included a persistent model in our study as the benchmark. It is also known as Random Walk and Naive forecast model. The model estimates that the solar irradiance at the next timestep $t_{+1}$ is equal to the one measured at the current timestep t, namely $\mathrm{GH}{I}_{t+1}=\mathrm{GH}{I}_t+{ϵ}_t$, where ${ϵ}_t$ is forecasting error at the timestep t. So the persistent model predicts the Global Horizontal Irradiance (GHI) for PV panels as ${\widehat{\mathrm{GHI}}}_{t+h}=\mathrm{GH}{I}_t$, where h is a forecasting horizon. The model assumes that the variable, in each period, randomly deviates from its previous value, with these deviations being independently and identically distributed in size. It's akin to saying that the initial difference in the variable forms a series, suitable for application of the mean model. This simple benchmark demonstrated competitive performance in many scenarios, especially in time series that demonstrate random walk properties (Hewamalage, Ackermann, and Bergmeir 2023)

Despite its inherent simplicity, the persistence model is broadly adopted across diverse fields for energy demand and price modelling (Löhndorf and Wozabal 2023), as well as in macroeconomic time series (Perron 1988), financial markets (Black and Scholes 1973), and even fundamental sciences, including physics (Risken and Voigtlaender 1984), chemistry (Van Kampen 1992), material science (Doi and Edwards 1988), and biology (Goel and Richter-Dyn 2016).

Marquez and Coimbra (2011) demonstrate the use of the persistent model for global and direct irradiance forecasting, and Paulescu and Paulescu (2019) used the persistent model to benchmark various statistical techniques. In a similar manner, we present the results of the persistence model for each forecasting horizon in Table 4. The $R^2$ score compares the performance of the persistence model with the performance of the average of the observed data points. Table 4 shows the persistence model has $R^2$ scores decreasing from 0.91 to 0.76 and nMAEs increasing from 13.58% to 28.13% in the 15-minute to 1-hour ahead horizons. This indicates that the persistence model performs well for minutes ahead horizons. Table 4 also indicates a performance decrease as the horizon expands. Our results show that the $R^2$ score becomes negative for forecasting horizons longer than 2 hours for hours ahead and days ahead horizons. Therefore, our data has high variability for horizons longer than 2 hours, and the persistence model performs poorly over those horizons.

Table 4. Results of the persistence model over the different forecast horizons.

	Minutes ahead				Hours ahead					Days ahead
	15mn	30mn	45mn	1h	2h	4h	6h	12h	24h	2 days	3 days	4 days	5 days	6 days	7 days
MAE [W/m2]	50.04	66.65	83.06	98.33	154.17	208.92	202.75	89.25	105.45	89.76	90.76	77.08	72.97	74.07	81.02
nMAE [%]	13.58	18.44	23.10	28.16	46.66	64.75	74.51	24.65	29.04	24.63	25.03	21.27	20.30	20.70	22.28
RMSE [W/m2]	79.44	93.07	108.98	124.48	187.00	265.92	267.33	125.86	146.72	114.72	116.60	104.10	93.06	95.99	95.23
nRMSE [%]	21.56	25.75	30.31	35.64	56.59	82.41	98.25	34.76	40.41	31.48	32.16	28.73	25.89	26.83	26.19
$R^2$	0.91	0.87	0.82	0.76	0.43	−0.40	−0.77	0.23	−0.16	−0.10	−0.27	−0.15	−0.10	−0.13	−0.25

4.3. Proposed deep learning solar forecasting model evaluation

This subsection presents our proposed model for each forecasting horizon and evaluates its performance. In our study, we ensured optimal readability and interpretability by limiting each figure to a maximum of 250 data points. It's essential to note that the variation in timestamps across Figures 4 through 7 stems from the differing timescales, which range from 15 minutes to 2 days. In each figure, the red line denotes the one-step-ahead predicted GHI, the green line shows the measured GHI, and the 95% confidence interval of the forecast is represented by the gray area.

Figure 4. Minutes ahead GHI forecast (15 min ahead).

Minutes ahead GHI forecast. The GHI variable changes overtime. 15 minutes are used as a resolution.

Transitioning from Figures 4 to 7 involves expanding the observational timescale from a window of 15 minutes to 2 full days, which provides a broader data perspective. The apparent disappearance of the daily seasonal component, or solar irradiance variability, across these figures can be attributed to the Earth's rotation around the sun. Figure 4, offering a glimpse at 15-minute-ahead forecasts, showcases not only the daily seasonal pattern but also irregular variations. Progressing to Figure 5, we extend our forecasting horizons to 2 hours, which preserves the daily seasonality with minimal irregular variations. By the time we reach Figure 6, with forecasts taken every 6 hours, daily seasonality fades from view. Finally, Figure 7 presents forecasts over a 2-day horizon, revealing the yearly seasonal behaviour amidst irregular variations.

Figure 5. Hours ahead GHI forecast (2 h ahead).

Minutes ahead GHI forecast. The GHI variable changes overtime. 2 hours are used as a resolution.

Figure 6. Hours ahead GHI forecast (6 h ahead).

Minutes ahead GHI forecast. The GHI variable changes overtime. 6 hours are used as a resolution.

Figure 7. Days ahead GHI forecast (2 d ahead).

Minutes ahead GHI forecast. The GHI variable changes overtime. 2 days are used as a resolution.

More specifically, Figure 4 represents 15-minute-ahead forecasts spanning 4 days. This figure illustrates the daily solar irradiance pattern, characterised by a rise in GHI from 7:30 am to mid-day, followed by a decrease. As described by Provost and Fawcett (2013), such regular short-term fluctuations, coupled with irregular variations due to factors like cloud shadows, define seasonality in time-series data. Figure 5, displaying 2-hour-ahead forecasts over 26 days, exhibits a decline in irregular variations as the forecasting window broadens, leaving only daily seasonality intact. Conversely, Figure 6, showing 6-hour-ahead forecasts across 5 months, presents an increase in irregular variations as the day progresses from sunrise to sunset or vice versa, depending on the season. In contrast, Figure 7 shows a disappearance of daily seasonality as the forecasting window expands, leaving only yearly seasonality visible within a broad confidence interval. Figure 7, illustrating 2-day-ahead forecasts over 10 months, emphasises the challenge of modelling significant irregular day-to-day weather variations, which remain high even as the forecasting model mainly captures yearly seasonality. The forecasting framework does not capture these variations but only models yearly seasonality. To do so, the model provides a wide 95% confidence interval to include the irregular variations.

Table 4 encapsulates the results of the proposed AI-based Forecasting Framework. The accuracy of the Deep Learning GHI forecasting model shows a diminishing trend as the forecasting horizon expands from 15 minutes to 24 hours, signaling a shift from daily to yearly seasonality. This trend is reflected in Table 4, where $R^2$ sharply drops from 0.79 to 0.17 between the 6-hour and 24-hour horizons. The MAE and RMSE peak at 99.0 W/m${}^2$ and 124.41 W/m${}^2$, respectively, for the 24-hour horizon, reinforcing the observed shift from daily to yearly seasonality within these timeframes (Table 5).

Table 5. Results of AI-based forecasting framework over the different forecast horizons.

	Minutes-ahead				Hours-ahead					Days-ahead
	15mn	30mn	45mn	1h	2h	4h	6h	12h	24h	2 days	3 days	4 days	5 days	6 days	7 days
MAE [W/m2]	43.72	48.50	51.70	50.64	56.83	67.12	67.95	90.63	99.00	79.48	75.65	71.56	65.93	66.18	61.34
nMAE [%]	11.87	13.42	14.38	14.50	17.20	20.80	25.00	25.00	27.27	21.81	20.86	19.75	18.34	18.49	16.87
MAE – Confidence [%]	8.68	8.33	11.84	1.24	6.70	3.24	12.60	2.11	0.87	0.33	0.49	2.99	3.72	2.94	8.07
RMSE [W/m2]	71.12	76.96	79.93	79.83	85.31	97.13	93.03	119.42	124.41	96.89	93.87	85.68	77.37	76.50	74.33
nRMSE [%]	19.30	21.30	21.96	22.86	25.82	30.10	34.23	32.95	34.35	26.59	25.89	23.65	21.52	21.38	20.44
RMSE – Confidence [%]	1.63	3.57	2.68	1.24	5.20	1.75	5.44	1.36	0.39	0.51	0.66	2.81	5.23	4.12	9.49
$R^2$	0.92	0.91	0.90	0.90	0.88	0.81	0.79	0.31	0.17	0.22	0.18	0.22	0.24	0.29	0.24

5. Discussion

In this section, we delve deeper into the critical facets of our study. We discuss the proposed AI-based Framework for solar forecasting, Deep Learning model accuracy, and its potential use cases and implementation within the Smart Grid ecosystem. First, we turn our attention to solar forecasting, where we engage in an in-depth examination of our approach. Next, we critically analyse the performance of our proposed model. Here, we scrutinise our findings compared to other studies, aiming to shed light on our model's strengths and areas for potential improvement. Then, we shift our focus to the practical implications of our research. In this context, we explore the potential applications of the proposed AI-based Forecasting Framework within the Smart Grid ecosystem, envisioning its role in enhancing grid reliability and efficiency.

5.1. Solar forecasting

As the global climate crisis escalates, nations are compelled to transition their energy production towards more sustainable, renewable sources (Tollefson 2022). Accomplishing this necessitates a well-rounded energy portfolio that can reliably meet user demands, irrespective of prevailing weather conditions. Among the array of renewable options, solar energy stands out as a significant contributor, serving as a source for both heating and electricity production (Adeh et al. 2019; Mercure et al. 2021). However, given the intermittent nature of solar power, accurate forecasting is pivotal to predicting electricity output and thereby streamlining the production process (Rai et al. 2021).

Accurate prediction of solar power output not only ensures enhanced profitability for both prosumers and solar farm owners but also catalyses the economic growth of the broader solar industry (Gautier, Jacqmin, and Poudou 2018). This profitability incentive boosts the sector's development and thereby drives the larger shift towards sustainable energy practices. Simultaneously, improved solar power forecasting allows ISOs to enhance power grid balancing, thereby conserving energy through minimised losses. This helps protect electrical infrastructure from potential damage due to power surges caused by overproduction. Precise solar power forecasting fosters sustainable growth, aids in grid management, and bolsters the profitability of renewable energy ventures.

5.2. Deep learning model accuracy

5.2.1. Forecasting minutes ahead

Minutes-ahead GHI forecasts range from 15 minutes to 1 hour. Forecasting in these time horizons allows solar electricity producers to place their final bids in the intra-day market before it closes. Thus, the producers can manage their risks depending on their trading strategy and risk exposure. From another perspective, these forecasts allow ISOs to plan PV electrivity production with a narrow confidence interval and anticipate the grid balance. Table 4 shows that the RMSE of our study ranges from 71.12 W/m${}^2$ to 79.93.74 W/m${}^2$ for minutes-ahead horizons resulting in an nRMSE going from 19.30% to 22.86%. In the study by Feng and Zhang (2020), in which the authors use a satellite-based Deep Learning approach, they obtain RMSE ranging from 81 to 151 W/m${}^2$ for horizons going from 10 minutes to 1 hour ahead. In the comparative study of ML-based approaches by Gbémou et al. (2021), they obtain best nRMSE ranging from 20% to 32% for a horizon of 10 minutes to 1 hour. In our study, we achieved better performance than the reviewed articles for the same horizons. This confirms the accuracy of the proposed AI-based Forecasting Framework.

5.2.2. Forecasting hours ahead

Hours-ahead forecasts range from 2 hours to 12 hours. Forecasting in these time horizons allows solar electricity producers to adjust their bids in the intra-day market. The shift from daily seasonality to a yearly trend occurs in this time selection as the forecasting horizon expands. Solar prediction becomes increasingly harder as the forecasting horizon grows. This can be seen in Table 4 for hours-ahead as the persistence model nRMSE ranges between 56.59% to a peak value of 98.25% at 6 hours ahead before dropping off. The results of our framework are shown in Table 4 with an nRMSE peak of 34.35% over the same period. Compared with other studies such as Huang et al. (2020), our nRMSE range from 25.82% to a peak of 34.35% at 24 hours whereas the best paper from their study range from 32% to 36%. Our model offers either identical or better results compared with other studies. We differentiate by providing a confidence interval for our errors which allows readers to statistically appreciate the accuracy of our AI-based Forecasting Framework.

The 24-hours-ahead forecast allows market players to make bids in the day-ahead market. In the study by Aryaputera, Yang, and Walsh (2015), their best result for day forecasts has an error of 188 W/m${}^2$ whereas we obtain an RMSE of 124.41 W/m${}^2$. Thus our model shows a significant improvement of 34% compared to their study.

5.2.3. Forecasting days ahead

Days-ahead forecasts range from 2 days to 7 days, allowing energy suppliers and ISOs to have a long-term vision of solar electricity production. These longer forecasts are used for maintenance and network operation planning. In this time selection, the daily seasonal components of GHI vanish and only the yearly trend remains, with variations. From Table 4, we can see that the framework only forecasts the trend resulting in nRMSE between 20.44% and 26.59% and a wide confidence interval between 0.51% and 9.49% to capture the variations. As a reference point, in their study, Ibrahim et al. (2015) forecasted the GHI in different European cities. In Switzerland, they obtained nRMSE ranging from 45.2% to 47.4% for 2-day ahead forecasts and 42.7% to 48.1% for 3-day ahead forecasts. In their study, the best nRMSE were obtained when forecasting GHI in Spain where the weather is more stable. In both, their study and ours, only GHI time-series data was used. When comparing our results with theirs in Switzerland, we obtained better results with the same type of input.

5.3. Augmenting energy management strategies through integrated AI-based forecasting frameworks

Practicality and Scalability of the AI-based Forecasting Framework while image-based methodologies offer valuable insights, they come with a hefty computational cost. These approaches necessitate the processing of extensive image datasets, a task that demands significant time and computational resources. Furthermore, satellite images pose an inherent limitation when accurate forecasts for smaller time horizons (less than 60 minutes) are required due to their intermittent availability. In contrast, models based on LSTM networks offer a more resource-efficient alternative. Once trained, these models require minimal computational power and are capable of delivering GHI forecasts within mere seconds. This model could be implemented in a cloud-based environment, providing a low-cost machine-learning service accessible to customers. This efficient forecasting solution not only optimises resource usage but also ensures rapid, accurate predictions, making it a viable and cost-effective option for various stakeholders (Ivanov, Dolgui, and Sokolov 2022).

The ML market, which was valued at USD 19.20 billion in 2022, is projected to grow from USD 26.03 billion in 2023 to USD 225.91 billion by 2030 (Fortune Business Insights 2023). This surge in interest has catalysed the rise of ML as a service (MLaaS) enterprises, which provide ML capabilities as cloud-based services. The MLaaS model empowers customers to harness the potential of data without necessitating any technical expertise. Our AI-based Forecasting Framework, specifically designed for solar irradiance forecasting, empowers users to estimate the electricity output of any solar facility, irrespective of the installation size or the type of PV panel utilised. This MLaaS-eligible approach could cater to a wide range of stakeholders, including electricity producers, ISOs, and even weather forecasting agencies.

GHI forecasts are used as the initial input for solar production forecast models. Overall, accurate forecasting guarantees better integration of solar electricity production into the grid. This improved integration of solar energy eases the burden of managing decentralised production sites that have come online due to the increase in prosumers. In addition to their use in electricity grid management, solar forecasts are also relied on by public agencies and heating producers. These results are essential for both ecological and economic reasons since cities' energy consumption (heat and electricity) is correlated with sun irradiance (Saloux and Candanedo 2018). Our AI-based Forecasting Framework could also be used in such cases as these load models use solar irradiance as input.

The application of our AI-based forecasting framework could be further enhanced when amalgamated with other significant advancements in energy management. This combined approach can shape a more holistic and effective energy management and efficiency solution. For example, in a study conducted by Tuo, Liu, and Liu (2019), attention was drawn to the immense energy consumption in machine tools and their high potential for energy savings. They suggested the development of energy-efficient machine tools and selecting appropriate ones in procurement processes as key methods to conserve energy. They developed 'inherent energy performance' (IEP) indexes, a set of key performance indicators which considered various process controls impacting the energy performance of machine tools. The proposed AI-based forecasting framework, when coupled with these IEP indexes, could yield accurate energy forecasts, thereby helping in the design and selection of energy-efficient machine tools and improving overall energy management.

H. Li et al. (2018) presented an approach to evaluate energy consumption performance and energy-saving potentials of the ceramic production chain. They proposed an integrated approach of first-order hybrid Petri net (FOHPN) model, objective linear programming model, and sensitivity analysis. Integrating our AI-based forecasting framework with this approach could offer more efficient energy management and further savings in the energy-intensive ceramic production chain.

In another study, Ouyang and Fu (2020) examined the adoption of energy efficiency by energy-intensive manufacturers in response to increasing consumer environmental awareness (CEA). He studied several mathematical models that incorporated CEA into the energy-saving domain. Integrating our AI-based forecasting framework with this approach can enhance the manufacturer's energy efficiency strategies by providing accurate and timely energy forecasts, thus responding effectively to CEA.

Finally, Aghelinejad, Ouazene, and Yalaoui (2018) investigated energy management in a single-machine manufacturing environment. They presented two mathematical models to reduce the total energy costs of a production system. By aligning the AI-based forecasting framework with these models, energy consumption predictions could be more effectively incorporated into the production scheduling process, thereby boosting energy efficiency in the manufacturing sector.

These diverse studies illuminate the potential of integrating the proposed AI-based forecasting framework with other notable energy management advancements, broadening its applicability and efficacy in diverse sectors.

6. Conclusion, managerial implications, and future research

This section concludes the research, provides managerial implications, and outlines promising directions for future research.

6.1. Conclusion

In our study, we commence by outlining the structure of the electricity market and discussing the intricacies involved in incorporating solar electricity production into power grids. We then detail a Global Horizontal Irradiance (GHI) forecasting framework, covering aspects like pre-processing, data wrangling, building a long-short-term memory model, and subsequent forecasting. We also explore the optimisation of Deep Learning model performance through custom input sets determined by a grid search.

When contrasted with similar studies, our findings demonstrate equivalent, if not superior, results for regions with a mild climate. In addition, our literature review highlights that our research uniquely spans a diverse range of forecasting horizons, extending from 15 minutes to a week ahead. The accuracy of our forecast is sufficient to be utilised for load and production modelling. In addition, we provide a confidence interval, allowing users to anticipate the minimum and maximum energy production values. Such vital information could benefit Independent System Operators (ISOs) and energy agencies by enhancing profitability, facilitating energy grid balancing, and ultimately diminishing the system's ecological footprint.

6.1.1. Research question 1: how can an AI-based forecasting framework be leveraged to optimise solar energy management?

Our findings comprehensively explore an LSTM-based Deep Learning model for accurate solar electricity forecasting. This AI-driven approach aids power grid operators in effective planning and ensures a stable supply of electricity. In order to quantify the effectiveness of the proposed AI-based Forecasting Framework, we employed such metrics as MAE and RMSE, nMAE, nRMSE, and $R^2$. We also incorporated a persistent model as a reference benchmark in our study. The performance of the persistent model diminishes for horizons longer than two hours due to the high variability in our data. This suggests that the model's predictive capability is particularly effective for forecasting horizons within the span of minutes. Although the model's predictive accuracy declines as the forecast horizon expands, it still outperforms the persistence model for horizons longer than two hours.

6.1.2. Research question 2: how can the AI-based forecasting framework be effectively implemented within the smart grid ecosystem to enhance grid reliability and efficiency?

Through the implementation of an LSTM-based Deep Learning model, we have demonstrated that AI-based forecasting can significantly optimise the integration of solar electricity into power grids. While the model's forecasting accuracy decreases with expanding horizons, it provides a substantial improvement over the persistent model for horizons longer than two hours. The incorporation of such AI-based frameworks into the Smart Grid ecosystem not only enhances grid reliability and efficiency but also promotes the wider adoption of renewable energy sources. Our study, therefore, contributes to the evolution of Smart Grids, marking a significant stride towards achieving sustainable and efficient energy practices with the aid of advanced AI technologies.

6.2. Managerial implications

The managerial implications of integrating an AI-based forecasting framework within the Smart Grid ecosystem are multifaceted:

• Enhanced Strategic Decision-making: The introduction of AI-based forecasting within the Smart Grid ecosystem arms managers with data-driven insights, a powerful tool for strategic decision-making. This predictive ability facilitates a more proactive approach to management, allowing for better anticipation of issues and the crafting of effective responses. Operations can be optimised based on reliable forecasting, leading to more efficient resource allocation, and maximising overall operational efficiency. This forward-planning ability could lead to substantial cost savings and increased service reliability.

• Improved Reliability and Efficiency: AI-driven forecasts can significantly enhance the reliability and efficiency of the power grid. Managers will be better equipped to balance supply with demand, reducing instances of power wastage or shortages. More reliable service provision enhances customer satisfaction, leading to better customer retention rates and potentially attracting new customers. Consistent power provision also ensures the smooth functioning of other dependent industries, thereby contributing to overall economic stability.

• Risk Management: Integrating solar energy more effectively into power grids, facilitated by accurate AI-based forecasting, enables managers to mitigate various risks. These include risks associated with over-reliance on non-renewable energy sources, such as price volatility, supply chain disruptions, and changing environmental regulations. With a more reliable forecast of solar energy generation, energy providers can diversify their energy mix, thereby reducing their exposure to these risks.

• Promoting Innovation and Competitiveness: The use of AI in energy management signals an innovative and forward-thinking approach. Such a stance can provide a competitive edge in the rapidly evolving energy market, differentiating the company from others that are slower to adopt such technologies. Embracing AI technology could also open new avenues for collaboration with tech companies, research institutions, and other innovative entities.

6.3. Future research

Further research could be carried out by adding other inputs to our forecasting model such as ground measurements (air temperature, humidity), satellite imagery (cloud motion vectors) or using more measurement sites.

Additionally, exploring the integration of AI-based forecasting within the Smart Grid ecosystem in the context of emerging technological trends could be a promising direction for future research. The deployment of 5G networks, for instance, presents significant opportunities (Dolgui and Ivanov 2022). With its improved communication and data transfer speeds, and reduced latency, 5G could optimise the performance of AI-based forecasting models within Smart Grids. Researchers could probe how to leverage the capabilities of these enhanced networks to refine real-time forecasting and data-driven decision-making in Smart Grid management. Similarly, digital supply chains' evolution could impact energy demand and usage patterns. Studying AI-based forecasting's role within digital supply chain operations could reveal valuable insights. It may offer strategies for optimising energy usage and improving overall supply chain efficiency (Ivanov, Dolgui, and Sokolov 2022). Additionally, these models could be explored for their potential in demand forecasting and managing energy-intensive operations.

The onset of Industry 4.0 and the transition to Industry 5.0 (Ivanov 2023), characterised by the integration of digital technologies into manufacturing and industrial practices, ushers in an era of increased attention to sustainability and resilience (Choi et al. 2022; Ivanov 2022). In this context, the potential of AI-based forecasting in optimising energy usage in automated and data-driven industrial processes becomes more salient. These algorithms can guide the implementation of more sustainable practices (Ivanov 2018) and promote resilience (Ivanov 2022) by aiding in efficient resource management, which is crucial in an era of climate change and dwindling resources. Moreover, the intersection of Smart Grids and smart factories provides a fascinating and promising domain of study. As the demand for green and renewable energy sources grows, understanding how these interconnected systems can optimise their operations for sustainable energy consumption becomes a crucial endeavor. By leveraging AI-based forecasting models, we can gain insights into how to increase the reliability and resilience of these systems against uncertainties and disturbances.

Additionally, acknowledging the Smart Grid ecosystem within a broader framework of a shortage economy, where resources are scarce, provides further impetus for sustainable and resilient practices (Ivanov and Dolgui 2022). In such a scenario, effective energy management is indispensable, and AI can play a key role in mapping strategies for maximising the use of renewable energy sources. This approach minimises waste and promotes sustainable practices, contributing to increased resilience in the face of resource scarcity. Furthermore, it can lead to the development of robust systems capable of withstanding various shocks, thus ensuring the continuity and sustainability of industrial processes.

By investigating these areas in future research, we could deepen our understanding of AI-based forecasting models' role within our rapidly evolving technological landscape. The aim is to optimise energy usage across different sectors and contexts, ultimately contributing to a more sustainable future.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The authors confirm that the data supporting the findings of this study are available. The experiments were conducted in a reproducible manner, and the source code, as well as data are available on the GitHub repository https://github.com/Pierre-Bouquet/LSTM_GHI_Forecasting.

Additional information

Notes on contributors

Pierre Bouquet

Pierre Bouquet is a first-year Ph.D. student at the Massachusetts Institute of Technology (MIT), under the mentorship of Prof. Yossi Sheffi. He received both his bachelor's degree and an M.Sc. in Mechanical Engineering from the Swiss Federal Institute of Technology in Lausanne. During his M.Sc., he specialised in Automation Systems and also pursued a minor in Data Science. Pierre is deeply interested in the fields of optimisation, machine learning, and data science, with a particular focus on supply chain and operations optimisation. For inquiries or collaborations, Pierre can be contacted at his academic email: [email protected] or via LinkedIn.

Ilya Jackson

Dr. Ilya Jackson is a Postdoctoral Associate at MIT Center for Transportation & Logistics. He earned his Ph.D. in Civil Engineering and Transportation from the Transport and Telecommunication Institute, where he spent one year as an assistant professor shortly after that. The main ideas of his Ph.D. thesis have been summarised in the paper 'Neuroevolutionary approach to metamodel-based optimisation in production and logistics', which received the Young Researcher Award in 2020. Dr. Ilya Jackson currently focuses on Machine Learning and AI for Supply Chain Management.

Mostafa Nick

Mostafa Nick received the Ph.D. degree in electrical engineering from EPFL, Lausanne, Switzerland, in 2016. From 2016 to 2017, he was a postdoctoral researcher in Distributed Electrical Systems Laboratory, EPFL. Since 2017, he has been with the National Grid ESO, Warwick, UK, where he is currently a team lead on power systems analysis, optimisation, and advanced analytics.

Amin Kaboli

Dr. Amin Kaboli is a lecturer at Swiss Federal Institute of Technology in Lausanne (EPFL), specialising in AI Product Management, Production Management, and Continuous Improvement. He coaches and advises tech founders and business executives across industries, holding advanced leadership diplomas from IMD Business School and a Ph.D. in Manufacturing Systems & Robotics from EPFL.

Notes

1 https://github.com/Pierre-Bouquet/LSTM_GHI_Forecasting.

2 https://github.com/DESL-EPFL/DESL-Photovoltaic-timeseries.