Forecasting cross-border power transmission capacities in Central Western Europe using artificial neural networks

Abstract

Flow-based Market Coupling (FBMC) provides welfare gains from cross-border electricity trading by efficiently providing coupling capacity between bidding zones. In the coupled markets of Central Western Europe, common regulations define the FBMC methods, but transmission system operators keep some degrees of freedom in parts of the capacity calculation. Besides, many influencing factors define the flow-based capacity domain, making it difficult to fundamentally model the capacity calculation and to derive reliable forecasts from it. In light of this challenge, the given contribution reports findings from the attempt to model the capacity domain in FBMC by applying Artificial Neural Networks (ANN). As target values, the Maximum Bilateral Exchanges (MAXBEX) have been chosen. Only publicly available data has been used as inputs to make the approach reproducible for any market participant. It is observed that the forecast derived from the ANN yields similar results to a simple carry-forward method for a one-hour forecast, whereas for a longer-term forecast, up to twelve hours ahead, the network outperforms this trivial approach. Nevertheless, the overall low accuracy of the prediction strongly suggests that a more detailed understanding of the structure and evolution of the flow-based capacity domain and its relation to the underlying market and infrastructure characteristics is needed to allow market participants to derive robust forecasts of FMBC parameters.

Introduction

Within the current pursuit of a single electricity market in Europe, the so-called Price Coupling of Regions project was initiated in 2009. It allows to efficiently balance demand and supply in one market zone irrespective of internal transmission constraints, while simultaneously optimizing cross-zonal exchanges of electricity under coupling capacity constraints between the zones. Two fundamental principles for the calculation of the coupling capacity available for commercial cross-border exchange are currently implemented in Europe: the Available Transfer Capacity (ATC) method and the Flow-Based Market Coupling (FBMC) method. The latter is currently applied in Central Western Europe (CWE) at the borders between Germany/Luxembourg, Austria, Belgium, the Netherlands and France for the day-ahead market.

Although common regulations define both the ATC and FBMC methods (Official Journal of the European Union 2015), their implementation, which is actually carried out by the Transmission System Operators (TSOs), is not fully published (EFET - European Federation of Energy Traders 2018). TSOs have some degrees of freedom to individually apply models and set parameters for ensuring the security of supply. For parties other than the TSOs, it is therefore impossible to fundamentally model the ATC or FBMC method precisely. Therefore, it is very difficult for market participants to forecast the capacities which TSOs will make available for future trading intervals. This lack of knowledge is an obstacle for the formulation of trading strategies from the perspective of the market participants.

In view of the above, this contribution discusses an attempt to model the FBMC by applying Artificial Neural Networks (ANN), and to let the ANN discover patterns adaptively from published data in order to forecast one specific indicator from the dynamic capacity domain calculated through the FBMC method, which is the Maximum Bilateral Exchanges (MAXBEX). Publicly available data related to the FBMC parameters is used as an input to the ANN, and the results are evaluated by comparing them to published actual outcomes. Although the observed low accuracy of the forecast rules out the proposed method as a high-quality forecast approach, the findings discussed in this contribution serve as important guidelines for future research on understanding and predicting the flow-based capacity allocation process.

Flow-based market coupling

FBMC was launched in May 2015 for the region Central Western Europe, with the goal of increasing the market performance and maximizing social welfare (Van den Bergh et al. 2016). The CWE region, initially consisting of the borders between the Belgian, French, Dutch and German-Austrian bidding zones, is the first region to have implemented FBMC for the day-ahead market. The FBMC flow domain represents all feasible combinations of commercial exchanges between participating bidding zones. To calculate it, TSOs assess a number of quantities that are named below (Belgian Federal Commission for Electricity and Gas Regulation 2017; Plancke et al. 2016). Figure 1 summarizes the flow-based capacity calculation on the basis of these quantities.

Fig. 1

Interaction of the flow-based parameters for the calculation of the capacity domain (European Commission 2017)

Critical Branch Critical Outage (CBCO) are network elements such as lines, cables, transformers, generation units, or others that are significantly impacted by CWE cross-border trades under monitored critical outages conditions. The maximum allowable power flow (Fmax) is calculated for each CBCO under rated voltage conditions. Based on market data, the Reference Flow (RF) on a critical branch is determined as the power flow caused by commercial transactions outside the day-ahead power exchange, such as bilateral trades or forward markets. In order to account for the uncertainty of the assumptions, forecasts and approximations of the flow-based method, a Flow Reliability Margin (FRM) is determined. The Final Adjustment Value (FAV) is an additional margin that allows TSOs to take account of additional knowledge and experience that cannot be formally modeled.

The flow-based computation starts from the 2-Days Ahead Congestion Forecast (D2CF), which constitutes the "Base case" and represents the best estimate for the CWE electrical system at the delivery day. Each TSO also determines the Generation Shift Keys (GSK), which define how a change in the zonal net position is mapped to the generating units in a bidding area. Two other adjustment values are applied to the FBMC in order to account for special uncertainties that can hardly be modeled or computed within the method. These are the so-called Remedial Actions (RA) and External Constraints (EC).

The flow-based parameters that are finally computed and published are the Zonal Power Transfer Distribution Factors (PTDF’s), which define the physical flow on a transmission line resulting from one unit of cross-zonal power exchange between bidding zones (resulting from the market-clearing result), and the Remaining Available Margin (RAM), which give the maximum flows that are allowable for day-ahead trading. For each CBCO, the RAM is calculated as the Fmax minus the sum of RF, FAV and FRM (Boury 2015). These values allow to determine the flow-based domain, which corresponds to all combinations of concurrent mutual commercial exchanges of the countries in the CWE region, which according to the FMBC method are in accordance with the physical constraints of the transmission system.

The results of the flow-based calculation is a linearised form of the physical flow margins of the network elements that are available for further power exchange. They define the boundaries of the flow-based domain with the limits of each critical line. Each constraint is a hyperplane in an n−1-dimensional search space, where n is the number of coupled bidding zones. With four zones in CWE (until October 2018), the flow-based search space was made up of an average of 15 to 20 planes, corresponding to the most constraining critical branches.

Since the introduction of the FBMC to the market, a series of inconsistencies have raised distrust in the correct functioning and transparency of the method; reasons for this are the anonymization of the critical network elements, which impedes market participants to link those with the data of the planned network element outages announced by ENTSO-E. It is only since mid-2017 that all TSOs agreed to make this data public in the Joint Allocation Office (JAO) platform, nevertheless creating concern regarding the lack of standardization on critical elements location and naming. The ambiguity in the definition of several parameters has been documented by several authors (Boury 2015; Van den Bergh et al. 2016; Sőrés et al. 2013). Several major cases of TSO discretionary action resulting in a lack of market transparency were observed and documented (Belgian Federal Commission for Electricity and Gas Regulation 2017), to mention some: the unannounced introduction of critical elements restraining transmission capacity additionally, transparency of seasonal thermal limits (Fmax), increase of FAV values, and unannounced changes in ECs.

Related work

Due to the worldwide importance of energy resources and price predictions, neural networks and deep learning have received increasing attention over the years, considering their usefulness for energy consumers and generators in relevant decision-making processes. The application of ANN in the energy market cover such a wide range of use cases as wind turbine signal assessment (Qin et al. 2019), day-ahead photovoltaic power forecasting (Wang et al. 2019a), or crude oil forecasting (Cen and Wang 2019; 2018), to mention a few.

Recent day-ahead electricity price forecasting was done by (Wang et al. 2019b; Yamin et al. 2004), while (Nazar et al. 2018) studied a hybrid model for simultaneous load and price forecasting. Machine learning algorithms have also been applied to modeling parameters related to grid capacity, e. g. (Staudt et al. 2018), who predict re-dispatch measures in the German electricity market with an artificial neural network, or (Fainti et al. 2016), who study line overloads applying an ANN which was trained using the Levenberg-Marquardt algorithm.

The prediction of flow-based capacities has been researched less frequently. Most studies in the literature focus on assessing the efficiency of FBMC for the market integration of power transfers between countries or price areas, in particular in comparison with the ATC method. For instance in Bjørndal et al. (2018); Plancke et al. (2016), different challenges of the FMBC model implementation are discussed, while (Bjorndal et al. 2018) study the influence of the bidding zone configuration on the performance of both the ATC and the FMBC model. Another focus of current research are modified methodologies and sensitivities for the determination of flow-based parameters in the FBMC model, e. g. by using more efficient algorithms (Matthes et al. 2017), parameter variations (Wyrwoll et al. 2018) or a statistical approach to Generation Shift Keys (GSK’s) (Schönheit and Sikora 2018). The influence of different GSK methodologies on both the market zone as a whole and individual generator dispatch has been analyzed in Finck et al. (2018). In Khatavkar et al. (2018), an ANN is used to estimate the transmission reliability margin of electrical interconnectors in Canada.

There are hardly any studies that try to predict the flow-based parameters for coming trading intervals. van Stiphout (2016) compares different methods for doing such forecasts based on past flow-based capacity data, e. g. estimating the future capacities on the basis of similar hours of the past. Wallin (Wallin 2016) addresses similar topics, but assesses electricity flows resulting from trading directly; he does that also on the basis of past data, applying multiple regression and Tobit models, and focusing on ATC-based transfers between selected European countries. To the best knowledge of the authors, no other models addressing predictions of the flow-based domain (or parameters derived from it) exist, and particularly no other uses of machine learning approaches for this forecast have been presented in the literature before.

The model

In order to compensate for the missing transparency in the flow-based calculation method, and to improve forecasting abilities of cross-border capacities available to day-ahead trading, a machine learning approach is proposed here to model the FBMC. One can distinguish two different perspectives to this problem. A forecast strategy based on time-series analysis, for instance the application of an autoregressive integrated moving average (ARIMA) model, could be addressed directly to information representing the capacities calculated from the flow-based market coupling. Such a purely statistical model could ignore the actual flow-based calculation method and the dependence of these capacities on the underlying state of the electricity system. A complementary strategy would be to reverse-engineer the capacity calculation from available past information, thus learning the non-disclosed relationship between the state of the electricity system and the resulting cross-border capacities. This approach could be coupled with available forecast models for power system dynamics, for instance load and generation forecasts, to yield forecasts of the cross-border capacities. Since a detailed exploration of both of these approaches and their possible combinations is beyond the scope of this short paper, in the following, a nonlinear autoregressive exogenous model (NARX) incorporating both perspectives is applied. Such a model relates the current value of a time series to past values of that same series, here corresponding to information representing the cross-border capacities, and to past values of an exogeneous series, here given by information representing the state of the electricity system. This model is implemented as a time-series artificial neural network (ANN). The ANN approach is chosen, as it provides a number of benefits, such as non-linearity, adaptability, response capacity (in the context of pattern classification, the network not only provides a pattern selection but also the reliability of decision making), or fault tolerance due to the massive interconnection (Haykin 2008). After learning, the chosen NARX neural network can be applied in closed-looped form, which yields a forecast of the target time series, here representing the flow-based cross-border capacities.

In the following subsections, the target values that the ANN should compute are described, and the input variables with their data sources are specified. Then, the specific design of the implemented ANN is illustrated.

Target values

The values of interest in this study are the flow-based domains for each market time unit. As described previously, the two parameter sets that define these domains are the zonal PTDF’s and the RAM. To allow a straightforward comparison between modeled and actual capacity values, in the following, a specific property of the flow-based domain, denoted as the Maximum Bilateral Exchanges MAXBEX, is considered. These values are explicitly published by the Joint Allocation Office and indicate the maximum feasible exchanges between two bidding zones (in MW) for each market time unit (which is one hour), under the assumption that the other net positions are null. This implies that the Maximum Bilateral Exchanges are not feasible simultaneously, but they form the cross-sections of the flow-based domain with the axes representing the twelve possible pairs of countries in the CWE region. In this study, MAXBEX values were chosen to be used as the target values for the neural network.

MAXBEX data was downloaded from the JAO Utility Tool in an XML format, and a parser Python code was created to convert the data to a CSV format. In order to focus on the capacity calculation for a stationary setup of the bidding zones, only data before the split between the market zones of Germany and Austria has been considered. The data set covers the period 01/01/2016 – 30/09/2018, which corresponds to 24 095 hourly time steps. For each time step, the twelve target variables associated with the MAXBEX values between each ordered pair of the four considered market zones are included. It has to be mentioned that the given links must not be understood in the physical sense. There is, for example, no direct transmission line between Germany and Belgium; and yet power trade can happen between these bidding zones, where electricity would physically flow through the lines in and between other bidding zones. Figure 2 shows a statistical analysis of the published MAXBEX values for the time period under consideration. It can be seen that available capacities increased over the years for many exchange links. The autocorrelation of the MAXBEX time series of the different links is illustrated in Fig. 3. The MAXBEX values show a high autocorrelation, which suggests that despite possible fluctuations, there is an apparent relationship between subsequent capacity values. For higher time lags, this autocorrelation decreases, while showing diurnal cycles.

Fig. 2

Arithmetic mean and standard deviation of MAXBEX capacities on all links. Note that capacities are not symmetric, i. e. MAXBEX from one country to the other is not the same as from the latter country to the former

Fig. 3

Autocorrelation of the MAXBEX time series for time lags up to 168 hours (one week). The blue line depicts the average over all links, the shade the corresponding standard deviation

Input variables

The calculation of parameters used to obtain the flow-based domain relies strongly on forecasts, such as the D2CF, GSK’s and PTDF’s. In addition, non-standardized constraints based on TSOs’ experience, like the FAV, EC, and RA are included. As this information is not publicly available, it was decided to take only power generation per type and load time series per country as the input data for the ANN. All input data was retrieved from the ENTSO-E Transparency Platform through the use of the Python client for the ENTSO-E API.

In order to retrieve and process the data for the different queries and countries, a Python-based tool was developed. The tool adjusts, if necessary, data from a quarter-hour or half an hour frequency to hourly time units. Finally, all types of generation and the load for the four CWE zones were put into a single Pandas frame and then converted to a CSV file for further processing. The raw data obtained from the retrieval tool in CSV format is the generation per production type (both renewable and conventional) and the load in hourly time units.

ANN architecture

In order to obtain a forecast of some essential features of the flow-based capacity domain, a time series neural network (Brockwell and Davis 2016) was developed in a Matlab R2018b environment. More specifically, a nonlinear autoregressive model with exogenous input (NARX) was selected to predict the time series of MAXBEX (Hudson Beale et al. 1999).

The defining equation for the NARX model applied in this contribution is

$$ \mathbf{y}(t)=\mathbf{f}\left[\mathbf{y}(t-1), \mathbf{y}(t-2),\mathbf{u}(t-1), \mathbf{u}(t-2)\right]~. $$

(1)

Here, y(t) is a vector representing the target time series at time t, i. e. the MAXBEX capacities for each link, and u(t) is a vector including the exogenous input time series at time step t. The standard NARX neural network is a feedforward network, with a sigmoid transfer function in the hidden layers and a linear transfer function in the output layer (Hudson Beale et al. 1999). The Levenberg-Marquardt supervised training algorithm is used for the learning of the network.

The learning, validation, and testing of the network occurs in open-loop mode. In this mode, according to Eq. (1), a forecast of the target time series only one time step ahead is realized. Once the network is trained, for a longer-term forecast it can be applied in closed-loop mode. In this mode, only two initial values of the target time series are used by the model, whereas for subsequent time steps, the function f in Eq. (1) feeds back the predicted values y(t) into the model. Nevertheless, the exogenous time series u(t) is assumed to be given for the closed-loop forecast. For instance, using the closed-loop mode for a forecast of the next twelve hours of the target values, the model uses the past two values of the target time series plus the past two and next eleven values of the exogenous input time series. Although in reality, this information would not be available, the choice of this model accounts for the potential availability of accurate forecasts for the input data, i. e. the load and generation time series. For such forecasts, the assumed actual values are representing the upper limit of accuracy.

Results and discussion

A NARX network with 20 neurons in the hidden layer and two delays was applied to the target and input time series for the period 01-01-2016 to 31-08-2018 (32 months). For the learning process, the time series has been divided into a training set with 70 %, a validation set with 15 % and a testing set with 15 % of the data. The learned NARX network then has been used in closed-loop mode to forecast for every hour of the period 01-09-2018 to 30-09-2018 the MAXBEX values of each ordered pair of CWE countries for the next up to twelve hours. This forecast has been compared to a simple carry-forward process. Note that due to the autocorrelation properties of the time-series (see Fig. 3), the carry-forward method is expected to yield a significantly better forecast than a random value in the interval of possible MAXBEX values, but will show an increasing error for longer time lags. Figure 4 displays the average root-mean-square deviation (RMSD) for both forecasts for lags of one hour up to twelve hours. The RMSD values have been normalized with respect to the average MAXBEX for each border. As a baseline, also the average normalized RMSD between the target series and a random time series in the range of maximum/minimum MAXBEX, and a randomly shuffled version of the target time series have been calculated. This yields an average relative RMSD of 48.3 % and 41.7 %, respectively (for corresponding value for each link consult Table 1).

Fig. 4

Average normalized RMSD between the target time series and the forecast based on simple carry-forward (blue bars) and a NARX network (orange bars). Forecast time lags range from one hour to twelve hours. Errorbars denote the standard deviation of the normalized RMSD over the twelve different pairs of countries

Table 1 RMSD between the forecast and the target time series of the MAXBEX of each pair of countries, normalized by the average MAXBEX

The results show that the one-hour forecast on average has an RMSD of 12 % of the average MAXBEX. For this lag, the simple carry-forward method yields almost the same accuracy as the application of the NARX network. For larger lags, the NARX network performs better than the carry-forward method, for instance with an RMSD of 25 % of the average MAXBEX for the NARX method compared to 34 % for the carry-forward method for the twelve-hour forecast. The link-specific results presented in Table 1 confirm this trend for all pairs of countries, but with considerable variation in the forecast accuracy. Based on the NARX network, the RMSD in terms of the average MAXBEX for the one-hour forecast ranges between 7.6 % for the link NL →BE and 16.8 % for the link DE/AT →NL. For the twelve-hour forecast, the spread is higher, with values from 14.7 % for the link FR →BE to 34.2 % for the link DE/AT →FR. Table 1 also gives the normalized RMSD between the target time series and a random time series in the range of the minimum/maximum MAXBEX, as well as the normalized RMSD between the target time series and randomly shuffled counterpart. These values serve as a baseline representing a trivial forecast. It is apparent that both these trivial forecasts and the simple carry-forward forecast mirror the accuracy of the forecast derived from the NARX network (for instance a high RMSD for the link DE/AT →FR, and a comparatively low RMSD for the link FR →BE). This indicates that the better accuracy of the NARX-based forecast for some links is related to properties of the corresponding MAXBEX time series, not necessarily to a better forecast performance of the network itself. Results are only displayed for a lag of 1 – 3 and 10 – 12 hours for brevity; it can be seen that RMSD increases strongly with longer lags in the first hours, and less in the last hours forecasted here.

Figure 5 provides some more details about the distribution of the errors for the carry-forward method and the NARX network, respectively. For the two directions of the link between France and Germany/Austria one observes a comparatively low (FR →DE/AT) and a comparatively high (DE/AT →FR) forecast error. In Fig. 5, the forecast error (predicted target value minus real target value) is plotted for both orientations as a scatterplot, with the x-value determined from the carry-forward method, and the y-value derived from the NARX network. For the FR →DE/AT orientation, a close correlation for the one-hour forecast can be observed, which shows that in this case, both methods yield similar forecasts (and corresponding errors). The pronounced values at x=0 illustrate that the MAXBEX value of this link often remains unchanged. This leads to an exact forecast with the carry-forward method, while the NARX network doesn’t achieve a similarly (actually artificially) high accuracy. For the twelve-hour forecasts, the low variability in the MAXBEX time series for this link is reflected by three pronounced stripes, whereas overall, the correlation between the forecast errors is reduced. This indicates that for the higher lag, the NARX network yields considerably different forecasts. A similar behavior is observed for the link DE/AT →FR, but with higher variability and spread in the forecast errors.

Fig. 5

Error per time step for one-hour forecast (left) and twelve-hour forecast (right) for the link FR →DE/AT (top) and the link DE/AT →FR (bottom). The x-value and the y-value of each point represents the difference between the real value and the predicted value at a certain time according to the carry-forward method and the NARX network, respectively

Analogously, calculations have been done for different parameters of the NARX network, including a higher number of delays or a high number of neurons in the hidden layer. Although the details of the forecast depend on these parameters, no overall significant improvement of the forecast accuracy for other parameter choices was observed.

Conclusion

An Artificial Neural Network was used to model the MAXBEX time series, a specific characteristics of the flow-based capacity domain of cross-border power exchange. This was done because the FBMC method, as implemented in the CWE region, has been criticized for lack of transparency and ambiguous technical definitions, which has drawn distrust to the method. Incomplete information makes it difficult for market participants in the CWE coupled day-ahead market to do precise market forecasts. It was investigated whether an ANN model could fill the gap of missing data on available transfer capacities between coupled countries.

The chosen NARX network combines information on past allocated capacities as well as generation and load data to predict future MAXBEX values. This approach was chosen as a middle ground between reverse-engineering the MAXBEX values from data describing the market state and the grid infrastructure, and a purely time series-based analytic approach based on properties of the MAXBEX values themselves.

Although in general, the NARX network yields a better forecast than a simple carry-forward method, it can be concluded that the accuracy of the prediction is still too low to provide reliable information to market participants. Nevertheless, the findings presented in this contribution can help to direct future research on this topic. It was observed that the MAXBEX values often do not change over several hours, despite changes in the load and generation time series of the associated countries. This indicates that the MAXBEX values do not represent the variability of the flow-based domain sufficiently to assess the functional interdependencies hidden in the flow-based capacity calculation process. Similarly, the presented results suggest that the load and generation data do not adequately support the time-series modeling inherent in the NARX network to provide an accurate forecast of the MAXBEX values. This information should be supplemented for instance with data on predicted line outages, newly available transfer capacity, or changes in the market structure.