Agent-based dynamic optimization of local controller configurations in converter dominated electricity grids using decoder functions

Abstract

With an increasing share of distributed energy resources (DER) in the electrical energy system it is becoming more and more important that DER not only take part in active power provision but are also involved in the provision of ancillary services like frequency control or voltage regulation. Due to the large number of DER connected to the lower voltage levels via power-electronic converters the distribution grid evolves from a formerly mostly passive to a highly active system with a high number of actuating variables distributed over multiple stakeholders. The coordination and optimization of this kind of distribution grid requires new control and optimization approaches, not only with regard to the distribution grid itself, but also with regard to the coordination with the overlying transmission grid. This abstract presents first ideas of a PhD-project that aims to use machine learning surrogate models and decoder functions for agent-based dynamic optimization of local controller configurations particularly with regard to voltage regulation. Decoder functions derived from machine learning surrogate models abstract optimization problems from technical system specifications and allow for constraint-free optimization with standard optimization heuristics such as evolutionary optimization methods.

Background

The electric energy system has undergone major changes in recent years. Gradually, large scale fossil or nuclear power plants are being replaced by distributed energy resources (DER) often based on renewable energies like wind power and solar energy. In contrast to large scale thermal power plants, DER are mostly connected to the low and medium voltage grids. Up to now, ancillary services like frequency or voltage control, which are essential for a stable operation of the electric power system, are predominantly provided by the remaining large scale thermal power plants (Braun 2009). To reduce carbon dioxide emissions large coal-fired power plants are to be replaced by further expansion of DER. This makes it increasingly important that DER not only contribute to active power provision, but are also involved in the provision of ancillary services (Deutsche Energie-Agentur GmbH 2012).

DER such as photovoltaic plants or wind turbines are normally connected to the grid via power-electronic converters. In the future converter dominated grid DER must take over ancillary services so far merely provided by large scale power plants (e.g. primary reserve for frequency control or reactive power provision for voltage regulation in the transmission grid). At the same time, DER meet new challenges, which arise from their distributed nature, their connection to the low- and medium voltage grid and the fluctuating and to some extent uncertain supply of wind power and solar energy. Among these challenges are voltage deviations and voltage band violations in the distribution grid, congestion of lines and transformers and a changing dynamic behavior of the power system.

Power-electronic converters are technically capable of providing arbitrary reactive power along with active power as long as current and voltage limitations are met. In Germany, reactive power control capability for low- and medium voltage connected DER is defined in the technical guidelines VDE AR 4105 (low voltage) and the BDEW Medium Voltage Guideline (VDE, FNN 2011; BDEW Bundesverband der Energie- und Wasserwirtschaft e.V 2008). The guidelines define local control characteristics such as cosφ(P) or Q(U) for reactive power supply of converter-connected DER. At the same time, grid operators incorporate new operating equipment to solve voltage balancing problems. This is changing the formerly largely passive distribution grid to an active system with a great number of actuating variables, which have to be coordinated to optimize overall system operation and to avoid controller conflicts (i.e. adverse control actions directed at one or more units based on different control targets and stakeholders) leading to inefficiencies or even instabilities of the system.

The DFG-project “Reliable Operation of Converter-Dominated ICT-Reliant Energy Systems”, in the context of which this work is conducted, aims to develop and evaluate a decentralized agent-based control and information exchange structure for grid operation with the help of DER and their coordination with top-level control centers. The main idea is to combine local converter-based control with multi-agent based optimization to form a hybrid control system as depicted in (Fig. 1). The converter-based control guarantees real-time behavior and relaxes time constraints for the overlying multi-agent system, which in turn ensures long-term optimization of system operation and controller conflict mitigation by dynamically optimizing the configuration of the local controllers.

Fig. 1

Hybrid control system

For optimization a formal description of the system boundaries is needed. This is typically done by providing constraints next to optimization objectives. For dynamic optimization of controller configurations this approach is not suitable, as it would require the reformulation of the optimization problem each time the system changes, for example when new DER are connected to the system or when the flexibility of an energy unit with regard to one service is temporarily reduced by other higher prioritized services.

A promising approach to deal with dynamic optimization problems is to use a machine learning surrogate model to describe the flexibilities of the underlying systems. A surrogate model is a black-box model that abstracts from the technical model of a complex system and approximates the observable behavior (without the internal causal relations) based on a limited set of sampled data. From the surrogate model a so-called decoder can be derived, which maps the unrestricted search space to the solution space and thus allows for contraint-free optimization with standard optimization heuristics such as evolutionary optimization methods (Bremer 2015).

In the PhD-project presented in this abstract, this approach shall be applied and evaluated for dynamic agent-based optimization of local controller configurations, particularly with regard to voltage regulation in active distribution grids. As mentioned before, the decoder approach does not require the reformulation of the optimization problem when parameters of the underlying system change and is thus more flexible and easier to automate. But still surrogate model and decoder have to be learned resp. derived from scratch in case of system changes, which is computationally expensive and time consuming. Therefore, one main research goal in this PhD-project is to develop a method, which allows for dynamic adoption of surrogate model and decoder to changes in scenario settings.

To allow for the optimization of higher grid levels based on lower level flexibilities, a description of the aggregated flexibilities resp. constraints of downstream entities is required. Therefore, another important research goal is to develop fast training set sampling techniques for ensembles (groups) of DER avoiding the distribution folding problem (the distribution of the sum of independent random variables results from the convolution of the individual random variables) when combining independently sampled training sets from different downstream entities for training aggregated decoders in upstream agents.

Research questions

As already mentioned in the previous section, the overall objective of the PhD-project presented in this abstract is to answer the following main research question:

• (RQ 1) How can decoder functions be used for solving dynamic optimization problems and mitigating controller conflicts in agent-based distributed optimization of local controller configurations?

So far, two important requirements for the applicability of the decoder approach in the above-mentioned use case have been identified: First, sampling strategies for groups of DER are required to enable learning of aggregated flexibilities, which is needed for optimization on different grid levels. Second, methods for an on-line update of decoders are required. These update methods not only allow for fast adjustment to changes in downstream units (e.g. operational state, behavior or commitment to higher priority ancillary services) but also open up the possibility to mitigate controller conflicts by modelling a prioritization of arriving set points from upstream units, rephrase them as restrictions of the feasible regions and finally update the concerned decoder accordingly. From these requirements two sub-research questions have been derived:

• (RQ 1.1) How can training sets for surrogate models of aggregated flexibilities from downstream units be sampled avoiding the distribution folding problem, which occurs with the naive approach of combining independently sampled training sets from downstream units?

• (RQ 1.2) How can existing surrogate models and decoders be adopted on-line to changes in underlying scenario settings to avoid computationally expensive and time consuming retraining of decoders from scratch?

From the research questions mentioned above two main development objectives (DOs) can be derived: First the development of a fast training set sampling technique for training groups of DER, which avoids the distribution folding problem (DO 1). And as a second development objective, the extension of the decoder approach by a method which allows for updating of decoders based on the mere difference between two successive situations (DO 2). Figure 2 shows how the DOs fit into the decoder approach. The decoder approach is explained in more detail in the next section.

Fig. 2

Process chain of decoder approach according to (Bremer 2015) and intended adaptions/extensions (denoted by dashed lines)

Related work

In this section related work from the fields of constraint-free optimization with decoders and agent-based ancillary service provision by DER is presented.

In (Bremer 2015) constraint-free optimization with support vector decoders is used for predictive scheduling of energy units. From a methodological perspective, the concept presented in (Bremer 2015) realizes the abstraction from the technical system in several steps (see Fig. 2): First, in a simulation based sampling process for each participating DER a set of example schedules is generated. In a second step, these schedules are not interpreted as time series but as vectors. This allows for a geometric interpretation of the solution space and opens up the possibility to learn the surface boundary, which separates valid from invalid schedules, by means of support vector data description (SVDD). Finally, this type of description offers a possibility to derive a decoder, which maps the unrestricted search space to the restricted solution space, and thus allows for constraint-free optimization. (Fröhling 2017) suggests a modified variant of the approach in (Bremer 2015), which allows for the abstract description not only of single energy units but also for coalitions of energy units. For this purpose, in (Fröhling 2017) an appropriately adapted search space model is developed, the so-called cascade classification model. It consists of a cascade of overlapping low-dimensional feature classifiers taking advantage of the fact that neighboring time steps employ a high correlation and the correlation decreases with increasing distance between two time steps. One problem with this type of search space representation is that so far it lacks an appropriate decoder. Furthermore by exploiting the high correlation between neighboring time steps, the cascade classification model takes advantage of a problem specific property of predictive scheduling, which hinders the applicability to other problem statements. Nevertheless, it is a promising entry point for answering RQ 1.1.

Concerning dynamic adaption to changes in scenario settings (RQ 1.2) a master’s thesis was conducted at the Energy Informatics Department of Oldenburg University (Faerber 2018). The thesis was originally intended to develop approaches for cutting and combining search space models by the example of predictive scheduling of CHP units and heat pumps. However, it turned out that for the intended use case even small changes in heat demand lead to largely disjunct search space models. The similarity measures for search space models used in the thesis are relevant for RQ 1.2 nevertheless. Regarding RQ 1.2 there exist few approaches for implementing methods for adaptive learning with support vector approaches in general (e.g. (Gieseke et al. 2012; Laskov et al. 2006)), but for the case of SVDD, comparable support is not yet available. Nevertheless, concepts for incremental learning of SVDD models exist (e.g. (Yin et al. 2014)) and may be candidates for adaption.

In (Lehnhoff et al. 2013) an integrated approach for identifying distributed coalitions of agents representing units capable of providing frequency response reserve is introduced. For each participating unit individual droop control parameters are tuned taking into account opportunity costs, device specific reliabilities and small-signal stability of such a coalition. Implications of frequency response for voltage levels are not considered in this paper. In the literature, agent-based approaches for the provision of ancillary services by DER are often discussed in the context of the microgrid concept. Microgrids are distribution systems with DER, storage devices and controllable loads, operated connected to the main power network or islanded, in a controlled, coordinated way (Hatziargyriou 2014). In connected mode, from the grid operators point of view, a microgrid can be regarded as a controlled entity within the power system that can be operated as a single aggregated load or generator and also provide ancillary services for the distribution grid it is connected to (Hatziargyriou 2014). In (Olivares et al. 2014) an overview of agent-based concepts applied to microgrid control is provided. The use of decoder functions in optimization of local controller configurations so far has not been discussed in microgrid related literature.

Methodology

To answer the research questions the following steps are envisaged: In a first step the degrees of freedom of DER usable for frequency and voltage control and possibly conflicting optimization goals are assessed. In a second step simulation models of different types of DER including their power-electronic converters reflecting these degrees of freedom are required. They serve as input for the Ensemble Sampling (DO 1) which is to be added to the decoder approach from (Bremer 2015) (see Fig. 2). The Ensemble Sampling can be taken as a distributed optimization problem with the objective of optimizing the distribution of instances in the resulting aggregated ensemble training set. To evaluate the distribution of the aggregated training set, suitable metrics have to be found. Candidates for this were identified in (Bremer and Lehnhoff 2018a; Bremer and Lehnhoff 2018b). In these articles also two central approaches to overcome the folding problem in ensemble scheduling using a Covariance Matrix Adaption Evolution Strategy (Bremer and Lehnhoff 2018a) and Simulated Annealing (Bremer and Lehnhoff 2018b) are proposed. Both are to be investigated with regard to their applicability in a fully distributed approach for example with the distributed optimization heuristic COHDA (Hinrichs and Sonnenschein 2017). Work on DO 2 is still at a preliminary stage. A possible first step would be to apply the methods for comparing the similarity between consecutive situations proposed in (Faerber 2018).

The hybrid control system depicted in (Fig. 1) and the DOs presented before are to be evaluated by means of co-simulation based computer experiments. The co-simulation framework mosaik (Lehnhoff et al. 2015) is intended to be a central component of the evaluation framework and shall be used to couple the MATLAB-based steady-state electricity grid simulator provided by the project partners, the Python-based agent framework aiomas (Scherfke 2014) and in a later stage an ICT simulator, which has yet to be defined.