Open-access tools for the modelling and simulation of electricity markets

Abstract

Globally, the amount of renewable energy generation is increasing, which raises the complexity of operating electrical grids to maintain stability and balance and boosts the need for developing new electricity market (EM) models fitting this new reality. To test, study, and validate the possible effects of novel EM designs, simulation techniques are frequently employed. This work proposes the use of two open-access tools for the modeling and simulation of complex EMs. These are the Electricity Markets Service (EMS) and the Spine Toolbox. EMS enables the simulation of two commonly used auction-based algorithms and the execution of three European wholesale EMs. Being published as a web service facilitates its integration with other services, systems, or software agents, such as the Spine Toolbox. The Spine Toolbox, in turn, is an open-source software for complex energy systems modeling. Combining them allows the modeling and simulation of complex EMs from the wholesale to local markets, as well as testing and validating new market designs. This work’s case study demonstrates how to use these tools to simulate the operation of the Iberian EM – MIBEL – for a month, using public data available from the market operator’s website. The results are analyzed from the perspective of the market operator and two players, i.e., a selected buyer and seller, for a specific day and the whole month.

Introduction

As a direct consequence of the environmental and climatic impacts of fossil-based energy usage, the energy policies and objectives defined by the European Union are promoting greater penetration and usage of renewable energy sources (RES) (The European Commission 2019; Hossain and Mahmud 2014). The EU has been performing a leading role as a pioneer in the development and application of climate change, environmental, and energy policies. Currently, it aims to reduce greenhouse gas (GHG) emissions by at least 40% regarding 1990’s levels, to increase the share of renewable energy consumption to at least 32%, and to improve energy efficiency at least 32.5% by 2030 (European Commission 2018). By 2050 the objective is to reach climate neutrality with a net-zero economy (European Commission 2018). Furthermore, relevant changes to electricity markets and business models were proposed by the 5 June 2019 the Directive (EU) 2019/9442 (European Parliament 2019), which deliver real alternatives to final energy consumers, increasing their empowerment, fostering new business prospects, ensuring higher service standards, and contributing to supply security and sustainability (European Commission 2014).

Along with these policies the formation of local energy communities to trade electrical energy among them by using RES along with information and communication technology (ICT) (European Parliament 2019) is also being incentivized. The objective is to promote local balance, reduce costs of electricity bills, and support a self-sustained energy community (Lešić et al. 2017; Coelho et al. 2017). This results in the conversion of consumers in prosumers which participate in community energy trading, use their generation for self-consumption and sell the surplus seamlessly. The operation of electricity markets (EMs) is significantly impacted by the intermittent nature of RES and their extensive integration, which imposes new limits and problems on the sector (Lund 2014; Moorthy et al. 2019). In order to comply with the new policies, new models and norms are continually being produced, and as a result, EMs are constantly adapting to new realities. The growing variety of business models, involved entities, and potential interactions among them also made EMs’ behavior more unpredictable and complex (Pinto et al. 2021).

In this regard, this ever-changing context poses several challenges that reinforce the need to study, experiment, and validate the operations of such a complex, dynamic, and competitive sector as the power systems. The significance of simulation and decision support tools is highlighted by the necessity to comprehend its complexity and unpredictability, the processes of new business models, the involved entities, how they interact, and how these synergies influence the outcomes of these participants (Pinto and Vale 2019; Conejo et al. 2010; Suvorov et al. 2017). As such, simulation and decision support tools must be able to deal with the continuously changing nature of power and energy systems (PES) and ensure that the stakeholders have the right tools and solutions to meet coming challenges while achieving expertise in these regulatory, economic, and financial environments.

In this context, a number of tools have arisen. In the US, the EMs restructuring gave rise to the Agent-based Modeling of Electricity Systems (AMES) (Li and Tesfatsion 2009). Using an agent-based testbed with strategically learning electric power traders, AMES examines the extent to which commonly used seller market power and market efficiency measurements are instructive for restructured wholesale power markets. The Electricity Market Complex Adaptive System (EMCAS) (Thimmapuram et al. 2008) is another agent-based EM modeling and simulation tool. It enables each agent to determine their own rules, gain knowledge from their choices, and enhance with each simulation. The goal is to emulate real-world market participants as closely as possible while contemplating their unique motivations and potential course corrections. The Multi-Agent Simulator for Electricity Markets (MASCEM) (Santos et al. 2016) is also a modeling tool to study and explore the restructured EMs. Its multi-agent model includes players with dynamic strategies, acting in several types of markets, allowing the simulation of various market models, namely day-ahead and intraday markets (asymmetric or symmetric), bilateral contracts, forward markets, and ancillary services. Hybrid simulations are also possible by combining these market models.

The value of these tools, and many more, was already confirmed both in the literature and industry. However, the majority of these tools are not accessible to the general public and are intended for specific users with in-depth domain expertise, making it challenging for novice users or students to use them. In this scope, this work proposes the use of the Electricity Markets Service (EMS) (Veiga et al. 2021), a publicly available web service for EMs simulation, in conjunction with the Spine Toolbox (Kiviluoma et al. 2022), an open-source software for complex energy systems modeling. Integrating these tools, we’ll demonstrate how to automate the simulation of the Iberian Electricity Market (MIBEL) (Mibel—Mercado Ibérico de Electricidade [Internet] 2021) for a month, i.e., February 2019, using real bids’ data gathered from the Iberian market operator (OMIE) (OMIE [Internet] 2022).

After this introductory section, an overview of MIBEL, as well as the tools used for this use case, namely EMS and the Spine Toolbox, are presented. Next, the case study section specifies the methodology applied and analizes the results obtained. Finally, the last section presents the most relevant conclusions and contributions from this work.

Iberian electricity market

The governments of Portugal and Spain unified their electric systems in the so-called Iberian Electricity Market (MIBEL) (Mibel—Mercado Ibérico de Electricidade [Internet] 2021). OMIE (the Spanish Iberian Market Operator) (OMIE [Internet] 2022) and OMIP (the Portuguese Iberian Market Operator) (OMIP [Internet] 2022) are the principal operators of MIBEL. The day-ahead and intraday wholesale EMs are under the control of OMIE. OMIP, in turn, manages the derivatives market, futures and forward markets, among others. This work focus on OMIE operation.

In 2014, MIBEL integrated the European EM for the day-ahead horizon and in 2018 for the intraday horizon. The Single Day-Ahead Coupling (SDAC) (Operador del Mercado Ibérico de Energía—Polo Español SA 2022), MIBEL’s day-ahead market, is a session of 24-h periods that uses a double auction bid mechanism to establish the market price and the total amount of traded energy for each period of the next day for Portugal and Spain. OMIE uses a split method to address each area separately if there are congestions on the interconnections between the two countries, which, in this case, leads to different market prices for each country. Each regular bid allows 25 tranches of energy-price pairs per period. For each period, demand bids decrease its price from the first to the last tranche, while supply bids increase. Complex conditions are allowed in the day-ahead for supply bids. Sellers use them strategically, and if those conditions are not met, these players leave the market negotiation as they are not interested in participating.

Intraday markets are an important tool for market players to adjust the day-ahead resulting program. The Intraday Auction Market and the Continuous Intraday Market are two possibilities offered by MIBEL (Operador del Mercado Ibérico de Energía—Polo Español SA 2022). The Intraday Auction Market is similar to the SDAC market. However, it includes six trading sessions regarding different hourly periods. To participate in this market, players must have participated in the SDAC and submitted bids in the same trading periods. In this market, buyers may sell their surplus, sellers can buy missing demand, and both types of players may submit complex conditions. In this case, more constraints are added to the ones available in the SDAC. The Intraday Continuous Market (Operador del Mercado Ibérico de Energía—Polo Español SA 2022), also known as Single Intraday Coupling (SIDC), differs from the previous in two fundamental functionalities: i) participating players can trade with other European areas as long as the cross-border transportation capacity is available, and ii) SIDC enables players to adjust their needs until one hour before delivery.

Electricity markets service overview

The Electricity Market Service (EMS) (Veiga et al. 2021; GECAD’s Electricity Market Service! [Internet] 2022) is a publicly available web service (https://em.gecad.isep.ipp.pt/) for simulating three of the main European EMs, as well as the most common auction-based pool mechanisms. MIBEL, the Iberian EM, is one of the European EMs available in EMS, along with the European Power Exchange (EPEX) (Home | EPEX SPOT [Internet] 2021) and the Nord Pool Spot market (Nord Pool Group 2021). The implementation of MIBEL EM in EMS includes both day-ahead and intraday market types with complex conditions, while EPEX and Nord Pool only consider the day-ahead market type and respective constraints, namely block offers for EPEX and block and flexible orders in the case of Nord Pool. These wholesale EMs allow the simulation of a market session for a given day and market type, i.e., day-ahead or intraday. The auction-based pool algorithms available in EMS are the double auction (Khorasany et al. 2017; Zhang et al. 2016), commonly referred to as symmetrical pool, and the single-sided auction (Marzband et al. 2017; Rosen and Madlener 2013), also known as asymmetrical pool. These algorithms provide the simulation of a market pool for a single period. The wholesale EMs implemented in EMS use the symmetrical pool auction for the day-ahead and intraday market clearings. EMS provides distinct HTTP request endpoints for each available EM and auction-based pool algorithm. More details about the service’s base URL and available endpoints are accessible on the EMS documentation webpage (GECAD’s Electricity Market Service: Documentation [Internet] 2022), as illustrated by Fig. 1.

Fig. 1

EMS documentation webpage

EMS encapsulates Java libraries developed for the operation of the Market Operator agent of the MASCEM, developed by the authors’ research team. It is available as a REST API so that it can be (re)used by software agents, web-based services, APIs, or third-party developers that may need the execution of any of the available algorithms. EMS only supports JSON as input and output, and it provides the appropriate JSON schemas for each algorithm. JSON schemas facilitate the creation of appropriate inputs for a seamless interface with EMS in addition to validating input data. Each request’s details, including the input JSON schema, request method, request path, and a sample of an input, are provided by EMS on the documentation page (GECAD’s Electricity Market Service: Documentation [Internet] 2022). Whereas the format of the wholesale EM inputs is similar, the content varies depending on the market being simulated. Table 1 presents snippets of the inputs of the three EMs, underligning their main differences.

Table 1 Snippets of input examples for simulating MIBEL, EPEX, and Nord Pool

By observing Table 1, one can see that only the MIBEL EM enables the definition of the “marketType” property. This property only accepts one of the “DAY_AHEAD” or “INTRADAY” constants. On the other hand, while MIBEL takes complex conditions (“complexConditions”) into account, EPEX and Nord Pool both take into account block offers (“blockOffers”) and orders (“blockOrders”), and Nord Pool also supports flexible orders (“flexibleOrders”). These properties are optional for each EM. Regarding the “price” and “power” properties, input values must be normalized in the same respective units. The user may specify the number of periods, the number of offers permitted per bid, the players’ bids, the session to simulate, and the number of offers per bid for each EM. When running MIBEL, the user must also select the type of market to simulate, i.e., DAY_AHEAD or INTRADAY.

The input for both the symmetrical and asymmetrical pools, in turn, is a JSON object that takes into account two JSON arrays, namely the supply bids and the demand bids lists. Table 2 presents a snippet of each auction-based pool algorithm.

Table 2 Snippets of input examples for running asymmetrical and symmetrical pools

As underlined in Table 2, the only difference between them is that, in the asymmetrical pool, only sellers are able to submit prices in their bids. Meaning that the demand bids of the asymmetrical pool do not accept a “price” property (Klemperer 1999; Sheblé 1999). The “instant” property allows to order the bids by order of submission when there is no “price” (i.e., demand bids of the asymmetrical pool) or when there are multiple offers with the same “price”.

The data flow for the EMS service from the request to its response is shown in Fig. 2.

Fig. 2

EMS data flow (adapted from Veiga et al. (2021))

EMS was created as a containerized Java REST API, providing a versatile and user-friendly approach to simulate and execute EMs. This makes it ideal for both real simulation scenarios and for researching and testing how the available markets and market pools operate. The EMS documentation page, available online, makes its integration effortless to accomplish. EMS also has a test page where users can instantly run any of the available EMs or market pools. As a result, using EMS doesn’t require installing any software nor programming any code. The user sets the market or market pool to simulate, changes the input as desired, and executes it. A button for validating the input JSON is also available as an option. Additionally, this validation is always made when using the run button. An input sample is given for each request to help users grasp the essential components of each market or market pool. Not all are necessary, though. Users can modify the examples that are given to run the algorithms so long as they are valid in comparison to the relevant schemas. The output contains the amount of energy and monetary values traded for each period globally and for each one of the players that took part in the market.

EMS offers an adaptable and simple solution for executing electricity markets, allowing a seamless integration with other applications. It can be used to simulate real-world scenarios as well as to learn how these markets operate.

Spine toolbox

The Spine Toolbox (Kiviluoma et al. 2022) is an open-source software (https://github.com/Spine-project/Spine-Toolbox, https://spine-toolbox.readthedocs.io/en/latest/) developed in the scope of the EU Spine project that enables the definition, management, and execution of increasingly complex energy system models. It allows open, practical, flexible and realistic planning of future European energy grids. It is a modular and adaptable tool offering a framework able to assemble tools, models, and data, providing a visual representation of the entire data processing chain. The Spine Toolbox provides a set of features specialized for decision-making under uncertainty. It is written in Python, allowing easy integration of Python-based tools. Also, the Spine Toolbox workflows can incorporate other data processing tools available in the open-source community.

Spine Toolbox allows the design and execution of workflows consisting of multiple tools and models, supporting problem-independent data and scenarios with user interfaces that automatically support different data structures. However, data must conform with the entity–attribute–value with relationships and classes data model. The high-level structure strives for modularity within the Spine Toolbox and the workflows it can execute. Regarding workflows, each node (i.e., tool or model) e has four main elements, namely: the input from the previous node, an output to the next node, node internal operations, and access to external data sources. The composition of multiple nodes results in a computational workflow.

The Spine Toolbox integrates different modules of the Spine project, as illustrated in Fig. 3.

Fig. 3

Spine Toolbox modules (Project Spine [Internet] 2022)

Spine’s software architecture follows the workflow control architectural pattern, using a tight integration between the toolbox and the other components. Spine supports multi-vector energy systems and performs as a generic model generator and simulator. The abstract flow-based multi-commodity energy models are defined and constructed using the Spine data structure in conjunction with the data importing and exporting tools. This data structure is built on the following abstract classes: Node; Unit; Connection; Commodity; Temporal block. A Node is a location in space where the energy from a certain vector balances. A Unit represents a device that can switch energy from one vector to another and is placed in between two nodes. A Connection is a component inserted between two nodes that can transfer energy along the same vector or allow a commodity to flow. A Commodity is a specific kind of energy flow connected to an energy vector. A Temporal block is a time period with a given time resolution.

In order to generate a model, the user interacts with the drag-and-drop functionality of the Spine Toolbox interface. This interface has a variety of abstract building blocks available, which enable the specification of relationships that must exist between them based on the system’s topology. Various parameters can be supplied to capture the physics of the flow of each commodity and the transformations between them, depending on the particular demands of the system to be described. Figure 4 presents the Spine Toolbox user interface.

Fig. 4

Spine Toolbox user interface (Started and—Spine Toolbox 0.6.12.dev0-dev.0 documentation [Internet] 2022)

Case study

This section presents a case study to demonstrate how to model the simulation of the MIBEL day-ahead market for an entire month using EMS and the Spine Toolbox. The case study considers public data gathered from the OMIE’s data repository (Curvas agregadas de oferta y demanda del mercado diario | OMIE [Internet] 2022) to set the players and respective bids for each day of the month. The month selected was February 2019. The data collected keeps the players anonymous, not being possible to know which tranches belong to each player. For this reason, in the present study, each tranche represents a bid from a player. Such resulted in a minimum of 2659 and a maximum of 3736 bids per period. Further, the number of buy and sell bids is not constant throughout the days and their hourly periods.

To ease the results interpretation and simplify their verification, the authors decided to reduce the number of players/bids per period to 2500. Despite reducing the number of players, such number still demonstrates the scalability of EMS comparing to the actual number of players of MIBEL. For instance, if we pick the maximum number of tranches per period of February 2019 (i.e., 3736) and consider that at least half these bids (i.e., 1868) were composed by two tranches per period, the number of player bids for this period would be 2802 (i.e., 1868 + 1868/2). However, in the period of February 2019 the maximum number of players per day in MIBEL was 1327 on the 25th (see file “cab_201902.zip” at (Cabecera de las ofertas al mercado diario | OMIE [Internet] 2022)). Complex conditions were also not considered. On the one hand, the complex conditions submitted by players are not publicly available. On the other hand, the aim of the case study is to demonstrate how to model and execute the simulation of an EM model publicly available at EMS within the Spine Toolbox and not analyzing specific player strategies. Considering the number of sellers is significantly lower than the number of buyers in each period, the authors decided to use all seller bids available for each period of each day, completing the remaining positions with buyer bids.

The following section demonstrates how the case study was modeled and configured in the Spine Toolbox. After, the case study scenario is presented, and the results are analyzed for a specific buyer, seller, and market operator. Monthy results are also provided and interpreted.

Methodology

EMS allows simulating a market session at a time for each wholesale EM available. To execute the day-ahead session of MIBEL for each day of February 2019, the authors used the Spine Toolbox to automate this process. To this end, two types of elements were used, namely the Data Connection and the Tool items. Figure 5 illustrates the Spine Toolbox configuration in the Design View at the center.

Fig. 5

Spine toolbox configuration

The Data Connection item, named “february_2019”, allows importing the players’ data for each session of MIBEL’s day-ahead market. It imports the data gathered from OMIE’s data repository converted into 28 JSON files, one per day, for each day-ahead session of MIBEL, making it available for the next Tool item, i.e., the “run_EMS_MIBEL_Feb2019” item.

The “run_EMS_MIBEL_Feb2019” item is the core of this example, as it enables the simulation of MIBEL using EMS for the whole month. This item allows coding scripts in Python, Julia, and GAMS, or even to run executable files. In this specific example, a Python script was used to automate the execution of MIBEL day-ahead market in EMS for the 28 days of February 2019. Figure 6 presents a snippet of the “run_EMS_MIBEL_Feb2019” item’s Pyhon script.

Fig. 6

Snippet of “run_EMS_MIBEL_Feb2019” item’s Python script

To run MIBEL’s day-ahead market for each day, this Python script uses the “EMS_BASE_URL” (see the script’s line 7) concatenated with the “mibel” endpoint, which is declared in the input file of this Spine Toolbox project. Another option would be to declare the complete EMS MIBEL URL in the script. However, this way allows us to run different EMS EMs using the same Spine Toolbox project. This Python script handles the input/output data workflow, saving the results in JSON format. These results are then sent to the next Tool item.

Finally, the “create_charts” Tool receives the previous item’s JSON output to generate the output charts to ease the analysis and interpretation of the results. This item generates charts for each player and the market operator regarding each session (day) and the aggregated results of the month (market operator only). The charts’ generation is also supported by a Python script, which stores them in the project’s folder.

Results

As previously stated, the results are provided from different perspectives. From the market operator’s point of view, the results are seen as the overall session outcomes and the market’s social welfare, as well as the monthly results overview. From each player’s standpoint, the results are seen individually according to his outcomes in each trading session.

To facilitate the interpretation of the simulation results, this subsection starts by analyzing the simulation outcomes from the market operator’s perspective of a randomly chosen day, i.e., 17th February 2019, followed by the analysis of the results achieved by a buyer and a seller, namely Buyer 1391 and Seller 3. This day-ahead session counted a total of 1330 buyers and 884 sellers. The monthly aggregated results are also presented.

Figure 7 presents the overall session results for the day-ahead session of 17th February 2019. It must be noticed that, being a day-ahead session, the simulation is, actually, determining the market-clearing for the next day, i.e., 18th February.

Fig. 7

Day-ahead session results for the 18th February 2019

From Fig. 7, the total demand and supply submitted per day can be observed. The different color tones identify the satisfied (dark colors in the bottom) and unsatisfied (light colors on the top) demand (orange) and supply (green). The blue line, in turn, represents the market price set for each trading period. A closer look at periods 9–19 seems that there was no negotiation in those periods, but there was, as will be shown next. Such has to do with the granularity. In this chart, the energy unit is GWh, and the traded energy is not enough to be seen in this dimension.

Figure 8 shows the session results achieved by Buyer 1391.

Fig. 8

Buyer 1391 Session Results for the 18th February 2019

As it is possible to see, Buyer 1391 only submitted bids for the first 16 hourly periods. Unfortunately, for this player, his demand was only satisfied from periods 12 to 16. It means this player submitted lower prices in the first periods of the day (from 1 to 11) than the last seller to sell (although not perceptible from the chart between periods 1 and 6), not being able to trade. This chart also proves that there was trading from periods 12 to 16, at least. But, as shown by Fig. 8, the maximum amount of traded energy for this player was near 8 MWh in period 16.

Figure 9 displays Seller 3’s session results.

Fig. 9

Seller 3 Session Results for the 18th February 2019

Regarding the results of Seller 3, this player submitted bids for all trading periods. However, he was only able to sell his supply in 13 periods out of 24, meaning the prices offered in his bids, from period 9 to period 19, were a bit higher than the market price (which is zero in those periods). In the remaining periods, this player was able to sell all of his supply.

Finally, Fig. 10 presents the overall market results for each day of the month from the perspective of the market operator.

Fig. 10

Overall Day-ahead Session Results for February 2019

Analyzing Fig. 10, one can see the aggregated demand and supply for each day of February 2019, similar to Fig. 7 regarding the satisfied and unsatisfied power. However, in this case, the blue line represents the average market price for each day. It is also visible that the amount of supply is significantly higher than the demand. It is clearly a consequence of the way data was selected for this specific scenario. It is also interesting to observe the average market price variation along the days of the month. The highest average price was settled on the 19th of February (16.76 €/MWh), while the lowest was on the 1st (0.0 €/MWh).

Using the Spine Toolbox, we modeled a whole month of the Iberian day-ahead wholesale market using the publicly available MIBEL EM model of EMS. Besides, using the Spine Toolbox also allowed to automatically generate charts from the different stakeholders’ points of view for a more comprehensive study and analysis of the market sessions’ outcomes, as well as for an overall monthly perspective. While the daily results returned by the EMS are directly stored locally and used to generate daily charts, the monthly aggregated data is calculated using the Python script of the “create_charts” Tool item.

Conclusions

The use of RES is rising, making it hard to operate electrical grids to preserve stability and balance, increasing the demand for new EM models appropriate for this brand-new reality. Simulation approaches are commonly used to test, investigate, and validate the potential consequences of innovative EM designs. A number of tools arose in this context and have proven their valuable contribution. However, it can be difficult for inexperienced users to use these tools since they are usually designed for specific users with in-depth domain expertise. Additionally, most of them are not publicly available.

In this scope, this work proposes the integration and use of the EMS and Spine Toolbox open-access tools for modeling and simulating EMs operation. EMS is a publicly available web service for EMs simulation, while the Spine Toolbox is an open-source software for complex energy systems modeling. Combining them allowed the modeling and simulation of MIBEL day-ahead market session for a whole month. To simulate other EM available in EMS, the users only need to update the HTTP request endpoint to the respective market.

The presented case study was designed to demonstrate how to automate the simulation of the Iberian Electricity Market for February 2019, using real data collected from OMIE’s file repository. EMS is a proven effective tool for EMs simulation. This case study also shows its scalability and ability to handle large amounts of data. Although the Spine Toolbox’s main directive is to simulate energy system models, it has a broad scope of usage, also demonstrated with this case study. It enabled not only the execution of a monthly day-ahead market as well as the generation of aggregated results and the respective graphical representations. The results were analyzed from different prespectives, namely the market operator, a buyer and a seller, for a specific day and the whole month.

In future work, the authors intend to include another web-based service for the power flow calculation and validation of the day-ahead market results from the viewpoint of a system operator. Such will allow to validate the EM transactions according to the characteristics of an electrical grid. If simulating the wholesale market, it will consider a transmission grid. Otherwise, in a local EM simulation it considers a distribution grid. Also, additional metrics can be calculated and added for the results’ analyses and comparison. For instance, regarding the market operator, additional charts can be generated for each period’s pool to illustrate which players trade or not in a given period. In the perspective of market players, new charts can also be generated for aggregated monthly results. Furthermore, integrating the power flow service will make available new data regarding the electrical grid power flow that can also be translated to specific charts regarding different metrics, such as the buses voltages, lines rates, and power losses, to name a few.