Finding individual strategies for storage units in electricity market models using deep reinforcement learning

Energy Informatics

Table 1 Overview of existing electricity market models

Model family	Behavior modeling method	Use case	Limitations	Examples
Optimization-based	Optimization	System expansion Investment planning	System optimal behavior Assumption of perfect foresight	ELMOD (Leuthold et al. 2012) PyPSA (Brown et al. 2018) Oemof (Hilpert et al. 2018)
Equilibrium-based	Game theory	Emerging strategies Emerging market dynamics	Mathematically challenging Computationally expensive Inflexible to market changes	Orgaz et al. (2017) Liu and Conejo (2021) Huppmann and Egging (2014)
Equilibrium-based	Static rule-based	Static market dynamics Market results	No strategy adaptation to changing environment	flexABLE (Qussous et al. 2022) AMIRIS (Deissenroth et al. 2017)
Agent-based	Classic RL	Emerging strategies Emerging market dynamics	Limited performance Discrete action space No multi-agent learning	AMES (Tesfatsion and Battula 2020) Weidlich (2008) MASCEM (Pinto et al. 2014)
Agent-based	Novel DRL	Emerging strategies Emerging market dynamics	Computationally expensive Only applied to small cases	Du et al. (2021) Ye et al. (2019)