A comprehensive overview on demand side energy management towards smart grids: challenges, solutions, and future direction

Abstract

Demand-side management, a new development in smart grid technology, has enabled communication between energy suppliers and consumers. Demand side energy management (DSM) reduces the cost of energy acquisition and the associated penalties by continuously monitoring energy use and managing appliance schedules. Demand response (DR), distributed energy resources (DER), and energy efficiency (EE) are three categories of DSM activities that are growing in popularity as a result of technological advancements in smart grids. During the last century, the energy demand has grown significantly in tandem with the increase in the global population. This is related to the expansion of business, industry, agriculture, and the increasing use of electric vehicles. Because of the sharp increase in global energy consumption, it is currently extremely difficult to manage problems such as the characterization of home appliances, integration of intermittent renewable energy sources, load categorization, various constraints, dynamic pricing, and consumer categorization. To address these issues, it is critical to examine demand-side management (DSM), which has the potential to be a practical solution in all energy demand sectors, including residential, commercial, industrial, and agricultural. This paper has provided a detailed analysis of the different challenges associated with DSM, including technical, economic, and regulatory challenges, and has proposed a range of potential solutions to overcome these challenges. The PRISMA reviewing methodology is adopted based on relevant literature to focus on the issues identified as barriers to improving DSM functioning. The optimization techniques used in the literature to address the problem of energy management were discussed, and the hybrid techniques have shown a better performance due to their faster convergence speed. Gaps in future research and prospective paths have been briefly discussed to provide a comprehensive understanding of the current DSM implementation and the potential benefits it can offer for an energy management system. This comprehensive review of DSM will assist all researchers in this field in improving energy management strategies and reducing the effects of system uncertainties, variances, and restrictions.

Introduction

The mechanism that allows electricity to be transmitted from power plants to energy customers is known as the “power grid”. This electricity goes from the power plant through the substations in one direction before it reaches the energy user when the voltage is changed via the transmission and distribution line (Piette et al. 2004).

The need for energy has expanded significantly along with the increase in the global population during the last century. The International Energy Agency predicted that by 2030, global electricity consumption will have increased by more than 50% (Freeman 2005). This is related to the growth of business, industry, agriculture, and the increasing use of electric vehicles (Martínez-Lao et al. 2017).

Due to the sharp increase in global energy consumption, it is currently extremely challenging to manage problems such as controlling power loss, dependability, efficiency, and security challenges. A “smart grid,” which combines self-monitoring, self-healing, pervasive control, adaptive, and islanding mode mechanisms, has been suggested to allow for energy transit from the point of production to the site of consumption to solve these problems (Fang et al. 2011; Xu et al. 2016b).

The hardware and software components of smart grids provide the utilities the capacity to immediately identify and address any problems that could develop between the customers and the producing plants and endanger the consistency and quality of the power supply. The smart grid component is classified as shown in Table 1.

Table 1 Smart grid component (Moreno Escobar et al. 2021)

Electrical energy management is used to reduce energy expenses and alter the load profile on both the supply and demand sides. The goal of supply side management (SSM) is to make energy generation, transmission, and distribution more operationally effective. SSM has many advantages, such as maximizing customer value by ensuring efficient energy production at the lowest practical cost, satisfying demand for electricity without the need for new infrastructure, and limiting environmental impact. However, supply-side management is affected by fuel price volatility because of its techniques for managing thermal generators (Haffaf et al. 2021).

Demand side energy management (DSM) reduces the cost of energy acquisition and the associated penalties by continuously monitoring energy use and managing appliance schedules (Dranka and Ferreira 2019). In order to lower peak loads, control time of use (TOU) levels of power demand, evaluate user profiles for electricity loads, lower carbon emissions, and provide consumers a choice of preferred energy source, the electrical industry originally developed the DSM in 1970 (Gellings 2017; Maharjan et al. 2014).

Several nations, including the UK (Warren 2014), China (Ming et al. 2013), North America (Wang et al. 2015), and Turkey (Alasseri et al. 2017), have adopted the Energy Management System (EMS), which is the most effective way to save energy costs while preserving system stability. However, there are still several constraints that prevent EMS from being fully implemented in underdeveloped nations. These components might be related to:

• Adopting an EMS comes at a significant expense, and the long-term rate of return on investment is low.

• Time-varying electricity tariffs are ideal. Making the switch from an older model to a newer one is tough for electrical companies and merchants.

• Not all stakeholders benefit equally from the transformation;

• Population knowledge has a significant impact on implementation speed.

• Upgrading the network infrastructure could be very expensive for the system, and bidirectional power flow is still in the research stage, which could delay the idea of EMS.

Cappers et al. examined the prospective benefits of DSM to the electrical power system as illustrated in Fig. 1. These enhancements have the potential to provide considerable secondary advantages, such as decreased losses and premature aging (Cappers et al. 2010).

Fig. 1

Benefit achieved by the DSM program (Cappers et al. 2010)

To effectively reduce costs without the involvement of operators, a control system that selects the energy sources to power different loads according to the period of the energy demand is required. The most frequently used controllers in the literature to accomplish the aforementioned goal are programmable logic controllers (PLC), supervisory control and data acquisition (SCADA), building management systems (BMS), energy management systems (EMS), and automation systems (home automation systems, etc.) (Jabir et al. 2018).

Numerous studies have focused on the load control strategies used by DSM (Jabir et al. 2018), the roles played by DSM in the electricity market (Morgan and Talukdar 1979), the economic benefits of DS (Conchado and Linares 2012), the impacts of DSM on the commercial and residential sectors (Esther and Kumar 2016; Shoreh et al. 2016), the interactions between DSM and other smart grid technologies (Khan et al. 2015b), the business strategies used by DSM (Behrangrad 2015), the impacts of DSM on the reliability of the power system (Kirby 2006), the optimization strategies used by DSM (Hussain et al. 2015; Vardakas et al. 2014), and the load control strategies (Khan et al. 2016).

The electrical market has just entered a phase of transformation where one of the primary objectives is to lower peak demand while making the greatest use of all resources available. Over the world, incentives have been created to motivate consumers by offering them a range of monetary benefits and different power rates at different load-dependent intervals. Dynamic pricing is an inherent aspect of the home energy scheduling problem in this situation since it encourages consumers to move their load from the on-peak to the off-peak period. Marginal cost, load pattern, social considerations, and the power utility’s capacity are the main variables utilized to define the energy tariff structure (Phuangpornpitak and Tia 2013).

All consumers must benefit from greater DSM effectiveness, which requires detailed consumer consumption data. With the advent of advanced metering infrastructure (AMI), utilities may collect all consumer consumption data, and various DSM programs may be developed depending on the data attributes. The scale, complexity, and unpredictability of smart meter data are addressed for use in load forecasting and DSM systems. When implementing DSM, it is important to consider some important factors, including the load profile of an appliance, the integration of renewable energy, load categorization, constraints, dynamic pricing, consumer categorization, optimization techniques, consumer behaviors, problems with electricity data, enough knowledge, a solid framework, and smart grid technology with its intelligent applications (Khan and Jayaweera 2019).

As the load profile of appliances heavily depends on the stochastic behavioral patterns of consumers and the surrounding environment, developing a universal DSM optimization method that works for all types of consumers is quite challenging. It is also difficult to develop a generic forecasting system that can accurately predict the power consumption of various appliances for different users. Thus, the load profile of the consumers’ appliances plays a crucial role in the development of a consumer-specific optimization algorithm that takes into consideration their preferences for comfort (Sharda et al. 2021). Different appliances have different characteristics, power requirements, and operating styles. For DSM optimization, the right grouping of home appliances based on consumer preferences or behavior is essential. Survey techniques, bottom-up models, top-down models, and hybrid methods have all been explored to do accurate appliance forecasting. Nonetheless, it is believed that utilizing smart appliances and meters is the best option (Proedrou 2021).

The effectiveness of demand scheduling optimization depends critically on customer classification. Customers should be made active DR participants by ensuring their comfort which is done by arranging various appliances within their own time and temperature ranges. likewise, customers may be grouped according to their behavior and demand (Liu et al. 2015). It is necessary to overcome consumers’ resistance to adopting and taking part in DSM programs, and this may be done by creating consumer awareness initiatives that will urge customers to use the DSM system. Increased expenses for installing and maintaining control devices must also be taken into account. It is necessary to address the impact of the accelerated development of storage systems brought on by the availability of cheap local storage. The majority of the increasing energy consumption is caused by thermostatically regulated equipment. Hence, there is a lot of room for energy savings via effective management of these devices. The following suggestions, which were emphasized in Ming et al. (2015) may truly aid in overcoming the difficulties associated with DSM.

• The planning for the power sector and regional economic growth should all use DSM as a resource. To be properly implemented, rules, laws, and regulations need to be created by the governments and electricity grid businesses.

• It is important to gradually establish the DSM’s assessment and monitoring methods. It might be put into practice by constructing a post evaluation system for DSM, an expert committee and oversight mechanisms for DSM, an energy efficiency evaluation system for performing energy inspections, and an analysis of the energy efficiency criteria for electrical equipment. It is also necessary to promote the creation and improvement of relevant supporting policies for DSM.

To fulfill the expanding energy demand and reduce the rising CO₂ emissions, energy generation from renewable energy sources has become more crucial. Several DSM methodologies are utilized to govern distributed energy resources, renewable energy resources, and storage devices to ensure the overall system operates as effectively as feasible. It is difficult to plan for optimal energy requirements since renewable energy sources and power costs are unpredictable. Each operating location must be thoroughly analyzed to pinpoint the areas where natural capital provides notable advantages for certain types of renewable energy consumption. Several optimization techniques, such as mixed-integer linear programming (MILP) (Erdinc et al. 2014), two-stage robust optimization (Liu and Hsu 2018), and heuristic optimization, have been proposed to enhance the scheduling of distributed energy sources (Luo et al. 2018). The ability of the electric vehicle to function as a battery energy storage system has also been researched for applications like vehicle-to-home (V2H) and vehicle-to-grid (V2G) (Erdinc et al. 2014).

An effective management system for scheduling various smart appliances and integrating renewable energy (RES) like solar, wind, distributed micro-generators, and energy storage devices, including plug-in electric automobiles and batteries, may be offered to DSM to provide an optimal management system (Qureshi et al. 2021; Wang et al. 2019; Wu et al. 2019). Electricity prices have a big impact on how much energy people use (Rahman and Miah 2017; Zhang and Peng 2017). But both the analysis and reshaping of the load profiles as well as the load market’s load patterns in SG may be handled by the DSM. This method lowers energy prices, carbon emissions, and grid running costs by lowering customer peak load demands. It also increases the system’s sustainability, security, and stability (Awais et al. 2015).

Numerous studies have been written about the DSM of SG, with the majority of them concentrating on distributed generation with renewable energy integration, optimal load scheduling of demand response (DR), and innovative enabling technologies and systems (Kakran and Chanana 2018; Lu et al. 2018). This paper reviews and examines carefully the DSM methods as well as the effects of distributed renewable energy generation and storage systems on SG. These strategies, seek to lessen peak load demands and uphold a highly developed synchronization between network operators and customers. This paper major contributions is shown below:

• Challenges related to the full implementation of DSM in SG and their accompanying solution.

• DSM policy, techniques, and their applications to lessen peak demands and price of electricity.

• Recent trends of optimization techniques in the DSM.

The paper’s remaining section is shown as follows: The methodology used for this systematic DSM process and the existing work from the literature are also covered in depth in section “Methodology”. In section “The demand side energy management policies”, the DSM policy and related work done on these policies are examined. Section “Demand side management techniques” reviewed the DSM techniques extensively. The challenges related to the full implementation of DSM in SG are carefully examined in section “Challenges of DSM”. The future study is highlighted in chapter “Future work” with the concluding part shown in chapter “Conclusion”.

Methodology

PRISMA stands for Preferred Reporting Items for Systematic Reviews and Meta-Analyses. It is an evidence-based minimum set of guidelines meant to help scientific writers publish different kinds of systematic reviews and meta-analyses. PRISMA focuses on the methods through which authors may ensure accurate and comprehensive reporting of this type of research (Cortese et al. 2022). The PRISMA standard superseded the previous QUOROM standard by demonstrating the high review’s quality, allowing review process replication, and allowing readers to assess the review’s benefits and drawbacks. It offers the replication of a systematic literature review that will completely examine all papers published on the issue to identify the answers to a clearly defined research question. To do this, it will choose the reports to be included in the review using a range of inclusion and exclusion criteria, and it will then summarize the findings (Sarkis-Onofre et al. 2021).

Any research project’s main emphasis is centered on three crucial elements: the purpose, the research technique, and the output with potential future application. The planning, executing stage, and reporting are the three stages of the evaluation stage that are used. What are potential solutions to the problems encountered when implementing DSM in the smart grid? was one of the research questions that were developed in the initial step of planning the literature study. Which optimization method has recently become popular in DSM? How do DSM’s policies and methods affect peak demands and power costs in their use? The goal of the present research is to address these issues using the examined literature.

The second step of a systematic review, known as the “executing stage,” comprises the inclusive and exclusive criteria. Inclusive criteria give a full and in-depth assessment of current research papers, and an academic database is employed for this study, which comprises IEEE Explore, MDPI, ACM Digital Library, Springer, Science Direct, Google Scholar, and Taylor and Francis. These databases include reputable, excellent peer-reviewed materials including journal articles, conference papers, and review articles. To incorporate relevant terms in a single search, boolean operators are utilized. For instance, keywords and synonyms are combined using Boolean operators like “AND” and “OR.”. Hence, any article matching the keywords “Demand Side” Management,” “Demand Response,” “Load categorization,” “Optimization methods,” “Customer classification,” and “Distributed Energy Sources integration.” will show up in the search results. An organized approach based on PRISMA is used to cover the published material from the last 10 years. Which provides a guideline with features in the form of a checklist to improve openness and clarity in reviews (Page and Moher 2017) as shown in Fig. 2. Based on keyword searches of published articles during the last 10 years, we found 95,736 review papers in the chosen database that were all authored in English.

Fig. 2

Overview of an articles search strategy

The Selection procedure was carried out based on the paper’s title, abstract, and English-written content. The publication should be published in an English journal or conference paper, feature a prominent DSM name, and make a significant contribution to the DSM’s practical application. Articles are not excluded based on their citation records, as is the case with traditional reviewing processes, and publications found in a general database like Google Scholar were tracked down to the relevant publishing journal and counted there rather than under Google Scholar to avoid duplicate entries. Parents or unpublished manuscripts are also excluded.

The final collection of papers is summarized, stored in Microsoft Word and Excel files, and then utilized in the R-Classify online tools, which help readers find the manuscript’s most important idea. In this last phase, the results are described together with any possible limits and prospective future study areas. The findings of earlier research on energy management systems are summarized in Table 2. The total number of works considered and cited in the final analysis is 255. Of the 255 articles, 24 are peer-reviewed papers while the others are technical papers. The following details were obtained from each article included in this study: The DSM, demand response techniques, implementation challenges, customer-driven adoption, methodology, approaches, and upcoming optimization work. Table 3 indicated the relationship between the existing and current studies.

Table 2 An overview of existing work on energy management system

Table 3 Summary of comparison between the current and existing works

Table 3 shows that most review works focused on DSM policy, DSM techniques, and optimization techniques, with little or no consideration for the remaining work. As a result, this paper thoroughly analyzes optimization techniques while also providing future directions to bridge these existing gaps.

Demand side management (DSM) is the concept of allowing users to monitor their energy consumption while taking peak energy demand into account. This continuous monitoring and management of energy consumption aim to improve system reliability while lowering energy costs. Many studies have been conducted on demand side energy management due to its enormous complexity (Li et al. 2018). The following is a discussion of the principles, techniques, issues, optimization techniques, and future developments used in literature.

The demand side energy management policies

Energy Efficiency (EE), Demand Response (DR), and Distributed Energy Resources (DER) are three categories into which the strategies used to manage energy on the demand side are divided (Sharifi et al. 2017; Wu and Xia 2017).

Energy efficiency

Energy efficiency provides energy consumers with a comparable and superior service to lower the quantity of energy needed in an economically effective manner since these methods eliminate excessive power loss in the power network (Bukoski et al. 2016). Among the energy-efficient tactics are shown by (Jabir et al. 2018).

• Using energy-efficient equipment and buildings, as well as promoting consumers’ energy-conscious behavior, to reduce energy usage. Typical instances are switching to energy-saving lights from incandescent bulbs and switching to variable-speed air conditioning from standard air conditioning.

• Enhancing and performing routine maintenance on electrical equipment by recovering heat from waste, improving maintenance techniques, using contemporary equipment with optimum designs, and implementing cogeneration.

• Increasing the efficiency of power transmission and distribution networks by utilizing distributed generation, advanced control systems for voltage regulation, three-phase balancing, power factor correction, data acquisition and analysis in supervisory control and data acquisition systems, and modern technologies such as low-loss transformers, gas installation substations, smart meters, fiber-optics for data acquisition, and high transmission voltages.

Demand response

Customers’ energy expenses are reduced through demand response, an optional alteration to the load pattern in response to a change in the electricity tariff (Aghaei and Alizadeh 2013). However, it may create inconvenience during appliance waiting periods. Price-based and incentive-based DR policies are the two categories. The split and subdivision of the incentive-based DR are shown in Fig. 3. The emergency demand response (EDR) program, which pays users for voluntarily decreasing power during crises, and the direct load control (DLC) program, which enables the utility to remotely regulate customers’ appliances to fulfill demand, are both components of the voluntary program. It should be emphasized that under the voluntary initiative, consumers who decide not to participate in energy adjustment will not suffer sanctions (Chen et al. 2014; Imani et al. 2018).

Fig. 3

Incentive based Demand Response (Aalami et al. 2019)

Energy consumers who violate utility company rules under the mandatory program, which consists of the Interruptible Curtailable Service (ICS) and the Capacity Market Program (CMP), are fined (ICS). Another scenario is where the utilities set a predetermined load reduction that the capacity market participants must strictly adhere to maintain a balance between supply, demand, and system dependability. Interruptible/curtailabe uses the emergency response paradigm to stabilize the system, but this paradigm is different from the latter in that users are still required to participate despite the inconvenience involved (Aalami et al. 2010; Conteh et al. 2019).

The last component of the incentive basis for DR is the market clearing scheme, in which users that participate are compensated with load reduction profits. When attempting to balance energy output and consumption in a market clearing program, procedures like demand bidding/buyback (DBB) and auxiliary service market service (ASM) programs are utilized (Aalami and Khatibzadeh 2016). Large energy users, such as industrial and commercial customers, favored this strategy because it gave them a way to bargain for the cost of energy for the load they would be prepared to reduce during a system outage. A negotiated quantity of load reduction with the related rates serves as the electric grid’s reserve energy in an ancillary service market program (Elma and Selamoğullari 2017; Yan et al. 2018).

Price-based DR is used to persuade energy users to participate in different electricity pricing signals with the aim of lowering energy usage. The primary goals of these regulations are to reduce energy prices and shift demand away from peak times. Several signs related to power price are shown in Fig. 4.

Fig. 4

Price based Demand Response (Shewale et al. 2020)

The cost of producing energy at a certain time of day depending on consumer demand is reflected in the time of use (TOU). The price signal of TOU, which is broken down into on-peak, mid-peak, and off-peak times, is determined by demand and cost. It has the excellent benefit of being simple for customers to follow, comprehend, and arrange for their schedule demands. Countries including China (Zeng et al. 2008), Ontario (Adepetu et al. 2013), Italy (Torriti 2012), USA (Faruqui and Sergici 2010) and Malaysia (Hussin et al. 2014) have implemented TOU after it was recommended in (Moon and Lee 2016; Vivekananthan et al. 2014) to minimize costs and energy consumption patterns in residential structures.

Critical peak pricing (CPP) is a price control signal that uses higher power charges to restrict energy usage at a peak time. It offers two time frames (the peak and off-peak). Customers were advised that CPP is granted on days that are predicted to have higher energy use in advance of this period. Since the system is not constantly subject to this constraint, CPP is not a daily DP, but it is also ineffectual at reducing energy costs and carbon emissions. Customers of energy have been urged to participate in DR via CPP, and significant energy and cost reductions have been noted (Kim et al. 2015; Yang et al. 2016). Most especially in countries like North America (Faruqui and Sergici 2010) and Sweden (Renner et al. 2011).

The real-time pricing (RTP) scheme is subject to frequent changes due to the utility price signal, which is made available to consumers an hour or day in advance. It is difficult for the consumers to actively participate in it due to its high level of intricacy and the fact that there are two lines of communication between the parties. This pricing strategy is recommended by (Yoon et al. 2014a, b) as a way to increase system stability at a reduced cost and with favorable environmental impacts in a country like the USA (Yoon et al. 2014a, b).

When Inclined Block Rate (IBR) is paired with RTP or TOU, both price signals may be utilized. Customers’ energy use and electricity prices are connected, thus if energy consumption falls below a certain point, so will the price. The RTP and TOU pricing scheme works well in terms of energy cost and stability when the IBR is utilized to boost its efficacy (Zhao et al. 2013).

A fixed price is a form of pricing indication that is consistent throughout the day or season and is not negotiable. Fixed power pricing in a nation like Nigeria makes it almost difficult to actively engage in any suggested fixed tariff to reduce the cost of energy (Faria et al. 2013; Pan et al. 2014).

Distributed renewable energy

An integrated decentralized power generating system that is connected to the electrical grid is known as a distributed energy resource (DER). With the increasing integration of DER into the grid, a variety of benefits and opportunities, including affordability, reliability, efficiency, power quality, and energy independence for the power system and its stakeholders emerge. The classification of DER into Distribution Generation (DG) and Electric Energy Storage is shown in Fig. 5. The DER is powered by convection and renewable energy sources (RES). Conventional energy sources including diesel, gas, microturbines, and combustion turbines still make up the bulk of the energy market despite their limited availability. These sources, nevertheless, are constrained by high production costs, transmission loss, anthropogenic climate change, the greenhouse effect, and acid rain (Bongomin and Nziu 2022).

Fig. 5

Classification of distributed energy resources (Oskouei et al. 2022)

Despite being stochastic in nature, intermittent, unexpected, and uncontrolled, renewable energy sources (RES) including solar, biomass, wind, solar thermal, geothermal, and small hydro turbines have grown to be a popular source of energy (Platt et al. 2014). According to their storage concept, electrical energy may be transformed into mechanical, electrochemical, electromagnetic, thermodynamic, and chemical energy. The present energy storage methods, prices, guiding principles, benefits, and kinds of ESS applications can be found in Oskouei et al. (2022).

Demand side management techniques

As illustrated in Fig. 6, Demand Side Management (DSM) techniques for load shaping include peak clipping, valley filling, load shifting, strategy conservation, strategic load growth, and variable load shape (Macedo et al. 2015).

• Peak clipping is a concept used in poor countries to decrease the effect of peak demand during peak hours when the installation costs of additional power units are prohibitive. This strategy simultaneously reduced demand and the peak time by directly reducing user appliance loads (Al-enezi 2010).

• Load shifting involves changing the demand for loads from peak hours to off-peak hours by applying filling and clipping strategies. The TOU and storage devices are used in this method with a constant level of total energy consumption (Chokpanyasuwan et al. 2015).

• To preserve system balance, valley filling requires a structure during off-peak times, especially when the average cost is lower than the load cost. This often occurs when a plant’s energy production is not fully used and its running expenses are minimal. Even if the peak demand is unaltered, this leads to an increase in total energy usage. By using thermal storage to apply this technology, system efficiency is greatly raised at a reduced energy cost.

• Strategic conservation reduces energy loss and consumption efficiency of seasonal energy consumption through technological change incentives. This technique is quite comprehensive and less considered as a technique in load management because it involves a reduction in sales that is not necessarily accompanied by peak reduction.

• Strategic load growth increases peak demand in a particular season by managing the seasonal energy usage and a drastic rise in both effect of the energy usage and peak demand is recorded. However, the utilities make use of a more intelligent system to meet their target, especially in the electrification of industrial and commercial heating processes.

• Flexible load shape uses load limiting devices to reduce energy consumption at the user’s end without affecting the actual system conditions, the utility interrupts the loads when necessary to reduce the peak demand and change the total energy consumption.

Fig. 6

Demand side management techniques (Macedo et al. 2015)

This paper reports some of the work on demand side energy management strategies and takes into account the three main categories of energy consumers, namely residential (R), commercial (C), and industrial (I) energy users. As indicated in Table 4, certain authors in some of the examined works took into account all (A) energy users at once.

Table 4 Summary of work done on demand side energy management policies

Challenges of DSM

Planning and managing decision parameters and operating constraints are necessary for the implementation of DSM and depend on several important factors, including the load profile of an appliance, the integration of renewable energy, load categorization, constraints, dynamic pricing, consumer categorization, optimization techniques, consumer behaviors, issues with electricity data, adequate knowledge, a reliable framework, technology-smart, and grid-intelligent appliances, appropriate control strategies, and these challenges encountered during the DSM’s deployment are briefly mentioned below:

Load profile of appliance

Smart appliances are an essential part of creating an accurate and efficient load management system since they come with built-in communication sensors that can link with the smart meter to analyze their energy usage. This is accomplished by collecting ambient data and operating in accordance with the power and tariff parameters provided to them. To create a more precise and trustworthy system, the energy profiles of smart appliances must be taken into consideration during the deployment phase. A normal survey load profile may take the role of smart meters, although it is less accurate. If you are aware of every piece of equipment your clients use, setting up a DR program is easy. To assess load profile management, a survey of various energy consumers is conducted, with an emphasis on quality of service (QoS) (Pilloni et al. 2016). Similar in approach, the authors (Vivekananthan et al. 2014) urge users to discuss their preferences for using controlled appliances and place greater emphasis on scheduling appliances according to time and preferences. According to a study published in (Yilmaz et al. 2019), the variables used to construct the experimental load profiles for 60 residential structures were consumer availability, occupant population, and age. The deployment of smart meters with specific devices, as well as the methodology for monitoring and analysis, are presented in Issi and Kaplan (2018), Teng and Yamazaki (2018). The writers in Yilmaz et al. (2020) investigate the major appliances that are responsible for this high energy consumption at the designated time of day to lower peak demand to 38% by implementing energy-efficient equipment. The stochastic ambient environment and user behavior, according to the currently available literature, make it challenging to develop a generalized load profile optimization algorithm that can accurately predict the energy consumption of various electrical appliances for various consumers.

In conclusion, compared to the usage of smart appliances and smart meters, load profiling assessment techniques like surveys, questionnaires, bottom-up, and top-down approaches are less technically complex, accurate, and time-consuming. However, performing this assessment comes at a far higher cost. By using the data produced by these smart devices, stakeholders may have a better knowledge of how they consume electricity. This is a crucial tactic to raise the power grids’ dependability and effectiveness.

Renewable energy integration

Since the use of renewable energy sources (RES) in the current power system seems to have a bright future, it is one of the factors considered while using DSM. Integration is very difficult, although encouraging, it may sometimes be irregular and intermittent (Elma et al. 2017). But in order to deal with the problems of power instability, power quality, and reliability brought on by RES’s intermittent nature, battery energy storage systems (BESS) are especially helpful (Elma et al. 2017). To address these difficulties, four battery consumption management techniques using centralized, decentralized, and distributed control structures have been investigated (Worthmann et al. 2015). The authors in (Yao et al. 2015) suggested an autonomous energy scheduling strategy to solve the problem of voltage escalation in HEMS. The DSM has recommended the optimal charging methods for plug-in electric cars (PHEV) and BESS to reduce the peak load demand (Mou et al. 2014). To assess how well the system uses its batteries, two metrics of battery efficiency factor and utilization factor have been created. It has been shown that system operating costs may decrease as battery efficiency increases (Nguyen et al. 2014). Since RES is rapidly evolving into one of the fundamental elements of DSM, it is imperative to develop cutting-edge optimization solutions for efficient load scheduling with the lowest cost while maintaining customer satisfaction.

By reducing system strain, which lowers the likelihood of power outages, diversifying the generation mix, and possibly improving power quality, it can be deduced from the literature that the integration of renewable energy can increase power network reliability. Moreover, it may help countries with climate change mitigation, energy cost reduction, and improving resistance to price volatility. Decentralized energy production, less environmental impact, and improved energy security are advantages of RES in DSM (Dincer and Bicer 2020). Yet, because the efficiency is lower than that of the conventional energy grid, synchronizing energy production and consumption is a significant issue for the energy sector. Nonetheless, the development of batteries has positively impacted the aforementioned constraint. The cost of production and the quantity of space needed for the use of this various energy are further barriers to the full integration of RES (Basit et al. 2020).

Load categorization

Electrical appliance classification is vital for efficient load management. These electrical loads may be categorized according to three standards:

• Based on the appliances’ time of operation (Puente et al. 2020).

• Based on power rating of appliances (Kim and Lee 2019).

• Based on appliances’ total energy consumption (Ibrahim et al. 2023).

Deferrable and nondeferrable operated appliances make up the first standard’s loads, adjustable and nonadjustable operated appliances make up the second standard’s load, and basic and heavy operated appliances make up the third standard’s loads. It is important to note that there is presently no approved worldwide classification system for home appliances (Leitao et al. 2020). It should be noted that despite writers using the categorization suggested in Beaudin and Zareipour (2015), there is still no agreement on the appliances that belong to each group.

The literature classifies various smart home appliances based on user comfort and classification clarity. For scheduling home appliances, authors in the literature have used their own classification. Faisal et al. classified fifteen appliances as interruptible, non-interruptible, or base appliances. Among the interruptible appliances are the vacuum cleaner, sensors, PHEV, dishwasher, stove, microwave, and other intermittent loads. The clothes washer and spin dryer are non-interruptible appliances, while the oven, TV, PC, laptop, radio, and coffee maker are basic appliances (Faisal et al. 2019).

Shuja et al. classified fifteen appliances as shiftable, non-shiftable, or fixed. Water pumps, water heaters, vacuum cleaners, dishwashers, steam irons, air conditioners, and refrigerators are all shiftable appliances. Washing machines and tumble dryers are non-shiftable appliances, while TV, oven, desktops PC, blender, laptops, and ceiling fans are fixed appliances (Shuja et al. 2019). Thirteen smart home appliances were utilized (Rahim et al. 2016b), including eight shiftable and five non-shiftable items. Shiftable appliances include an air conditioner, clothes dryer, washing machine, dishwasher, refrigerator, coffee maker, water heater, and space heater, whereas non-shiftable appliances include a fan, lamp, iron, toaster, and microwave oven. Abbasi et al. utilized eleven items divided into three categories: fixed appliances, shiftable appliances, and interruptible appliances. Fixed appliances include a lamp, oven, blender, and coffee maker. Shiftable appliances include the clothes dryer, washing machine, and dishwasher, whereas interruptible appliances include the water heater, iron, vacuum cleaner, and space heater (Abbasi et al. 2019). Eight shiftable appliances (dishwasher, refrigerator, air conditioner, clothes dryer, water heater, coffee maker, space heater, dishwasher) and six non-shiftable appliances (fan, light, blender, clothes iron, oven, and vacuum cleaner) were utilized (Rahim et al. 2018).

Deferrable and nondeferrable operated appliances

The time of operation of a deferrable appliance can be stopped, and restarted at other time slots. This is simply subdivided into interruptible and non-interruptible operated appliances (Abideen et al. 2017; Li et al. 2017).

• Interruptible operated appliances may be stopped, interrupted, and resumed for a brief time without affecting the quality of the energy services provided, provided that it is completed before the deadline. Air conditioners, electric heaters, cold appliances, and hybrid electric automobiles are a few examples of interruptible operated equipment (PHEV). These appliances are also referred to as adjustable, shiftable, thermostatically controlled, and limitable operated equipment. These loads may be scheduled using a demand response system. Depending on the cost of the power or a financial incentive, they might be shifted from peak to off-peak hours, which will reduce the demand for peak load.

• Non-interruptible operated appliances must finish their scheduled operation within a certain time frame. Non-interruptible appliances, also known as regular, fixed, non-adjustable, and non-controllable operated appliances, include lighting and kitchen systems. These loads are unsuitable for DR programs since they do not permit a time shift or interruption.

Adjustable and nonadjustable operated appliances

Most thermal loads are examples of adjustable operated appliances since they may be set to a lower level. These kinds of loads may actively take part in DR programs by reducing their total energy usage in line with energy pricing and financial incentives. However, it’s crucial to be informed that the DR software employed for these sorts of devices might make you uncomfortable while you wait. The overall consumption for non-adjustable loads is fixed (e.g., TVs and computers). An algorithm for demand response cannot plan for non-deferrable or non-adjustable loads (Li et al. 2017).

Basic and heavy operated appliances

An electrical appliance’s rating decides which categories it will fall under. Appliances with simple operating systems are those that use less energy. Lighting systems, televisions, laptops, and other basic operated appliances are just a few examples, and they hardly ever take part in DR programs. In contrast, appliances that require a lot of power consumption are more likely to be included in DR programs. The heavily operated appliances include things like air conditioners, electric cookers, and washing machines. The control of various appliances, particularly thermostatically controlled loads like air conditioning systems and electric water heaters, has already been the subject of several studies created by various authors (Du and Lu 2011; Goh and Apt 2004; Ibrahim et al. 2023; Ilic et al. 2002; Pedrasa et al. 2010).

Constraint

The scheduling optimization problem involves many constraints. These restrictions apply to the system level as well as the appliance level. The restrictions listed below are addressed:

• Electrical demand supply balance (Tasdighi et al. 2013):

The balance between the need for and supply of electricity at any given hour is shown in the equation below, which also accounts for power from batteries and the grid, load shifting, and both shiftable and non-shiftable load demands. Without considering load shifting

$$P_{grid} (t) - P_{bat} (t) = D_{e} (t)$$

(1)

Considering Load Shifting

$$P_{grid} (t) - P_{bat} (t) = D_{nsh} (t) + \sum\limits_{Nsh}^{n = 1} {D_{sh}^{n} }$$

(2)

• Temperature constraints (Tasdighi et al. 2013):

In this case, it is necessary to schedule thermostatically controllable loads (TCLs) with the understanding that the water and room temperatures must be maintained within a certain range.

$$T^{\min } \le T \le T^{\max }$$

(3)

The water temperature at the outlet is given as:

$$T_{outlet}^{\min } \le T_{outlet}^{i} \le T_{outlet}^{\max }$$

(4)

The HVAC room temperature is given as:

$$T_{room}^{\min } \le T_{room}^{i} \le T_{room}^{\max }$$

(5)

• Battery constraints (Huang et al. 2016):

The manufacturer’s recommended range for battery level maintenance should be followed. As a result, the following constraints are put in place

$$SoC_{\min } (t) \le SoC(t) \le SoC_{\max } (t)$$

(6)

$$SoC(t) = \frac{{E_{bat}^{t} }}{{E_{bat}^{cap} }}$$

(7)

Battery maximum charging and discharging power limit can be represented as:

$$0 \le \frac{{P_{bat}^{ch} (t)}}{{\eta_{ch} }} \le P_{\max }^{ch}$$

(8)

$$0 \le P_{bat}^{dch} (t).\eta_{ch} \le P_{\max }^{dch}$$

(9)

• Charge and discharge rate constraints for Electric vehicles (Zhao et al. 2012)

Electric vehicles (EVs) are supposed to be charged and discharged at residential locations in this scenario. When parked at homes, EVs are typically wired into the residential metering systems.

During the charge cycle:

$$0 \le P_{ch} (t) \le P_{\max } (t)$$

(10)

During the discharge cycle:

$$0 \le P_{dch} (t) \le P_{\max } (t)$$

(11)

• Grid constraints (Wong 1991):

Each time slot’s energy import from the grid must be upper bound by a predetermined limit to avoid overloading the utility.

$$0 \le P_{grid} (t) \le P_{grid}^{\max } (t)$$

(12)

• User comfort-enabling constraints (Tamilarasu et al. 2021):

The wants and satisfaction of the users are given precedence in various circumstances. Certain limitations must be met to guarantee that the optimization process moves forward without significantly sacrificing comfort

$$d_{r} = \sum\limits_{i = 1}^{24} {} \sum\limits_{r = 1}^{n} {S_{r} (i)}$$

(13)

Total daily load requirement:

$$\sum\limits_{i = 1}^{24} {} \sum\limits_{r = 1}^{n} {D_{1} (i)_{r} = } \sum\limits_{i = 1}^{24} {} \sum\limits_{r = 1}^{n} {D_{2} (i)_{r} }$$

(14)

Instantaneous power demand:

$$PD_{i} \le PD_{\max } \forall i \in \left[ {1,24} \right]$$

(15)

Idle constraint:

$$S_{r} \left( i \right)\forall i < st,i > et\;{\text{and}}\;i \in \left[ {1,24} \right]r \in \left[ {1,n} \right]$$

(16)

• Phase wise energy requirement of appliances (Sou et al. 2011):

Since controllable appliances such as washing machines, and dishwashers have different power requirements at each operation cycle. This limitation guarantees each appliance’s operational cycle gets adequate energy for its functioning

$$\sum\limits_{k = 1}^{m} {P_{ij}^{k} = E_{ij,} \forall } i,j$$

(17)

• Power safety (Sou et al. 2011):

This constraint places a maximum on the total energy allotted during any period, requiring that it always be less than the maximum energy from the grid.

$$\sum\limits_{i = 1}^{N} {} \sum\limits_{j = 1}^{m} {P_{ij}^{t} \le P_{grid}^{\max } (t),\forall i,j}$$

(18)

• Prioritization of appliance constraints (El-Metwally et al. 2006):

In this instance, the DSM optimization places a focus on the appliance priority. A priority index (PI), which is inversely proportional to the appliance’s load factor and proportionate to the peak demand of the appliance, is used to classify the loads

$$PI \propto \frac{{P_{\max } }}{loadfactor}$$

(19)

• Up time required to finish a task (Paudyal and Ni 2019; Tasdighi et al. 2013):

When an appliance is switched on, it shouldn’t be shut off until the associated task is finished, for example, a dishwasher

$$W_{n} (t) + W_{n} (t + 1) + .... + W_{n} (t + TOP_{n} - 1) \ge (TOP_{n} - 1)(W_{n} (t - 1) - W_{n} (t - 2)),\forall t \in t_{n}$$

(20)

where $W_{n} (t)$ is the operation state of nth shiftable load at a time (t) 1: on, 0: off and $TOP_{n}$ is the number of nth shiftable load’s time of operation.

• Operation ordering of appliances (Paudyal and Ni 2019; Tasdighi et al. 2013):

The maintenance of the appliance’s operational ordering should be ensured. For instance, it is best to use the dryer after the washing machine has done its work. If shiftable load m is activated after shiftable load in such a scenario:

$$start_{m} \ge start_{n} + operating\_duration_{n} + gap$$

(21)

Dynamic pricing

Another element that exacerbates DSM challenges is dynamic pricing. One of the main goals of the reform of the energy market is to lower peak demand while increasing the use of all resources. Through various incentives provided by the utilities, customers are encouraged to participate in different dynamic pricing schemes. Since dynamic pricing encourages consumers to transfer their load from peak to off-peak periods, the scheduling issue for home energy usage must be addressed in this situation. The key elements influencing the structure of the electricity tariff are marginal cost, load pattern, societal considerations, and the profitability of the power company (Phuangpornpitak and Tia 2013). Numerous pricing strategies have been used, as can be shown in Fig. 6 to balance the supply and demand for energy. To preserve customer happiness and boost the system’s overall cost efficiency, advanced optimization algorithms must be developed to allow efficient energy consumption scheduling in addition to the reduction of dynamic tariffs (Panda et al. 2022).

Customer categorization

A thorough examination of numerous consumer categories may aid in a better understanding and design of DR. The customers are divided into four categories including the residential, commercial, industrial, and transportation sectors. In any of these categories, transportation is not a key problem for DR.

The residential sector is more challenging because of the diverse appliance consumption patterns, consumer dispersion, and individual user preferences. This suggests that rather than treating customers equally, each one is treated differently. Because the load profile and appliance use data are not readily available, DR adoption for industrial clients is quite challenging. Even with access to this data, the activities’ dependency on time makes it difficult to change energy use. Commercial users’ energy profiles may be modified with ease if they are identical. The most commonly used equipment, including air conditioners, heaters, ventilators, and lights, may be managed in line with the established specifications. It is crucial to remember that the DR is simple to deploy in the commercial and industrial sectors, allowing the system to react to DR fast.

Consumer behaviors

Some customers don’t respond well to price changes and it is unclear how people will respond to these programs. Customers have a variety of reactions to the price of electricity, and these reactions can be categorized as extremely flexible and unassuming behavior (Sharifi et al. 2017). Although there are many ways to implement DR and it offers many advantages, if the end user encounters any kind of difficulties, they may become disillusioned and leave the program or demand more money or incentives (Duncan and Hiskens 2011). The motivations behind these difficulties posed by each consumer’s decision to install microgeneration in their home are examined by the authors (Balcombe et al. 2014). They assert that inconveniencing people can prevent them from adopting technology.

The study by (Balcombe et al. 2014) does highlight an important aspect of end-use customers, namely that financial considerations are frequently more important than a desire to contribute to environmental change, even though micro-generation is a distinct but related problem. It is important to emphasize the importance of financial motivations, particularly in light of the high level of uncertainty previously mentioned regarding the potential financial benefits of enrolling in a DR program. The possibility is raised in (Boisvert and Neenan 2003), and raises a related financial concern, that the electricity bill savings from customers may not be sufficient to support equipment investment and make up for the inconvenience of continuously monitoring electricity prices when they may only need to react in exceptional circumstances. Naturally, this will depend on the type of software being used and the required level of customer interaction.

There will be little interest in DR if financial considerations are the primary factors influencing the adoption of DR programs and it is demonstrated that consumers will not be able to save money on their future power bills or recover their initial investment in DR technology. This dissuades people from using DR programs extensively. Despite receiving feedback on their energy use from in-home displays, most study participants continued with their regular routines and habits, according to research published in (Herrando et al. 2014). This is a great example of unanticipated or possibly irrational customer behavior, a challenge that needs to be taken into account when evaluating the DR implementation.

This study also emphasizes the importance of promoting greater DR knowledge and giving consumers the right information about DR programs for them to make informed decisions. As a result, utility companies won’t frequently send the DR resource (Cutter et al. 2012). This is a crucial factor to take into account when estimating the resource’s worth. It is crucial to take into account when estimating DR resources because it is connected to the traits and physical composition of electrical loads.

The main challenges are recognizing and properly accounting for the DR resource’s limitations as a result of end-user behavior and preferences in DR deployment. Understanding the variables that affect customers’ choices to accept or reject a DR program, as well as how these restrictions are reflected in the assessment study, is essential. Recognizing the potential effects that unanticipated consumer behavior may have on the DR features is essential as it successfully manages it throughout the evaluation process (Nolan and O’Malley 2015). Overall, different lifestyles and household activities have a significant influence on how much energy is used since it is predictable. Both long- and short-term trends are easily predicted. Participants reduce their electricity bills and Non-participating users may also save money since the programs shift power consumption from times when demand is highest to times when energy is least expensive.

Optimization techniques

Numerous optimization strategies have been used to address the problems related to energy management. However, demand-side optimization methods are further divided into deterministic, stochastic, and hybrid approaches as illustrated in Fig. 7.

Fig. 7

Optimization techniques

The goal of this method of optimization is to find a universally optimal solution by using the analytic properties of the problem. It is also important to note that as the problem constraint shrinks, the likelihood of discovering global solutions increases, as well as the assurance of the quality of the optimal solutions attained. Linear programming (LP) (Erol-Kantarci and Mouftah 2011; Zhu et al. 2012), nonlinear programming (NLP) (Althaher et al. 2015), gradient base (GB) (Huang et al. 2015), Lagrangian algorithms (Boyd; Gatsis and Giannakis 2011), Lagrange–Newton (Dong et al. 2012), interior point method (Samadi et al. 2012) and Lyapunov techniques (Guo et al. 2012), and mixed integer nonlinear programming (MINP) (Behrangrad et al. 2010) are few examples of deterministic methods used in energy management to reduce the amount of electricity used.

Zhu et al. (2012) proposed an integer LP system to schedule electrical appliances, together with power sources and operating time, in accordance with user preferences to decrease peak loads. Similarly to this, Wang et al. developed the ideal dispatching model for a smart HEMS with distributed energy resources and smart home appliances using the MINLP methodology (Wang et al. 2012). The cost of electricity and total energy used are both decreased. Due to consumers’ unexpected, impulsive, non-linear, and complex energy usage behaviors, the MINLP was unable to regulate some appliances. Existing work on Deterministic Optimization Techniques is shown in Table 5.

Table 5 Deterministic optimization techniques

Stochastic approach

The stochastic method is an iterative algorithm that makes use of the unpredictable nature to identify the optimal solution from the parent solution. It employs a variety of techniques to the problem in an attempt to identify the best answer conceivable because of the high dimensional nonlinear objectives issue; however, unlike the deterministic method, the optimal solution is not guaranteed. Even though the problem where determinism methods have several local solutions, its singularity makes it a powerful tool in engineering. This approach is broken down into heuristic, meta-heuristic, and artificial intelligence categories in Fig. 7.

Every strategy has advantages and disadvantages that vary depending on the optimization problems. Because of this, there isn’t a perfect answer to every optimization problem. The fundamental weaknesses and advantages of each random method examined in this work are summarized in Table 6. A fuzzy inference system (FIS) is recommended by Hasaranga et al. (2017) for the management of an energy storage system that utilizes renewable energy sources and a storage unit. Comparison with a rule-based control method demonstrated the recommended system’s efficiency in lowering fluctuation and prolonging the lifetime of energy storage devices (ESS).

Table 6 Selected meta-heuristic optimization

Ambreen et al. published a heuristic technique for cost, PAR, and the load reduction in the smart grid in 2017. The recommended algorithms provide the appliances in a home with the best schedule possible, Cost savings, reduced PAR, and user comfort are all obtained when appliances are designed. Costs are cut by 52% using GA scheduling, while PAR is cut by 23% (Ambreen et al. 2017). Hsu et al. developed a DPbased optimization strategy to reduce the system’s energy-producing costs for the DLC dispatch. As a consequence, the dispatch DLC approaches and the unit commitment issue were integrated, and a DP strategy was developed to address both issues (Hsu and Su 1991).

A model predictive control strategy based on weather forecasts is offered to reduce the amount of energy required and improve the utilization of renewable energy sources for energy management in residential microgrids. The established MPC control approach is based on a constrained optimal control problem for a certain time horizon. The proposed approach was contrasted with conventional rule-based control logic. Primary fossil energy usage has dropped by 14.5% on average while home comfort levels have increased (Bruni et al. 2015).

Noor et al. proposed a GTA technique for a demand-side management model that includes storage components in distinct research. In addition to reducing the peak to average ratio for the benefit of the electric grid, the suggested model can smooth out dips in the demand profile caused by supply restrictions. This was decided by every player who took part, their strategies, and the awards they received. Customers are the participants in this strategy, and the reward is determined by the lowest cost (Noor et al. 2018).

For a variety of consumer loads, BFO was used to reduce peak load and energy expenditures by 7% and 10%, respectively. This method outperforms earlier evolutionary algorithms for controlling controlled devices (Priya Esther et al. 2016). Similarly to this, Bharathi et al. recommend combining GA with an appropriate load shifting technique to reduce and reconfigure the load needs of all sorts of energy consumers (Bharathi et al. 2017). Based on TOU and IBR, Rahim et al. employed ACO to decrease energy usage at the residential load. The recommended approach may dramatically lower peak load, PAR, and energy expenditures without affecting customer satisfaction (Rahim et al. 2016a).

Mahmood et al. recommended a HEMC model to control the scheduling of appliances, lowering user comfort, PAR, and electricity costs. However, energy is wasted significantly when appliances are used unnecessarily, and environmental concerns are also disregarded (Mahmood et al. 2016).

Another study advises evaluating a HEMS’s ability to control its energy expenses using GWO and BFO. These proposed techniques resulted in 45% and 55% energy reductions respectively (Barolli et al. 2020). Furthermore, (Elmouatamid et al. 2020) evaluated the performance of a HEMS by using three meta-heuristic optimization techniques and the HS, BFO, and EDE algorithms. Existing work on Stochastic Optimization Techniques is shown in Table 7.

Table 7 Stochastic optimization techniques

Another sub-category of stochastic optimization techniques worth discussing due to its constantly evolving field is machine language. Machine learning (ML) is an evolving branch of computational algorithms that are designed to emulate human intelligence by learning from the surrounding environment. They are considered the working horse in the new era of the so-called big data, which has been used to address different issues in DSM as shown in Table 8 (Antonopoulos et al. 2020). The main types of machine learning are supervised learning, unsupervised learning, and reinforcement learning as stated (Murphy 2012). Figure 8 shows the subtypes of machine learning used in DR.

Table 8 Application of machine learning in DSM

Fig. 8

Machine language used in DSM (Antonopoulos et al. 2020)

Supervised machine learning (SML) is the task of generating meaning from labeled training data that includes a set of training examples. In supervised learning, each example is a mainstay that contains an input object (typically a vector quantity) and an enforced output value (may also be referred to as a supervisory signal) (Praveena and Jaiganesh 2017). The authors in Giovanelli et al. (2017), Pal and Kumar (2016), Yang et al. (2018) proposed Support Vector Regression (SVR) to forecast the price of energy. This technique is also used for short time load forecasting for non-aggregated loads (Zhou et al. 2016).

Unsupervised machine learning (UML) approaches are very beneficial in description tasks because they try to discover links in a data structure without requiring a quantifiable output. Because there is no response variable to oversee the study, this kind of machine learning is referred to as unsupervised (Gareth et al. 2013). Cao et al. examine the clustering of 4000 households from the Irish CER dataset over 18 months using K-means, SOM, and hierarchical clustering algorithms with various distance calculations based on the 17 most significant PCA components (Cao et al. 2013).

Reinforcement learning (RL) is the task of determining how agents should perform actions in a given environment to maximize cumulative rewards. Q-learning is commonly used at the HEMS level to optimize appliance scheduling by using cost and user comfort as reward functions (O’Neill et al. 2010; Wen et al. 2015). O’Neill et al. consider pre-specified disutility functions for customers’ dissatisfaction with job scheduling (O’Neill et al. 2010), but Wen et al. address this limitation (Wen et al. 2015). A state in this context is made up of a price sequence from the retailer or aggregator, a vector that reflects the user’s consumption of specific appliances over time, and sometimes the priority of the considered device.

Hybrid approach

The hybrid approaches have been used in numerous engineering applications to get beyond the drawbacks of each optimization strategy and enhance their efficacy and accuracy to give a greater performance of the system (Tsipianitis & Tsompanakis). Several of the hybrid approaches used in DSM are briefly described below:

First, the teacher and learning-based optimization (TLBO) and the shuffling frog leap (SFL) methods of optimization are recommended. In this concept, the load is separated into three categories: shiftable, sheddable, and non-sheddable loads. The recommended strategy aimed to bring down the cost of electricity. This research employs ToU, RTP, and CPP as three alternative pricing models. The findings demonstrated that the recommended approach was successful in reducing consumption costs (Derakhshan et al. 2016).

Rahim et al. (2016a, b) investigated the efficacy of binary particle swarm optimization (BPSO), ant colony optimization (ACO), and genetic algorithm (GA). Lowering power prices and the peak-to-average ratio (PAR) while taking into consideration RESs and storage systems is the main objective of the proposed effort (Rahim et al. 2016b).

However, the validation results showed that GAPSO performed better than GA and BPSO in terms of cost and discomfort, lowering peak power use by 7.8532% and 27.7794%, respectively. While GA and BSPO reduced the cost of energy consumption by 24.0470% and 29.9702%, respectively, while GAPSO decreased peak power consumption (PAR) by 36.39%. While needing the least amount of waiting time, GAPSO was able to reduce consumption expenses by up to 25.2923% (Javaid et al. 2017a).

In Küçüker et al. (2017), a hybrid energy management strategy is proposed by using a hierarchical genetic algorithm (HGA) to alter the fuzzy inference system’s rule base. The fuzzy-HGA method seems to be more effective than the conventional fuzzy-GA approach, even with just 47% of the total rules in the rule base. By purchasing a more basic fuzzy logic controller, the entire control system can be implemented in real time on low-cost embedded electronic devices. A fuzzy logic-based EMS is presented in Panwar et al. (2017) to lower the fluctuations and peak powers of a grid-tied microgrid. In a similar line, the study (Pascual et al. 2015) proposes the conventional fuzzy-genetic algorithm approach.

A hybrid power system for residential structures was the subject of an energy management strategy developed by Zenned et al. (2017). When compared to buying electricity from the grid, this plan’s results show a decrease in energy use, however, the modeling fails to take energy costs into account (Zenned et al. 2017).

A nonlinear MPC approach is recommended (Merabet et al. 2016). Using a synthetic NN, the loading trough was estimated. Voltage stability may be maintained by regulating the battery state of charge (SOC) and planning the load. Grid Connected based MPC EMS is used to reduce energy expenses (Arcos-Aviles et al. 2017).

Javaid et al. developed a hybrid genetic wind-driven (HGWD) technique to build a DSM controller for a residential area in an SG. The result shows that the HGWD algorithm performed the best. By lowering the cost of power use by 33% and 10%, respectively, when compared to the WDO algorithm and GA. To get the best results, the HGWD reduced user comfort by 40%, PAR by 17%, and electricity costs by 30% (Javaid et al. 2017b). A hybrid method that combines PSO and Gray wolf optimization (GWO) is suggested using day-ahead scheduling (Hussain et al. 2016).

The hybrid GA/PSO method (HGPSO) was introduced by Ahmad et al. who also showed that it outperformed the GA, BPSO, BFO, and WDO algorithms. For the GA, BPSO, BFO, and WDO algorithms, the percentage of power bill decrease was 9.80%, 19.50%, 15.40%, and 15.80%, respectively. Each algorithm’s percentage of PAR reduction was 14.09%, 3.30%, 22.10%, and 33.54%. The PAR and the electric cost were reduced by 25.12% and 24.88% respectively by the HGPSO (Ahmad et al. 2017). In another investigation, the GA was put up against a more advanced PSO algorithm (IPSO). The peak load was reduced with the IPSO by about 30.26% while it was reduced with the GA by 25.78% (Yang et al. 2015).

The simulation results show how efficiently the proposed algorithm GHSA minimizes user discomfort while decreasing PAR and power costs. The GHSA reduces the peak load at 3.73 kWh in contrast to the present heuristic methods (13.84 kWh). According to the findings, smart home (SH) expenses have been decreased by WDO, HSA, GA, and GHSA to 2.61, 1.72, 1.12, and 1.34 cents/h, respectively (Javaid et al. 2017b).

Manzoor et al. introduced the teacher learning genetic optimization (TLGO) method and compared it to the teacher learning-based optimization (TLBO) and GA for residential load scheduling with a day-ahead pricing scheme. Cost reductions of 31%, 31.5%, and 33% were produced by the GA, TLBO, and TLGO, respectively. User discomfort was lowest with TLGO when compared to GA and TLBO. User discomfort with the GA, TLBO, and TLGO had corresponding values of 2.37, 2.14, and 1.83 (Manzoor et al. 2017).

The hybrid algorithm known as the bat-crow search algorithm (BCSA) was developed by Javaid et al. by combining a meta-heuristic bat algorithm (BA) and a crow search algorithm (CSA). Using the critical peak pricing (CPP) system for HEMS, they compared the outcomes of BCSA with BA and CSA in terms of the amount of power cost reduction. According to the findings of optimization, the BCSA algorithm lowered power expenses by 31.19%, while the BA and CSA cut costs by 28.32% and 26.70%, respectively. The description above suggests that hybrid algorithms perform better than single algorithms because they are more adaptable and effective (Javaid et al. 2018). Existing work on Hybrid Optimization Techniques is shown in Table 9.

Table 9 Hybrid optimization techniques

Future work

The majority of the review focused on thermal comfort and appliance waiting time to address customer satisfaction. The user’s experience at a DR event, their social comfort, and other social variables should be taken into consideration as they can boost user satisfaction. It’s crucial to model EVs as both a load and a generator to make the most out of the system. Peer-to-peer trade between prosumers may result in flexible assets with lower costs. Most of the work that was examined represented EVs as interruptible or storage systems.

Fairness between users, standardization, and SG interoperability must be guaranteed while developing a DSM program. For the real-time synchronization and integration of security, safety, smart appliances, and monitoring, extensive research is needed to secure the security and privacy of customers’ data. In addition to this, the agencies, shareholders, and policymakers need to step up and enact new rules and policies to increase the trust of the public. A thorough evaluation of the technical, economic, and environmental performance of current and upcoming DSM systems is required. This is needed to compare DSM and conventional treatments fairly.

The convergence and computation times of DSM optimization problems are improved by the hybrid algorithms-based optimization models. However, while choosing an algorithm to solve DSM optimization issues, other factors such as problem types (such as single- or multi-objective), optimization types (such as local or global), robustness, and accuracy should be taken into account.

As DSM, as previously said, enables both system operation and system development, it offers versatile advantages and value. However, the business case for DSM has not been well established since there are no tools for weighing costs and advantages. There is still a lot of work to be done in this area.

The primary system operating variables will often determine the DSM value’s size (i.e., the value of demand controllability). The system stress, or how close the system is to being loaded to its full capacity and hence needing reinforcement, should be taken into account in this situation. Even though it is often low in systems with significant spare capacity, the value of DSM will be high in system components that need reinforcement.

Conclusion

This paper provides a comprehensive analysis of the different technologies, approaches used in DSM as well as the impact of distributed renewable energy generation and storage technologies in SG. The main goal of these methods is to decrease peak load demands and achieve advanced synchronization between network operators and customers via the development and application of power-saving technologies, financial incentives, the price of energy, and government rules. This research thoroughly investigated DSM implementation issues that must be overcome for DSM to be effectively integrated into the SG with some proposed solutions, DSM optimization methodologies, and their related solutions, which were not included in the earlier review article. As a consequence, a comprehensive comparison of many algorithms used in DSM optimization problems is provided in terms of a variety of factors such as energy cost reduction, PAR, waiting time, power scheduling, Voltage limitations, DR, risk management, client privacy, and carbon emission. We determined, after examining multiple DSM-based research, that a single strategy is not the best solution to handle the high complexity of the DSM optimization problem due to its poor performance and low convergence rate. As a consequence, hybrid algorithms may outperform single algorithms in terms of convergence rate, complexity, noisy environment, imprecision, uncertainty, and ambiguity. Furthermore, these tactics may be improved in the future to improve SG’s efficiency by balancing supply and demand. Even though these current breakthroughs in the use of optimization techniques in DSM are widely known, extra research is undoubtedly necessary to discover the optimal solutions in many real-world scenarios.

The power system’s functioning will become more difficult if corrective control is used. This is just another obstacle to the adoption of DSM. Yet, given that adaptability is increasingly seen as a key tool for coping with the unpredictability of future developments, together with the ongoing cost reductions of DSM technologies, it is anticipated that DSM will become noticeably more competitive in the near future. Increasing trust in the employment of DSM schemes for the provision of system security will benefit from the establishment of targeted trial schemes. This comprehensive review of DSM will assist all researchers in this field in improving energy management strategies and reducing the effects of system uncertainties, variances, and restrictions.