Power data analysis and mining technology in smart grid

Abstract

This study proposes a smart grid model named “GridOptiPredict”, which aims to achieve efficient analysis and processing of power system data through deep fusion of deep learning and graph neural network, so as to improve the intelligent level and overall efficiency of power grid operation. The model integrates three core functions of load forecasting, power grid state sensing and resource optimization into one, forming a closely connected and complementary framework. Through carefully designed experimental scheme, the practical value and effectiveness of “Grid OptiPredict” model are fully verified from three aspects: accuracy of load forecasting, sensitivity of power grid state sensing and efficiency of resource allocation strategy. Experimental results show that the model has significant advantages in prediction accuracy, model stability and robustness, resource optimization, security, information security, social and economic benefits and user experience.

Introduction

In the context of today’s global energy transition, smart grids, as a core component of the energy Internet, are gradually becoming a key platform for connecting various energy production, transmission, distribution and consumption. The development of smart grid is not only a simple upgrade of traditional power grid, but through the integration of advanced information and communication technology, sensing technology, automation technology and data analysis technology, intelligent management and control of power grid, so as to improve energy efficiency, ensure safe and reliable power supply, and promote green and low-carbon development. With the large-scale access of renewable energy, the rise of distributed power generation and micro-grid, the operating environment of power grid is increasingly complex, and higher requirements are put forward for the flexibility, response speed and adaptive ability of power system (Li et al. 2021). Therefore, how to effectively manage and utilize massive power data and mine hidden rules behind data has become a key problem to be solved in the field of smart grid.The development of smart grid greatly promotes the generation and accumulation of power system data. These data cover the whole process from power generation, transmission, distribution to electricity consumption, including but not limited to equipment status monitoring, power quality monitoring, user behavior records, etc. (Huang et al. 2022). Therefore, the development and application of advanced data analysis techniques, especially those that can deeply mine the intrinsic correlation of data and predict future trends, is crucial for the further development of smart grids (Wang et al. 2020a, b).

In recent years, with the rapid development of artificial intelligence technology, the research in the field of power system has also ushered in innovation, especially for the exploration of power load forecasting and resource optimization allocation, which has made remarkable progress. Despite the remarkable results achieved by the above-mentioned studies in their respective fields, there are still a number of issues and challenges that have not been adequately addressed (Zheng et al. 2020; Li et al. 2022a, b). First, most existing models focus on single task optimization, lacking an integrated framework to consider both load forecasting accuracy and resource scheduling efficiency. Second, Attention Mechanism is a mechanism used in deep learning to highlight important parts of an input sequence, helping the model focus on the most critical information. Graph Neural Network (GNN) is a deep-learning model designed to process graph-structured data that captures the complex relationships between nodes. Although attention mechanism and GNN can enhance model performance in theory, their application effect and generalization ability on large-scale and actual power grid data still need to be further verified.

In the field of smart grid, with the increasing proportion of renewable energy and the complexity of power system, traditional forecasting and management methods have been difficult to meet the needs of modern power grid. Current methods have obvious shortcomings in load forecasting accuracy, sensitivity of grid state sensing and effectiveness of resource allocation strategy, which not only affect the stable operation of power grid, but also limit the overall efficiency and sustainable development of power system. In this study, a smart grid model named “GridOptiPredict” is proposed, which aims to solve the above problems by integrating deep learning and graph neural network technology, especially by introducing the “grid network prediction” module. This innovation not only makes up for the shortcomings of the existing technology in processing large-scale complex power system data, but also greatly improves the accuracy of load forecasting, the sensitivity of grid state sensing and the effectiveness of resource allocation strategies. The development of GridOptiPredict model has important theoretical and practical value for improving the intelligent level of smart grid, ensuring the safe and stable operation of power system and promoting the sustainable development of energy industry.

The point lattice network prediction, a core component of the GridOptiPredict model, significantly outperforms current methods by leveraging a unique architecture that captures intricate spatial and temporal dependencies in power system data more effectively. This advanced approach not only enhances the accuracy of load forecasting but also improves the sensitivity of power grid state sensing and the efficiency of resource allocation strategies. Compared to conventional methods, the point lattice network prediction exhibits superior model stability, robustness, and security, while also delivering notable social and economic benefits and enhancing user experience in the context of smart grid operations.

Correlational research

Smart grid

Smart Grid was originally defined by the Institute of Electrical and Electronics Engineers (IEEE) as a power supply network that integrates advanced sensing, control methods, advanced computing and information technology, and two-way communication to improve the efficiency, reliability, economy, and sustainability of the power grid. This definition highlights the innovation of smart grids over traditional grids, particularly in terms of information processing and interaction capabilities (Zhang et al. 2022).

The core features of smart grids highlight them as key to innovation in the energy sector in the 21st century. Firstly, its self-healing capability greatly reduces the fault recovery time through automatic control. The case study in literature (Nisa et al. 2021) clearly shows how smart grids effectively isolate problem areas and quickly reconfigure the network to ensure the continuity of power supply services. Secondly, the compatibility and integration design of smart grid supports the smooth access of multiple power generation modes, especially strengthens the efficient integration of renewable energy. Liu and Zhou (Liu and Zhou 2021) deeply discussed how this feature promotes the optimal utilization of distributed energy resources. Finally, the optimization feature comprehensively optimizes the grid operation and resource allocation with the help of big data analysis and advanced algorithm tools, which significantly improves the overall efficiency of the system (Zhou et al. 2020). Lu et al. (Lu et al. 2020) discusses this in detail and emphasizes the core role of data-driven strategy in smart grid optimization.

Analysis of data characteristics of power system

Power system data characteristic analysis is an important link in smart grid research and practice, which is directly related to the efficiency of data processing and the accuracy of decision-making.

Power system data covers the whole chain information from power generation, transmission, distribution to electricity consumption, including but not limited to real-time monitoring data, historical archives data, meteorological data, market transaction data, etc. (Li et al. 2022a, b). Real-time monitoring data, such as current, voltage, power, etc., high frequency, strong continuity, requiring immediate processing and analysis; historical file data is used for long-term trend analysis and fault tracing. Wang et al. (Wang et al. 2020a, b) highlighted the challenge of integrating data from different sources and formats. With the popularization of smart meters and the upgrading of monitoring systems, the amount of data generated by power systems per day has reached PB level, showing an exponential growth trend (Sun 2024). A series of quality control methods for power system data are proposed by Pinto et al. (Pinto et al. 2023), emphasizing the role of data preprocessing in improving the reliability of analysis results.

Current status of power data analysis

As the core support of smart grid operation, planning and decision-making, power data analysis has made remarkable progress in recent years, but it is also facing unprecedented challenges. Deep learning techniques, especially convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants (such as LSTM), show strong potential in power load forecasting (Fan 2021). By automatically extracting complex features, these models can handle nonlinear, high-dimensional data, significantly improving the accuracy and robustness of predictions. For example, Liu et al. (Liu et al. 2023) used LSTM combined with attention mechanism to effectively improve the learning ability of long-term dependence on power load time series. In addition, machine learning methods such as ensemble learning and transfer learning have also been applied to power system fault diagnosis and state evaluation, further enriching the analysis means (Elavarasan et al. 2020). With the increasing complexity of power system data structure, graph neural network has become an emerging tool for power grid state sensing and fault detection because of its ability to process node and edge characteristics (Jamil et al. 2021). GNN captures the physical and electrical characteristics of the grid topology, improving the accuracy and speed of identifying abnormal grid behavior. Li et al. (Li et al. 2023) shows GNN’s excellent performance in real-time monitoring of power grid status and identification of potential fault points, which provides a strong guarantee for safe and stable operation of power grid. Smart grid data analytics are increasingly used in demand response and energy efficiency management. Byanalyzing smart meter data and combining machine learning algorithm, Dalmaijer et al. (Dalmaijer et al. 2022) realized refined classification of user electricity consumption behavior, providing data support for customized demand response strategy. In addition, through predictive analysis and optimization algorithms, smart grids can balance supply and demand more effectively and improve energy efficiency. As the share of renewable energy increases, power data analysis plays a key role in facilitating its efficient interconnection and scheduling. Zhang and Li (Zhang and Li 2021) accurately predicts the output of wind power and photovoltaic power generation by using big data analysis and prediction model, supports flexible dispatching of power grid and optimal allocation of energy storage resources, and improves the renewable energy acceptance capacity of the whole power grid.

Despite technological advances, data quality remains a challenge for power data analysis. Missing data, noise, inconsistency and storage and management problems of large-scale data directly affect the reliability and practicability of analysis results (Liu et al. 2021). Improving data preprocessing and quality control technology and developing efficient data management platform is one of the emphases of current research.

“GridOptiPredict” model building

Model framework

The design of GridOptiPredict model aims to provide a comprehensive solution for smart grid, integrating load forecasting, grid state sensing and resource optimization into one, forming a closely connected and complementary framework system. The core goal of the framework is to achieve efficient analysis and processing of power system data through deep fusion of deep learning and graph neural networks, thereby improving the intelligent level and overall efficiency of power grid operation.

Fig. 1

Model hierarchy

The GridOptiPredict model architecture can be summarized into three levels: the data input level, the core processing level, and the policy output level. The data input layer is responsible for collecting and initially processing various data from the smart grid, including but not limited to real-time monitoring data, historical archive data, meteorological data, etc. The core processing layer is the heart of the model, which includes load forecasting module, power grid state sensing module and resource optimization configuration module. These three modules work together to realize in-depth analysis and strategy generation of power grid operation state. The policy output layer outputs optimized resource allocation scheme, fault warning and response strategy according to the analysis results of the core processing layer, providing scientific basis for daily management and emergency response of the power grid. The specific hierarchical structure is shown in Fig. 1 (Xu et al. 2023).

The modules within the model do not operate in isolation, but are closely linked through elaborate interaction mechanisms. The output of load forecasting module not only serves resource optimization configuration, but also provides auxiliary information for grid state sensing module to help understand the impact of load change on grid state. The innovation of the GridOptiPredict model lies in its strategy of deeply integrating multiple advanced technologies. The combination of LSTM and attention mechanism strengthens the processing ability of the model for time series data, making load forecasting more accurate; the application of graph neural network plays a key role in understanding complex power grid structure and real-time state; the introduction of reinforcement learning technology enables the resource optimization allocation module to dynamically adjust the strategy according to the real-time demand and future forecast of the power grid to achieve the best balance between cost and efficiency.

Detailed model components

Load forecasting module

In the field of power system load forecasting, traditional time series analysis methods are often difficult to fully mine complex, nonlinear and periodic features. Therefore, this section introduces an innovative model component-improved LSTM and advanced attention mechanism fusion model, which aims to significantly enhance the capture ability of complex time series data and improve the accuracy and stability of prediction. The power system load prediction module is shown in Fig. 2 (Wang et al. 2022).

Fig. 2

Power system load forecasting

Long Short Term Memory Network (LSTM), as a special recurrent neural network, can effectively manage long-term dependent information through its unique gating mechanism (including input gate, forgetting gate and output gate), and solve the gradient disappearance problem of ordinary RNN when dealing with long sequences. To further improve the modeling capabilities of LSTM for complex time series, we have made two key improvements to the standard LSTM:

By introducing time windows of different scales, LSTM can capture the characteristics of load variation on different time scales. Specifically, multiple LSTM layers with different time steps are arranged in parallel, with each layer focusing on different periodic load patterns such as diurnal variations, weekly cycles and seasonal trends. This is shown in Eq. (1) (Ma et al. 2019).

$$h_{t}^{l}=LST{M^l}(h_{t}^{{l - 1}},h_{{t - 1}}^{{l - 1}},.,h_{{t - {k_l}}}^{{l - 1}})$$

(1)

where, denotes the hidden state of LSTM of layer 1 at time step t, and is the unique time window size of layer 1. In order to alleviate the gradient disappearance and explosion problems in deep network training, we introduce residual connections between adjacent LSTM layers, so that deeper layers can directly learn the information differences of shallow layers, as shown in Eq. (2).

$$h{^{\prime}}_t^l = h_t^{l - 1} + h_t^l$$

(2)

Although LSTM itself has the ability to process time series, it is still insufficient to accurately filter important information in complex scenes. To this end, we designed a high-level attention mechanism that not only based on the intrinsic correlation of time series, but also incorporated the influence of external factors (such as weather conditions, holidays, etc.) to adjust the weight of information more finely. Specifically, our attention mechanism consists of two parts: (1) Intrinsic Attention: based on the hidden state of LSTM, the most critical time points for prediction are emphasized by weighted summation. The weight is determined by the similarity between the current hidden state and the previous hidden state, as shown in Eqs. (3)-(4) (Sarker et al. 2023).

$$a_t^l = {{\exp (e_t^l)} \over {\sum\limits_{i = t - {k_l}}^t {\exp } (e_i^l)}}$$

(3)

$$e_t^l = {v^T}\tanh ({W_h}h_t^l + {W_{hh}}h_{t - 1}^l + {b_a})$$

(4)

(2) Extrinsic Attention: Weighting to account for external factors adjusts the LSTM output through an additional layer of attention, as shown in Eqs. (5)-(6).

$$w_t^e = \sigma ({W_e}[x_t^e,h{^{\prime}}_t^l] + {b_e})$$

(5)

$$h'{'_t} = w_t^e \odot h{^{\prime}}_t^l$$

(6)

The feature vector representing the external factor is a sigmoid function used to generate the weighting factor.

Combining the output of LSTM and the characteristics after internal and external attention adjustment, we design a multilayer fully connected network as the prediction layer, and its output is the load prediction value at the future time, as shown in Eq. (7).

$${y_{pred}} = Dense(h_t^{''};{W_d},{b_d})$$

(7)

Model training uses mean squared error (MSE) as a loss function to minimize the gap between predicted and true values, as shown in Eq. (8).

$$L=\frac{1}{N}\sum\limits_{{i=1}}^{N} {{{({y_{pred,i}} - {y_{true,i}})}^2}}$$

(8)

Through the fusion of the above improved LSTM structure and advanced attention mechanism, the model can not only efficiently capture the long-term dependence and periodic features of time series, but also dynamically adjust for specific events in complex scenes, improving the accuracy and generalization ability of load prediction. The innovative application of this deep learning framework provides a powerful tool for power system load forecasting, helping to optimize energy allocation and improve system operating efficiency (Sarker et al. 2023).

Grid Status Awareness Module

In modern power system, real-time sensing and anomaly detection of power network state is an important link to ensure the reliability and security of power supply. Traditional monitoring methods are limited by inefficient understanding of complex power grid topology and limitations of static models, and it is difficult to accurately predict and quickly respond to abnormal conditions in power grids. Therefore, this section introduces a power grid state sensing module based on Graph Neural Networks (GNN), which can deeply understand the topological relationship of the power grid, capture and analyze the operating state of the power grid in real time, and significantly improve the accuracy of anomaly detection and fault prediction (Sheng et al. 2018; Yudho 2020).

Graph neural network is a deep learning model specially designed for processing graph data, which can act directly on nodes, edges and the whole graph structure, and is very suitable for expressing and learning complex topology information of power grid.

Specifically, for a node in the power grid${v_i}$, its characteristic represents$h_{i}^{l}$the update process in the l-th GNN as shown in Eq. (9) (Thurner et al. 2018).

$$h_{i}^{{l+1}}=UPDATE(\sum\limits_{{j \in \mathcal{N}(i)}} A GGREGATE(h_{j}^{l},{e_{ij}}),h_{i}^{l})$$

(9)

The AGGREGATE function aggregates the information of neighbor nodes, and the UPDATE function integrates neighbor information and its own characteristics to update the node representation.

The physical structure of power grid can be abstracted as a weighted graph G=(V, E), where V is the node set, E is the edge set, each edge carries information about physical attributes such as line capacity and resistance, and the eigenvectors of nodes may include electrical parameters such as voltage, current, power, etc. Through this graph representation, GNN can comprehensively consider the interactions and dependencies between nodes, and provide rich context information for subsequent state monitoring.

In this section, we will discuss the implementation of real-time state sensing and anomaly detection module, focusing on how to use graph neural network (GNN) model to analyze the real-time state of power grid and identify anomalies. We will start from the aspects of abnormal signal extraction, feature fusion, and abnormal scoring mechanism. The specific framework is shown in Fig. 3.

First, based on the graph structure G=(V, E) of the power grid, we use graph convolution (GC) operation to learn the abnormal feature representation of nodes. Suppose the initial node eigenmatrix is, where n is the number of nodes and d is the eigendimension. The GC process is shown in Eq. (10) (Altuntas and Gok 2023).

$${H^{(l+1)}}=\sigma ({\tilde {D}^{ - \frac{1}{2}}}\tilde {A}{\tilde {D}^{ - \frac{1}{2}}}{H^{(l)}}{W^{(l)}})$$

(10)

where the hidden representation matrix representing the lth layer is the weight matrix of the layer, is the nonlinear activation function (e.g. ReLU), is the adjacency matrix A plus the identity matrix I to consider the self-loop, is the diagonal matrix whose diagonal elements are the sum of each row of for normalization processing.

Considering the variation of power grid state with time, we need to incorporate time series characteristics into the model. A temporal convolutional network (TCN) is used to process time series data, and its formula can be expressed as Formula (11).

$${Z^{(l)}}={\text{ReLU}}({W^{(l)}}*{Z^{(l - 1)}}+{B^{(l)}})$$

(11)

where, is the output ofthe lthlayer, representing the convolution operation, and are the convolution kernel and bias term, respectively, and ReLU is the activation function. Through multilayer stacking, TCN is able to extract feature patterns across time steps (Fan et al. 2020).

By fusing the above spatial (graph convolution) and temporal (temporal convolution) features, we design a comprehensive scoring mechanism to evaluate the likelihood of anomaly of each node at a certain time. One possible approach is to combine the feature representations of the two and obtain the final anomaly score through a fusion layer as shown in Eq. (12).

$$S={\text{FC}}([\hat {H};\hat {Z}])+b$$

(12)

where, and are the node feature matrices processed by several layers of GC and TCN respectively, and are spliced after flattening operation; represent the fully connected layer, which is used to learn the mapping from features to anomaly scores, and b is the bias term.

To identify anomalies from the model output, we need to set a threshold. When the exception rating of a node is greater than 0, we consider the node to be in an abnormal state. Threshold selection is usually based on statistical analysis of historical data, or through cross-validation methods to determine the best threshold to balance false positive and false negative rates.

The training of the model aims to minimize the gap between predicted anomaly scores and true labels. If we adopt the perspective of binary classification problem, we can use the cross-entropy loss function, as shown in Eq. 13.

$$L= - \sum\limits_{{i=1}}^{n} {\left[ {{y_i}\log ({p_i})+(1 - {y_i})\log (1 - {p_i})} \right]}$$

(13)

where is the true anomaly label of node i (0 means normal, 1 means abnormal), and is the predicted anomaly probability transformed by sigmoid function.

To sum up, through the graph neural network model and anomaly scoring mechanism described above, the real-time state awareness and anomaly detection module can effectively extract features from the topology structure and time series data of the power grid, accurately identify abnormal states, and provide strong technical support for the safe operation of the power grid. Furthermore, by constructing spatial-temporal graph convolutional networks (STGCN), the model can predict future power grid state changes using historical data, thus realizing early warning of faults. STGCN combines spatial graph convolution (dealing with topological relations) with temporal convolution (dealing with time series characteristics), and its information transfer in the time dimension can be expressed as Eq. (14).

$$h_{{i,t}}^{{l+1}}=TCN(h_{{i,t - \tau :t}}^{l},h_{{i,t - 1}}^{{l+1}})$$

(14)

TCN stands for time convolutional network, which is the feature sequence of nodes in the past several time steps, so that dynamic changes in time series can be captured.

To sum up, by applying graph neural network to power grid state sensing, not only can the complex topology structure of power grid be understood and learned efficiently, but also the operation state of power grid can be monitored in real time, and the accuracy of anomaly detection and fault prediction can be effectively improved.

Fig. 3

Grid Status Awareness Module

Resource optimization configuration Module

In power system management, resource optimization allocation is the core link to ensure the reliability and economy of power supply. The resource optimization allocation module introduced in this section integrates the output of load forecasting module and reinforcement learning strategy to dynamically adjust resource allocation and maximize power supply reliability in a changing supply and demand environment with the goal of minimizing operating costs.

Reinforcement learning (RL) is a machine learning method that learns optimal strategies by interacting with the environment. In the power system resource optimization scenario, environment refers to the actual situation of power grid operation, including load demand, equipment status, transmission line capacity, etc.; Agent refers to the decision maker responsible for making resource allocation decisions; Action refers to the actions that the agent can perform, such as adjusting generator output, dispatching energy storage equipment, etc.; State describes the current overall situation of the power grid; Reward refers to quantitative feedback on the improvement of power grid performance after taking a certain action.

State space: defined as a comprehensive description of the grid, including but not limited to the current load level, the available capacity of each power station and line, and the state of energy storage equipment. Specifically, if the grid has n nodes and m state variables, the states can be represented as vectors.

Action space: Covers all possible resource adjustment strategies, such as increasing/decreasing the power generation in a certain area, scheduling backup power access, adjusting the charge and discharge of energy storage equipment, etc. The size of the action space depends on the complexity and control granularity of the system and can be expressed as.

The optimization objective is to minimize operating costs while ensuring power supply reliability. Reliability can be measured by indicators such as expected outage time and frequency, while costs include fuel consumption, maintenance costs, and electricity purchase costs. Therefore, the reward function\(R(s, a)\) needs to be designed to reflect this dual objective, as shown in Eq. (15).

$$R({s_t},{a_t})=\alpha \cdot {\text{Reliability}}({s_t},{a_t}) - \beta \cdot {\text{Cost}}({s_t},{a_t})$$

(15)

where and are weighting coefficients that balance reliability and cost, and the sum function quantifies the power supply reliability and cost changes in the post-action state,

Q-learning is a strategy based reinforcement learning method, which is suitable for solving resource allocation problems. It learns the optimal policy by iteratively updating a Q table or function, where Q(s, a) represents an estimate of the expected cumulative reward after state s takes action a. The core update formula for Q learning is Eq. 16.

$$Q({s_t},{a_t}) \leftarrow Q({s_t},{a_t})+\eta \cdot ({r_{t+1}}+\gamma \cdot {\hbox{max} _{a^{\prime}}}Q({s_{t+1}},a^{\prime}) - Q({s_t},{a_t}))$$

(16)

Facing the high dimensional state space of large-scale power grid, the traditional Q table is difficult to be directly applied, so the innovative Deep Q-Network (DQN) is studied. DQN uses deep neural networks to approximate the Q function, and the network structure typically includes an input layer, several hidden layers, and an output layer, where the input layer receives the grid state information and the output layer gives the expected reward for each possible action. The loss function for DQN is defined as Eq. (17).

$${L_i}({\theta _i})={{\mathbb{E}}_{(s,a,r,s^{\prime})\sim U(D)}}[{(r+\gamma \cdot {\hbox{max} _{a^{\prime}}}Q(s^{\prime},a^{\prime};\theta _{i}^{ - }) - Q(s,a;{\theta _i}))^2}]$$

(17)

where U(D) is the process of sampling data from the experience playback buffer D, and are parameters of the current policy network and the target network, respectively, which are used to stabilize the learning process and reduce training fluctuations. By periodically copying the parameters of the policy network to the target network (soft update or hard update), stability of the target values and effectiveness of learning can be ensured. This separation strategy (i.e., using two networks) is one of the keys to the success of DQN, and it solves the training instability problem encountered when estimating Q directly in high-dimensional states.

Fig. 4

Algorithm flow of optimal allocation of power system resources

In actual implementation, the algorithm flow of optimizing allocation of power system resources by using DQN can be summarized as the following steps, and the specific flow is shown in Fig. 4. (1) Initialization: Set initial parameters, including the size of experience playback buffer D, learning rate$\eta$, discount factor$\gamma$, exploration strategy parameters (such as initial$\epsilon$ values), Q network and target network architecture and parameters $\theta ,\,{\theta ^ - }$. (2) Environmental interaction: For each time step t, actions are executed according to the current policy (possibly actions chosen based on the-greedy policy), new states are observed and immediate rewards are obtained. (3) Experience storage: store experiences in experience playback buffer D. When the buffer is full, old experiences are replaced on a first-in, first-out (FIFO) basis. (4) Batch sampling: A batch of samples is randomly extracted from the experience playback buffer for training. (5) Update Q-network: Optimize Q-network using the following loss function, as shown in Eq. (18).

$$L({\theta _i})={{\mathbb{E}}_{({s_j},{a_j},{r_j},{s_{j+1}})\sim U(D)}}\left[ {{{({r_j}+\gamma {{\hbox{max} }_{a^{\prime}}}Q({s_{j+1}},a^{\prime};\theta _{i}^{ - }) - Q({s_j},{a_j};{\theta _i}))}^2}} \right]$$

(18)

(6) Target network update: Periodically (e.g., every C steps) copy the parameters of the policy network to the parameters of the target network to maintain the stability of the learning, as shown in Eq. (19).

$$\theta _{i}^{ - } \leftarrow {\theta _i}$$

(19)

(7) Exploration strategy adjustment: gradually reduce$\epsilon$ the value of, reduce the proportion of random exploration, and increase the proportion of action according to the current optimal strategy.

Experimental evaluation

Experimental design

In order to comprehensively evaluate the practical value and effectiveness of the “GridOptiPredict” model in the field of smart grid, we carefully planned a set of careful experimental schemes to comprehensively verify the accuracy of load forecasting, the sensitivity of grid state sensing, and the efficiency of resource allocation strategies. The experimental design strategy covers a variety of test scenarios, including both highly simulated and real-world datasets derived from real-world operations, to verify the universality and robustness of the model under various conditions.

In the data preprocessing stage, all data sets follow a strict 70%-15%-15% proportion division, which ensures that the whole process from training, parameter fine-tuning to final performance verification of the model is reasonable and efficient. This arrangement provides sufficient training material for the model, while also retaining independent data sets to objectively evaluate the generalization ability of the model.

To ensure objectivity and comprehensiveness of the assessment, we carefully selected a range of industry-accepted baseline models for comparison. For example, the classical time series model ARIMA is used to measure whether our load forecasting module exceeds traditional methods; the fixed rule-based system is used as a reference for the state sensing module to highlight the superiority of our graph neural network strategy; and in terms of resource allocation, we choose the linear programming model as a benchmark to verify the innovation of the reinforcement learning-driven DQN strategy in terms of actual cost savings and efficiency improvement.

To ensure the reproducibility and rigor of the experiments, we carefully selected a diverse set of data sources, including both synthetic and real-world datasets. The synthetic data was generated to simulate typical operational scenarios, while the real-world data was sourced from historical grid operations, providing a comprehensive evaluation of the model’s performance under various conditions. All datasets were preprocessed to maintain a consistent 70%-15%-15% split for training, validation, and testing, respectively.

For parameter settings, we employed a grid search approach combined with cross-validation to find the optimal hyperparameters for each model, ensuring that the models were well-tuned for maximum performance. The hyperparameters included learning rates, batch sizes, and regularization coefficients, among others.

Specific evaluation metrics used to assess the model’s performance included Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Coefficient of Determination (R² Score) for predictive accuracy. For model stability and robustness, we utilized cross-validation scores and sensitivity analysis. Resource optimization was evaluated through cost savings, power supply reliability improvement, and energy efficiency improvement percentages. Safety metrics included failure prediction accuracy, recall rate, and response time. Information security was assessed through data breach risk scores and access control effectiveness. Socio-economic benefits were quantified using Return on Investment (ROI) and energy-saving and emission reduction benefits. Finally, user experience was gauged through customer satisfaction scores. These metrics provided a comprehensive view of the model’s effectiveness and practical value.

The implementation environment of the experiment is deployed on the high-performance computing cluster, which not only speeds up the model training and inference process, but also ensures the stability of complex computing tasks, laying a solid technical foundation for the deep optimization and large-scale application of the model.

Experimental results

The data in Table 1 were obtained by training and testing each model using the same training and test sets. During training, the model makes predictions by learning the relationship between input data and target values. Mean Square Error (MSE), Root Mean Square Error (RMSE), Mean Absolute Error (MAE), and Coefficient of Determination (R² Score) were used to measure the predictive accuracy of the model. The MSE metric represents the average of the squares of the differences between the model predictions and the actual values, with smaller values indicating higher prediction accuracy of the model. The RMSE metric represents the square root of MSE, which visually reflects the magnitude of the error between model predictions and actual values. As can be seen from Table 1, the GridOptiPredict model performed best on prediction accuracy metrics, with MSE, RMSE, MAE, and R² Score outperforming the other models. This shows that the GridOptiPredict model has better performance in terms of prediction accuracy, being able to predict the value of the target variable more accurately.

Table 1 Prediction accuracy indicators

The data in Table 2 were evaluated for each model using cross-validation and sensitivity analysis. Cross-validation is a method of evaluating the performance of a model by dividing a data set into multiple subsets and using each subset in turn as a test set, with the remaining subsets being trained and evaluated as a training set. The cross-validation score represents the average performance of the model across multiple datasets, with higher values indicating better model stability.

Table 2 Model stability and robustness indicators

As can be seen from Table 2, the GridOptiPredict model performed best on model stability and robustness indicators, with higher cross-validation scores and sensitivity analysis scores than other models. This shows that the GridOptiPredict model has better performance in terms of model stability and robustness, and is better able to adapt to different data sets and resist the impact of data changes.

Table 3 Resource optimization indicators

The data in Table 3 are derived by testing each model with actual data sets. During the test, the percentage improvement in cost savings, power supply reliability and energy efficiency of the model was calculated. The cost savings metric indicates the proportion of costs that the model can help reduce, and a higher value indicates that the model has better performance in terms of cost savings. Power supply reliability improvement index indicates the degree to which the model can improve power supply reliability, and the higher its value, the better the performance of the model in power supply reliability. The Energy Efficiency Improvement Index indicates the extent to which the model can improve energy efficiency, with higher values indicating better performance of the model in terms of energy efficiency. As can be seen from Table 3, the GridOptiPredict model performs best on resource optimization indicators, with higher percentage of cost savings, power reliability, and energy efficiency improvements than the other models. This shows that the GridOptiPredict model has better performance in resource optimization, which can reduce costs, improve power supply reliability and energy efficiency more effectively.

Table 4 Safety indicators

The data in Table 4 are derived by testing each model using actual data sets. During the test, the performance indexes of the model in terms of fault prediction accuracy, recall and response time are calculated. The failure prediction accuracy index indicates the proportion that the model can correctly predict failures, and the higher the value, the higher the failure prediction accuracy of the model. The recall index indicates the proportion of positive classes that the model can correctly predict, and the higher its value, the stronger the recall ability of the model. The response time indicator indicates the time required for the model to go from input data to output prediction results, and the lower the value, the faster the response speed of the model. As can be seen from Table 4, the “GridOptiPredict” model performs best in safety indicators, and its failure prediction accuracy, recall rate and response time are better than other models. This shows that the GridOptiPredict model has better performance in terms of safety, can predict failures more accurately, and has higher recall ability and faster response speed.

Table 5 Information Safety indicators

The data in Table 5 are derived by testing each model using actual data sets. During testing, data breach risk scores and access control effectiveness scores were used to assess the information security of the model. The data breach risk score indicates the resistance ability of the model to data breach risk, and the higher the value, the better the information security of the model. Access control effectiveness score indicates the effectiveness of the model on access control, and the higher its value, the better the model can protect data from unauthorized access. As can be seen from Table 5, the GridOptiPredict model performs best on information security metrics, with higher data breach risk scores and access control effectiveness scores than other models. This shows that the GridOptiPredict model has better performance in terms of information security and can better protect data security and privacy.

Table 6 Indicators of socio-economic benefits

The data in Table 6 are derived by testing each model using actual data sets. During the test, the performance indicators of the model in terms of return on investment and energy conservation and emission reduction benefits are calculated. The ROI metric indicates the impact of the model on ROI, with higher values indicating that the model can deliver higher ROI. Energy saving and emission reduction benefit index indicates the contribution of the model to energy saving and emission reduction, and the higher the value, the better the performance of the model in energy saving and emission reduction. As can be seen from Table 6, the “GridOptiPredict” model performs best in socio-economic benefit indicators, and its return on investment and energy conservation and emission reduction benefits are higher than other models. This shows that the GridOptiPredict model has better performance in terms of socio-economic benefits, which can bring higher return on investment and greater energy conservation and emission reduction benefits.

Table 7 User experience indicators

The data in Table 7 are derived by testing each model using actual data sets. During testing, customer satisfaction scores were used to assess the user experience of the model.

The Customer Satisfaction Score metric indicates how satisfied users are with the performance of the model, with a higher value indicating a better user experience for the model. As you can see from Table 7, the GridOptiPredict model performed best on user experience metrics, with higher customer satisfaction scores than the other models. This shows that the GridOptiPredict model has better performance in terms of user experience and is better able to meet user needs and expectations.

In the GridOptiPredict model, the Grid Network Prediction module significantly improves the implementation efficiency of the model through its unique architecture. It can efficiently capture spatial and temporal dependencies in power system data, thus improving the accuracy of load forecasting and optimizing the execution speed of resource allocation strategies. Compared with existing methods, grid network prediction not only enhances the overall stability and robustness of the model, but also significantly reduces the computational delay, ensuring real-time monitoring and rapid response capabilities of the power system, thus achieving more efficient smart grid operation.

In GridOptiPredict model, the “grid network forecasting” module forms a closely linked and mutually reinforcing relationship by improving the accuracy of load forecasting, enhancing the sensitivity of grid state sensing and optimizing the effectiveness of resource allocation strategies. Accurate load forecasting provides a reliable basis for resource allocation, which enables power systems to allocate resources more efficiently during peak demand periods; sensitive grid state sensing capabilities can detect abnormal changes in the system in time, providing immediate feedback for adjusting resource allocation strategies, further enhancing the stability and flexibility of the entire system. This interaction not only improves the overall efficiency of grid operation, but also ensures the safety and reliability of power supply, resulting in significant socio-economic benefits.

Discussion

Table 8 Computational complexity, scalability, and Data Dependency

Comparison with Advanced methods

The GridOptiPredict model demonstrates superior performance across several key metrics, as shown in Tables 1, 2, 3, 4, 5, 6 and 7. However, to further substantiate its superiority, this section provides a more detailed comparison with the most advanced methods currently available. Compared with existing deep learning and graph neural network models, GridOptiPredict not only achieves better results in terms of prediction accuracy, but also excels in model stability, resource optimization, security, information security, socio-economic benefits, and user experience. Additionally, the GridOptiPredict model, through its unique grid network prediction module, significantly improves implementation efficiency and can efficiently capture spatial and temporal dependencies in power system data, as shown in Table 8.

Limitations discussion

Despite its many strengths, the GridOptiPredict model faces some limitations and challenges. For example, although its computational complexity is rated as medium, handling very large datasets may strain computational resources. Furthermore, while the GridOptiPredict model shows good scalability in experiments, it may require additional hardware resources when deployed in larger grid systems. Additionally, the model’s performance relies on high-quality and diverse datasets, which means that poor data quality or limited data variety could impact prediction performance. Lastly, despite scoring highly in security, the model will need to enhance its defensive mechanisms against complex cyber attacks and data breaches.

Future work directions

To overcome these limitations and further enhance the performance of the GridOptiPredict model, future research could focus on the following areas:

• (1) Develop more efficient data preprocessing and feature selection methods to reduce computational complexity and improve scalability.

• (2) Explore the model’s adaptability, particularly in handling extreme weather events and other atypical scenarios.

• (3) Strengthen the model’s security by enhancing its resilience to potential attacks and improving data privacy protection.

• (4) Further optimize the model design to achieve optimal performance in different geographical regions and grid configurations.

Conclusion

By constructing “Grid OptiPredict” model, this study effectively integrates three core functions of load forecasting, power grid state sensing and resource optimization allocation, forming a closely connected and complementary framework system. Experimental results show that the model has significant advantages in prediction accuracy, model stability and robustness, resource optimization, security, information security, social and economic benefits and user experience. In terms of prediction accuracy, the model adopts the improved LSTM and advanced attention mechanism fusion model, which significantly enhances the ability to capture complex time series data and improves the accuracy and stability of prediction. In terms of model stability and robustness, the model was evaluated by cross-validation and sensitivity analysis methods. The results showed that the “GridOptiPredict” model had higher scores in cross-validation and sensitivity analysis than other models, indicating its advantages in model stability and robustness. In the aspect of resource optimization, the model integrates the output of load forecasting module and reinforcement learning strategy to dynamically adjust resource allocation and maximize power supply reliability in the changing supply and demand environment with the goal of minimizing operating costs. In terms of safety, the model is tested by using actual data sets. The results show that the GridOptiPredict model is superior to other models in failure prediction accuracy, recall and response time, indicating its advantages in terms of safety. In terms of information security, the model uses data breach risk score and access control effectiveness score to evaluate the information security of the model. The results show that the “GridOptiPredict” model is higher than other models in these two scoring indicators, indicating its advantage in information security. In terms of socio-economic benefits, the model calculates the performance indicators of the model in terms of return on investment and energy conservation and emission reduction benefits. The results show that the “GridOptiPredict” model is higher than other models in these two indicators, indicating its advantages in terms of socio-economic benefits. In terms of user experience, the model uses customer satisfaction scores to evaluate the user experience of the model, and the results show that the customer satisfaction score of the GridOptiPredict model is higher than that of the other models, indicating its advantage in user experience.