Skip to main content

Non intrusive load monitoring for demand side management

Abstract

In the context of a pilot project, the Lugaggia Innovation Community (LIC), we address the problem of non-intrusive load monitoring for the purpose of demand side management on low voltage grids in presence of distributed power generation (photovoltaic). From the power readings of smart meters, we estimate the photovoltaic production and detect the activation of major loads (heatpumps and domestic water heaters). Experiments, conducted with real data and in silico, show that exploiting meter readings only, we can estimate PV production with MAPE ranging from 4.6% (best case) to 41.9% (worst case). Even with non negligible photovoltaic production estimation errors, the proposed method is capable of detecting the activation of heatpumps and domestic water heaters.

Introduction

Distributed power generation is spreading considerably in many countries thanks to different political and economic actions. In Switzerland, the “Energy Strategy 2050”Footnote 1 is supporting the installation of new renewable sources which, at domestic level, are mainly represented by photovoltaic installations. While this growth is beneficial for reducing the CO2 emissions, it poses a series of issues from the viewpoint of grid operators as they are contractually bound to provide a certain power quality. One of the possible mitigation actions to overcome these issues is represented by active Demand Side Management (DSM), that is a set of actions to directly control a site’s energy consumption while respecting end user needs. In the context of a pilot project, the Lugaggia Innovation Community (LIC) funded by the Swiss Federal Office of Energy SFOE project SI/501840-01, we study the application of active DSM on a set of electrical loads spread in a residential area with the presence of renewable sources as well as energy storage systems. The municipality of Capriasca (Ticino, Switzerland), where the LIC community is located, recently installed a PV plant on the roof of the local kindergarten and other PV installations are already present in the area. The controllable loads and photovoltaic installations are summarized in Table 1. Aerial pictures of the PV installations are shown in Fig. 1a and b.

Fig. 1
figure1

Photovoltaic installations, source: https://map.geo.admin.ch

Table 1 Characteristics of the installations in the LIC area

The focus of this paper is to detail the methodology we use for Non-Intrusive Load Monitoring (NILM). In particular we address the issue of estimating the photovoltaic production in absence of a direct monitoring infrastructure and in absence of accurate meteorological data.

State of the art

Energy disaggregation or non intrusive load monitoring has been studied since the 90s when Hart (Hart 1992) proposed his approach based on the detection of edge changes in the power signal, their clustering, and the subsequent optimal allocation to the possible appliances in order to get the best fit for the aggregate signal. This approach has influenced many authors which followed and some of its elements are still in use in the most effective algorithms. Extensive reviews of such algorithms are provided, for instance, by Zoha et al. (2012) and by Zeifman and Roth (2011).

NILM algorithms can be classified as proposed by Schirmer et al. (2020) who distinguish between algorithms using source separation, and those who does not. Source separation, or blind signal separation, is a known filtering problem where an aggregate signal composed by a mixture of a set of other signals is processed in order to obtain the individual contributions. This is exactly the problem that NILM attempts to solve and source separation has been solved in the past using a variety of approaches, from principal component analysis to singular value decomposition, to non-negative matrix factorisation. All these methods are characterised by the fact that they are unsupervised and they try to optimise an objective function that represents the distance of the aggregate signal from the sum of the independent contributions.

The problem of source-separation algorithms is that their computational load is quite intense and that, in order to improve their performance, they also need to know with a certain detail the number and type of appliances into which the signal is to be decomposed, thus trading in the advantage of being unsupervised.

Algorithms that does not use the source separation approach are typically supervised algorithms that require substantial amounts of training data to take advantage of machine learning algorithms such as Support Vector Machines, Decision Trees and Deep Neural Networks such as the one by (Kong et al. 2019).

The approach presented in this paper is based on a source separation method and it is described in the next section. It builds on previous work (Cominola et al. 2017; Piga et al. 2016; Rottondi et al. 2019) and it is also inspired by recent research from Bu et al. (Bu et al. 2020). The choice of this approach was favoured not only by our previous experience with it, but also by the consideration that we knew beforehand which devices we wanted to disaggregate: in our case, heat pumps and domestic water heaters, considering all the other devices as a residual noise after having removed the contribution of PV power generation to the aggregated signal.

Methodology

Over a given discretized time frame T, a given power profile is characterized by an aggregated effect of several loads. In particular we are given the power profile as series of P-Q (active and reactive) points pt and qt,tT. We are given a set of controllable electrical loads L each characterized by a nominal power pl and a nominal reactive power ql,lL. We are also given an invariant representative dataset of non-controllable profiles S that are used to represent the contribution of the uncontrollable component of the power profile, each characterized by a given P-Q power point at time t defined by ps,t and qs,t. The NILM problem consists in determining at which moment in time load l was absorbing power and which combination of power profile s is the most appropriate to represent non-controllable loads.

Mixed integer quadratic problem (MIQP) formulation

Let xl,t{0,1} be the controllable load variables (i.e. xl,t takes value 1 if load l is estimated to absorb power at time t), ys[0,1] be the non-controllable profile selection variables (note ys is a continuous variable and can therefore represent the selection of any convenient fraction of the representative non-controllable load profile s). Let wpt and wqt be auxiliary variables used to represent the estimation error at time t.

$$\begin{array}{*{20}l} z = & \min \sum_{t \in T} (wp^{2}_{t} + wq^{2}_{t}) \end{array} $$
(1)
$$\begin{array}{*{20}l} & wp_{t} = \left | p_{t} - \sum_{l \in L} p_{l} \cdot x_{l,t} - \sum_{s \in S} p_{s,t} \cdot y_{s} \right|\qquad\qquad\qquad\forall t \in T \end{array} $$
(2)
$$\begin{array}{*{20}l} & wq_{t} = \left | q_{t} - \sum_{l \in L} q_{l} \cdot x_{l,t} - \sum_{s \in S} q_{s,t} \cdot y_{s} \right|\qquad\qquad\qquad\forall t \in T \end{array} $$
(3)
$$\begin{array}{*{20}l} & \sum_{s \in S} y_{s} = 1 \end{array} $$
(4)
$$\begin{array}{*{20}l} & x_{l,t} \in \{0,1\} \qquad\forall l \in L, t \in T \end{array} $$
(5)
$$\begin{array}{*{20}l} & y_{s} \in [0,1] \qquad \quad \forall s \in S \end{array} $$
(6)

Objective (1) is a convenient formulation to minimize the RMS error between the given power profile pt and qt and the aggregation of controllable and non controllable loads. Constraints (2) and (3) compute the absolute error as the difference between the meter reading pt and qt and the estimated contribution of loads lL and the reference power profiles sS at time t. Constraints (4) ensure that a convex combination of reference power profiles is selected.

Each controllable load l is connected to an actuation device and sometimes the metering infrastructure is able to provide, together with the regular power metering measurements, the status of the actuation device as an additional data series. Let ul,t{0,1},lL,tT be the representation of the state of the actuation device: 1, if the load l is connected to the grid, i.e. it can absorb power, at time t, 0 otherwise. We can conveniently exploit this knowledge in the MIQP formulation by setting upper bounds to variables xl,t as follows:

$$\begin{array}{*{20}l} & x_{l,t} \leq u_{l,t}\qquad\qquad\qquad \forall l \in L, t \in T \end{array} $$
(7)

The MIQP formulation can be further improved by the analysis of the type of electrical loads to be monitored. Indeed, some of the controllable electrical loads l are used to heat materials characterized by a considerable inertia (e.g. water, air, concrete and furnitures). We can incorporate this knowledge in MIQP. First, we have to capture the state change (first derivative) by adding continuous auxiliary variables 0≤fl,t≤1,lL,tTt0 that are linked with expected activation variables xl,t, then we limit the total number of state changes as follows:

$$\begin{array}{*{20}l} & f_{l,t} \geq x_{l,t} - x_{l,t-1}\qquad\qquad\qquad \forall l \in L, t \in T \setminus{t_{0}} \end{array} $$
(8)
$$\begin{array}{*{20}l} & \sum_{t \in T\setminus{t_{0}}} f_{l,t} \leq \theta_{l}\qquad\qquad\qquad \forall l \in L \end{array} $$
(9)

with the set of constraints (8) and (9) we ensure that the expected number of activations of the electrical load l is limited by a constant θl.

Photovoltaic non intrusive monitoring

In presence of PV installations, their power production can be absorbed by local electrical loads in what is commonly referred as self-consumption. In such cases, the metering infrastructure records the net power consumption only. While PV installations may have a dedicated monitoring infrastructure and the load power profile can be easily computed, more and more installations are not monitored as the PV production is no more subsidized in many countries. This is the case of the PV installation in the LIC area. Figure 2 illustrates the case where some loads activate during the period when PV is producing power.

Fig. 2
figure2

Meter power profile in presence of a non-monitored PV installation

When the PV installations are not monitored, the methodologies for NILM must be enriched in order to account for PV production. One possibility is to enlarge the set of uncontrollable load profiles S with power profiles accounting for PV production, i.e. power profiles with negative values corresponding to the estimated PV production. This approach has been adopted in (Sossan et al. 2018; Nespoli and Medici 2017).

Another approach is to exploit global horizontal irradiance (GHI) data coming from weather services and estimate the PV power production based on each installation’s nominal data (nominal ac and dc power). In order to do so, software libraries are normally used. One popular software library is PvLib, a community supported tool that provides a set of functions and classes for simulating the performance of photovoltaic energy systems (Holmgren et al. 2018). This latter approach is usually more accurate than the first one in case weather and nominal data are accurate. In our case, accurate weather data is not available. Therefore we reconstruct an approximation of the global irradiance data exploiting the metering data of all installations that are close to one another. The rationale behind this approach is that not all major loads are absorbing power simultaneously and the effect of PV installations is directly visible in the metering data.

Let H be the set of metering infrastructures reading neighbourhood households equipped with PV installations, pt,h identifies the power reading at time tT of household hH. Let GCt be the global irradiance in clear sky conditions at time tT, computed using PvLib. Let \(\hat {p}_{t,h}\) be an upper bound on the power production of the PV installation of household h computed with PvLib using the known nominal data of the installation and the global irradiance with clear sky conditions and rt,h be the ratio between the upper bound and the negative portion of the power reading limited by an upper bound \(\hat {r}\) (equal to 1.1 in our tests to allow for some additional freedom) and \(\hat {r}_{t}\) be the maximum of such ratios for tT:

$$\begin{array}{*{20}l} & r_{t,h} = \min\{\hat{r}, \min\{0,p_{t,h}\}/\hat{p}_{t,h}\}\qquad\qquad\qquad \forall t \in T, h \in H \end{array} $$
(10)
$$\begin{array}{*{20}l} & r_{t} = \max_{h \in H} \{r_{t,h}\}\qquad\qquad\qquad \forall t \in T \end{array} $$
(11)

Using rt we approximate the global irradiance \(\tilde {I}_{t}\) as a fraction of the clear sky global irradiance GCt and estimate the PV installation power production \(\tilde {p}_{t,h}\) of household h with PvLib.

$$\begin{array}{*{20}l} & \tilde{I}_{t} = GC_{t} \cdot r_{t}\qquad\qquad\qquad \forall t \in T \end{array} $$
(12)

Numerical experiments

The present section is organized in two subsections. We first present the computational results related to the estimation of the global horizontal irradiance. Then, we present the computational results related to the NILM methodology applied to synthetic data.

Estimation of global irradiance

Five photovoltaic installations are present in the LIC area. The peak powers of the installations are 30.0, 14.0, 10.0, 10.0 and 9.0 kW for a total of 73 kWp.

The rooftop of the Kindergarten, see Fig. 1b, is also equipped with a pyranometer for measuring solar irradiance gathering data with one minute resolution. We use this data for validation purposes only. Figure 3 shows the global irradiance for a period of two months.

Fig. 3
figure3

GHI measured on the Kindergarten’s rooftop, in W/m2

We gather real data from the LIC area for the period comprised from 04.04.2020 to 03.06.2020. We collect one data point per minute. For space reasons we summarize results as weekly averages (daily results are available contacting the authors). Table 2 reports the test weeks and the average daily measured GHI in the first two columns. Then the remaining columns report on the Weekly Average Root Mean Square Error (RMSE) and the Weekly Average Mean Absolute Percentage Error (MAPE) of the estimated GHI with respect to the measured GHI. In our tests, we use time discretization of the set T of 5 minutes leading to |T|=288 time steps. Columns marked with >0 and >AVG report values when the time steps are filtered by considering those with measured GHI larger than 0 and larger than the daily average, respectively. Indeed, several time steps are characterized with a measured GHI = 0 (during night time) and the GHI is easily estimated. This is confirmed by results in Table 2 observing that the values in the column RMSE(>0) are always larger than those in column RMSE. Instead, by definition, MAPE is computed on time steps where the measured GHI is strictly positive. When observing time steps where the measured GHI is larger than the daily average, that is on time steps which should correspond to large PV power production, we do not observe a clear trade off in the quality of the estimation. Considering daily results, we report that MAPE(>AVG) varies from 4.6% to 41.9%. The plot of these two cases are shown in pictures Fig. 4a and b. In Fig. 4a we can appreciate the correct estimation of a sudden drop of the GHI around 9AM. We see that the worst cases occur when the daily avg GHI is very low (cloudy days) and the PV production is indeed less relevant. If we limit the investigation to sunny or scattered days (e.g., with GHI >100 W/m2) we observe that the worst MAPE is 21.2% (see Fig. 5).

Fig. 4
figure4

Measured and estimated GHI

Fig. 5
figure5

Measured and estimated GHI on 29.05.2020, MAPE = 21.2%

Table 2 Weekly average RMSE and MAPE comparing estimated GHI and measured GHI, from 04.04.2020 to 03.06.2020

While the estimation quality may seem particularly bad on some days, the outcome of PV production estimation is sufficient for the purposes of non intrusive load monitoring.

Non intrusive load monitoring

For non intrusive load monitoring, we tested the MIQP formulation approach with the help of a comprehensive low voltage simulation framework called OPTISIM and developed within SUPSI. The simulation framework assembles several state-of-the-art IT components and integrates a power flow solver (Rosato et al. 2018), in order to carry out the electrical calculations. For the simulation of the grid-interfaced agents driven by the control algorithms, OPTISIM simulates physical models of several classes of components, namely: Building thermal dynamics, Heat distribution systems, Heating logic, Heat pumps, Water tanks and boilers, PV plants, Batteries, EV chargers. Profiles of passive, uncontrolled loads and domestic hot water consumption were generated using an artificial load profile generatorFootnote 2. We remark that the uncontrolled load profiles used in OPTISIM are not used as a basis for set S.

The experimental scenario is modeled to simulate the LIC area. While the area is composed of 18 households, 13 of them are equipped with 12 Domestic Hot water (DHW) heaters and 12 Heatpumps (HP). PV installations are similar to those present in the LIC area. The OPTISIM simulator provides both the aggregated power reading as well as the separated power profiles that we use for validation. A time resolution of 1 min is used for simulation. Recorded meteo data (GHI, Temperature and Wind speed) over one calendar year in 2018 are used by OPTISIM to simulate heat needs and PV production.

The MIQP formulation is solved with Gurobi 9.0 on a standard desktop machine. We used two settings for the time-step discretization. The first setting uses a time resolution of one minute, that is |T|=1440, therefore compatible with the time resolution of OPTISIM. In this case, the model is solved to optimality with an average computational time of 20s seconds. The second setting uses a time resolution of 5 min by averaging the power reading provided by OPTISIM. In this case |T|=288 and the average computational time drops to 2s to reach optimality.

In Table 3 we compare the quality of NILM for the two time resolution over 20 consecutive days in January. The table reports the considered loads (the number in the load’s id refers to the relative household), the total energy consumption over the period and the fraction of time the load was absorbing power during the period. Then we report the RMSE error and the RMSE error restricted to samples where the load was actually absorbing power. Then we report in column ERROR the percentage error, that is the fraction of times the status of the load was wrongly estimated.

Table 3 Non intrusive load monitoring - comparing 1 and 5 minutes resolution

In January power consumption is relatively high, in particular heat pumps contribute to the energy consumption by a large amount and they are up and running for up to 29% of the time.

We observe that head pumps are correctly monitored with very small errors. This is mostly due to the fact that heat pumps exhibit a clear footprint in reactive power, which is not the case for resistive loads. When aggregating to 5 min resolution, estimation errors start to rise up to 21% for one of the DHW loads. Even with lower resolution heat pumps are monitored with minor errors, up to 6.8%. We observe that for 1 min resolution, the RMSE is larger than RMSE >0. That is, NILM correctly estimates activation periods and sometimes produces false positives. This behaviour is less critical for DSM applications as false positives constitute an overestimation of the power needs. Instead, for 5 min resolution, the RMSE is smaller than RMSE >0. That means that sometimes load activation is underestimated and that is more critical for DSM applications. This effect is due to aggregation when, for example, loads are active for a short amount of time (about 5 min) and the activation occur across half of the time interval.

In Table 4 we compare the quality of NILM for 20 days in two different periods of the year with time resolution set to 1 minute. For the sake of compactness, in this table we report the ERROR column only.

Table 4 Non intrusive load monitoring - spring and summer periods

We observe that energy consumption decreases significantly as the weather gets warmer. Heat pumps are still monitored with very high accuracy, while some major errors are seen on DHW.

In Table 5 we compare the quality of NILM for 20 days in the same period of that reported in Table 3 comparing the effect of limiting the estimated activation of DHW loads, i.e. the introduction of constraints (8) and (9). In our tests we set θl=12 that is, we expect the activation of DHW load to occur up to every two hours in average.

Table 5 Non intrusive load monitoring - Limiting the number of estimated activations

We observe that quality of the estimation increases significantly for almost all DHW loads except load dhw_11. Anyway, setting the correct value of θl is not trivial and further research should be devoted to that aspect.

For the sake of illustration, Fig. 6 shows the application of NILM on household 06 on 03.01.2018. We observe the presence of a PV installation which is correctly estimated by the method described in subsection Photovoltaic non intrusive monitoring and the correct detection of the activation of both the DHW and the heat pump.

Fig. 6
figure6

NILM on HP_06 and DHW_06, on 03.01.2018

Conclusions

We address the problem of non-intrusive load monitoring in presence of distributed power generation (photovoltaic). The proposed approach, based on mathematical programming, showed to be effective for source separation with a low frequency data sampling. We proved that the estimation of the global irradiance exploiting a set of non-monitored photovoltaic installations in the same area is sufficient to discount the contribution of PV power generation to the aggregated signal and therefore obtain accurate source separation.

Availability of data and materials

Data used in this paper is available upon direct request.

Notes

  1. 1.

    https://www.bfe.admin.ch/bfe/en/home/policy.html

  2. 2.

    https://www.loadprofilegenerator.de

References

  1. Bu, F, Dehghanpour K, Yuan Y, Wang Z, Zhang Y (2020) A data-driven game-theoretic approach for behind-the-meter pv generation disaggregation. IEEE Trans Power Syst 35(4):3133–3144.

    Article  Google Scholar 

  2. Cominola, A, Giuliani M, Piga D, Castelletti A, Rizzoli AE (2017) A Hybrid Signature-based Iterative Disaggregation algorithm for Non-Intrusive Load Monitoring. Applied Energy 185(2017):331–344.

    Article  Google Scholar 

  3. Hart, GW (1992) Non-intrusive appliance load monitoring. Proc IEEE 80(12):1870–1891.

    Article  Google Scholar 

  4. Holmgren, WF, Hansen CW, Mikofski MA (2018) pvlib python: a python package for modeling solar energy systems. J Open Source Softw 3(29):884.

    Article  Google Scholar 

  5. Kong, W, Dong ZY, Jia Y, Hill DJ, Xu Y, Zhang Y (2019) Short-Term Residential Load Forecasting Based on LSTM Recurrent Neural Network. IEEE Trans Smart Grid 10(1):841–851.

    Article  Google Scholar 

  6. Nespoli, L, Medici V (2017) An unsupervised method for estimating the global horizontal irradiance from photovoltaic power measurements. Sol Energy 158:701–710.

    Article  Google Scholar 

  7. Piga, D, Cominola A, Giuliani M, Castelletti A, Rizzoli AE (2016) Sparse Optimization for Automated Energy End Use Disaggregation. IEEE Trans Cont Sys Tech 24(3):1044–1051.

    Article  Google Scholar 

  8. Rosato, F, Medici V, Rudel R (2018) Krangpower: a Smart Grid Simulation Package. https://krangpower.readthedocs.io. Accessed 4 Oct 2020.

  9. Rottondi, C, Derboni M, Piga D, Rizzoli AE (2019) An optimisation-based energy disaggregation algorithm for low frequency smart meter data. Energy Inform 2(S1):13.

    Article  Google Scholar 

  10. Schirmer, PA, Mporas I, Sheikh-Akbari A (2020) Energy Disaggregation Using Two-Stage Fusion of Binary Device Detectors. Energies 13(9):2148.

    Article  Google Scholar 

  11. Sossan, F, Nespoli L, Medici V, Paolone M (2018) Unsupervised disaggregation of photovoltaic production from composite power flow measurements of heterogeneous prosumers. IEEE Trans Ind Inform 14(9):3904–3913.

    Article  Google Scholar 

  12. Zeifman, M, Roth K (2011) Nonintrusive Appliance Load Monitoring: Review and Outlook. IEEE Trans Consum Electron 57(1):76–84.

    Article  Google Scholar 

  13. Zoha, A, Gluhak A, Imran M, Rajasegarar S (2012) Non-Intrusive Load Monitoring Approaches for Disaggregated Energy Sensing: A Survey. Sensors 12(12):16838–16866.

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the active collaboration and support of Azienda Elettrica di Massagno SA (Ticino).

About this supplement

This article has been published as part of Energy Informatics Volume 3 Supplement 1, 2020: Proceedings of the 9th DACH+ Conference on Energy Informatics. The full contents of the supplement are available online at https://energyinformatics.springeropen.com/articles/supplements/volume-3-supplement-1.

Funding

Publication costs were covered by the DACH+ Energy Informatics Conference Organizers, supported by the Swiss Federal Office of Energy. This research was carried out within the frame of the Swiss Centre for Competence in Energy Research on the Future Swiss Electrical Infrastructure (SCCER-FURIES) with the financial support of the Swiss Innovation Agency Innosuisse (SCCER program and project CTI 26756.1), the Swiss Federal Office of Energy (project SI/501840-01) and the Renewable Energy Fund of Canton Ticino.

Author information

Affiliations

Authors

Contributions

Matteo Salani is the main contributor of this paper, his contributions are related to the methodologies for PV estimation and NILM, the presentation and test/validation. Marco Derboni contributed to software implementation and testing. Andrea E. Rizzoli contributed to the presentation of the results and acted as PI in the Innosuisse project. Davide Rivola is the PI of the BFOE project and is responsible for the LIC area. Vasco Medici, Lorenzo Nespoli and Federico Rosato are the authors of the OPTISIM simulation package which has been used for testing purposes. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Matteo Salani.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Salani, M., Derboni, M., Rivola, D. et al. Non intrusive load monitoring for demand side management. Energy Inform 3, 25 (2020). https://doi.org/10.1186/s42162-020-00128-2

Download citation

Keywords

  • Smart Grid
  • Load Monitoring
  • Demand Side Management