2
$\begingroup$

Suppose you have several years of monthly consumption (kWh) data for 500,000 electrical meters and your job is to look for outlier behavior of various types. How would you approach modeling the meters with an ideal goal of giving each meter an outlier score? If for each meter you had other data (factor and/or numeric) in addition to consumption, how would this change your approach?

$\endgroup$
2
  • $\begingroup$ Please tell us what you mean by "outlier:" for instance, would it be a single monthly reading, a specific meter, a specific month, or something else? $\endgroup$
    – whuber
    Apr 2, 2020 at 13:32
  • $\begingroup$ An outlier would be something along the lines of the meter completely fails and records no consumption or demand at the meter or a meter is tampered with and starts reporting a fraction of actual consumption. Outliers historically are either 0 reads or reads that suddenly drop by 1/3, 1/2, or 2/3 $\endgroup$
    – SPJ
    Apr 2, 2020 at 18:40

1 Answer 1

3
$\begingroup$

I would use Intervention Detection as discussed here http://docplayer.net/12080848-Outliers-level-shifts-and-variance-changes-in-time-series.html . There are 4 types of interventions ; Pulse , step/level , seasonal pulse and local time trend (I) . This approach can be used with or without user-suggested factors (X). The General Model is a SARMAX model https://autobox.com/pdfs/SARMAX.pdf . Note that the X's can impact The Y both contemporaneously and in a lag fashion. The historical effect of previous Y values is called the arima structure. Model building would be used using available automatic software that simultaneously identified the structure following iterative self-validating heuristics described here https://autobox.com/pdfs/ARIMA%20FLOW%20CHART.pdf and elsewhere.

One could then rank the meters by the frequency of the different kinds of "outliers" .

EDITED after OP asked for some more details about incorporating the effects of user-suggested causals the X's . https://autobox.com/pdfs/A.pdf lays out the flow whle How to use Dynamic Regression models in R to forecast future sales might also be of help and Tsay's intro to Transfer Function (SARMAX) identification here http://www.math.cts.nthu.edu.tw/download.php?filename=569_fe0ff1a2.pdf&dir=publish&title=Ruey+S.+Tsay-Lec1 . As a broad overview of causal modelling I wrote this piece to contrast regression with Transfer Function (SARIMAX) modelling https://autobox.com/pdfs/regvsbox-old.pdf.

Another reference that I had presented at a conference is also educational as to the whys and wherefores of causal model identification http://www.autobox.com/pdfs/WHY-WE-FILTER.ppt

$\endgroup$
8
  • 2
    $\begingroup$ The term "outlier" is vague while pulse is more general … consider a series 1,9,1,9,1,9,1,9,5,9,1,9 ..the value 5 is a pulse and could be called an inlier but it is as unusual as 1,9,1,9,1,9,1,9,14,9,1,9 the value 14 , I prefer the general phrase term "Interventions" with pulses, leve/step shifts , seasonal pulses and local time trends being the 4 classes. Note that a pulse is a one time change in the model-implied intercept while a step/level shift is a permanent change in the model-implied intercept. $\endgroup$
    – IrishStat
    Apr 2, 2020 at 15:32
  • 1
    $\begingroup$ The suggestion that there can be supporting variables such as weather, etc is made clear as they are possible X's in the SARMAX equation. Their particular effect (coefficients)would be different for each and every meter $\endgroup$
    – IrishStat
    Apr 2, 2020 at 15:40
  • 1
    $\begingroup$ I will add another flow diagram that should help and some web references. $\endgroup$
    – IrishStat
    Apr 2, 2020 at 19:10
  • 1
    $\begingroup$ not a problem at all if you have the right software and the right computer . Model revision can be done systematically .. a portion at a time … models can be reused until they get updated .. Anheuser-Busch asked me to help implement a project to predict daily sales for 50 beer products for 700,000 stores using weather and price as possible contemporary and lagged predictors for 700,000 retail outlets. This was necessary to make sure the shelfs were stacked with the right product at the right time $\endgroup$
    – IrishStat
    Apr 2, 2020 at 20:26
  • 1
    $\begingroup$ actuarial examples of forecasting at this scale include monitoring of financial accounts to detect "exceptional activity " and obviously sales of different classes of items at multiple stores. $\endgroup$
    – IrishStat
    Apr 2, 2020 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.