2
$\begingroup$

I have usage data of many machines like this:

machine usage

In this machine, the usage grows linear with the time. However, in early 2016, someone or something messed with the total usage in hours and the next measurement dropped down to about 2500. From there, the slope is equal to what it was before.

I'd like to automatically find these changes and correct them, i.e. pull the left "half" of the data down, or the right "half" of the data up, so that the data follows a line, as straight as possible. Are there well-known methods for this?

So far I've found the keyword 'structural change' and the R package strucchange, but that seems a bit too complicated for my case. I essentially have two linear models with the same slope of $\beta$, but different intercepts.

$\endgroup$
1
  • $\begingroup$ It isn't clear what these data mean (how were they measured, how were multiple results summarized, why don't you show the individual measurements, etc) or what such a "correction" is intended to accomplish. Could you elaborate on those issues? $\endgroup$ – whuber Jun 2 '17 at 14:43
1
$\begingroup$

If the data is as small and as simple as shown in your graphic, you can probably also use any ad hoc approach for finding the timing of the drop and its size. If you want to use strucchange it might be a bit "high-tech" but should also get the job done.

For illustration I use the following "regular" ts time series. Your data does not seem to be regular as you have plotted it but maybe you can easily get usage per month or per quarter (as in my example below). This might facilitate the analysis. My data has a slope of 3000 and a drop by 2000 in early 2016:

set.seed(1)
time <- seq(2015, 2017.5, by = 0.25)
hours <- 0 + 3000 * (time - 2015) - 2000 * (time > 2016.125) +
  rnorm(length(time), sd = 40)
usage <- ts(cbind(hours, time), start = 2015, freq = 4)

Visualizing either the total hours or their diffs/growth gives:

plot(usage[,"hours"], type = "o", main = "total")
plot(diff(usage[,"hours"]), type = "o", main = "diff")

example data

A very simple naive approach could be look at the difference between a robust location estimate of the growth (e.g., the median) and the minimum:

median(diff(usage[, "hours"])) - min(diff(usage[, "hours"]))
## [1] 2047.766

Note that if your data is not regular/equidistant then you need to look at hours diffs per time diffs.

If you want to fit the breakpoints in a linear regression model you can easily do so with breakpoints() from strucchange. Here, I set the minimum segment size to 3 and the maximum number of breaks to 1 because this is what we are looking for:

library("strucchange")
bp <- breakpoints(hours ~ I(time - 2015), data = usage, h = 3, breaks = 1)
breakdates(bp)
## [1] 2016
coef(bp)
##                   (Intercept) I(time - 2015)
## 2015(1) - 2016(1)   -21.41769       3053.177
## 2016(2) - 2017(3) -2063.59053       3041.693

This does find a single breakpoint in 2016, i.e., the change occurs between 2016(1) and 2016(2). Note that breakpoints() also allows the slopes to be different. The estimate for the drop is the difference of the intercepts:

coef(bp)[1,1] - coef(bp)[2,1]
[1] 2042.173

If you want to force the slopes to be identical you can also re-estimate such a single-slope model with lm():

m <- lm(hours ~ breakfactor(bp) + I(time - 2015), data = usage)
coef(m)
##             (Intercept) breakfactor(bp)segment2          I(time - 2015) 
##               -17.76357             -2053.65723              3045.86872 

So you see that the three methods lead to slightly different estimates of the drop but are very similar.

Comments:

  1. If your data are not a regular time series than the nice labeling of the time spans and the breakdates() will not work. But everything else should be unchanged.

  2. From your figure the growth does not seem to be completely linear but possibly exponential. In that case using log(hours) rather than hours would be the natural solution.

$\endgroup$
2
  • $\begingroup$ Thanks a lot, this helps! But in this way, the function breakpoints() will also find changes in the slope, right? Is there a way to have it only recognize intercept changes? $\endgroup$ – Alexander Engelhardt Jun 4 '17 at 10:48
  • $\begingroup$ The breakpoints are always estimated with all parameters potentially differing between the segments. But as shown in my reply you can easily estimate a partially constrained model given the breakpoints. The unconstrained breakpoint estimates will still be consistent. $\endgroup$ – Achim Zeileis Jun 4 '17 at 13:58
0
$\begingroup$

The problem with the structural change program is that it assumes that you know how many "lines" or connected points you wish to fit and even more importantly it begs the question of auto-correlation and finally it assumes no pulses or seasonal pulses e.g. a july effect. I "roughly eyeballed your data" and came up with 11 observations. My pseud0-data suggests am ARIMA model (1,1,0) containing first differences and two pulses effectively dealing with an intercept change not a slope change.enter image description here and enter image description here . Note the confidence limits reflect that the model residuals were not normal and outliers/pulses are considered possible in the future. In closing I was a developer for the forecasting package used to automatically perform this tour de force.

Your "simple problem" looking for a "simple solution" that works in all cases is not so simple and there are a lot of bad solutions looking for a place to roost. The "problem" with struct change is not that it is too complex it is too simple.

@whuber

I see (nearly) equal steps (intervals) between the 11 points. Since I am not comprehending your precise meaning ...can you elaborate and I as usual will try to answer your reflection.

enter image description here

$\endgroup$
6
  • $\begingroup$ Could you explain why any standard time series model and fitting procedure, as shown here, would be plausible for or applicable to data that have such extreme variation in distances between observed times? $\endgroup$ – whuber Jun 2 '17 at 14:41
  • $\begingroup$ I am not sure what you mean by "extreme variation" ... It appears to me there are 11 observations equally spaced . Can you be more specific ? Are you referring to the width of the forecasting limits ? $\endgroup$ – IrishStat Jun 2 '17 at 16:11
  • $\begingroup$ How do you get equal spacing from the plot in the question? Some points are so close they must be separated by a few days at most, while others are separated by almost a year. Your plot and the plot in the question don't seem to have anything at all in common. $\endgroup$ – whuber Jun 2 '17 at 17:42
  • 1
    $\begingroup$ apparently I used the pix that was presented by the other responder.. You are right that the op had VERY unequal points. Hopefilly the the OP had more readings but perhaps didn't take the time to report them. $\endgroup$ – IrishStat Jun 2 '17 at 18:13
  • $\begingroup$ My reply explains the reasoning for using equidistant points: I assumed these would be easy to obtain by aggregating monthly or quarterly usage hours. As for the time series modeling: Of course, in a structural change model you can also include seasonal dummies or autoregressive effect and, of course, these will have big effects on out-of-sample predictions. I did not include them because the OP asked for two separate linear regressions. Moreover, here the focus was just automatic estimation of the drop and not prediction. $\endgroup$ – Achim Zeileis Jun 3 '17 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.