Full disclosure: I am not a statistician, nor do I claim to be one. I am a lowly IT administrator. Please play gentle with me. :)

I am responsible for collecting and forecasting disk storage use for our enterprise. We collect our storage use monthly and use a simple rolling twelve month linear regression for forecasts (in other words, only the previous twelve months of data are considered when making a projection). We use this information for allocation and capital expense planning, e.g. "Based on this model, we will need to purchase x amount if storage in y months to meet our needs." This all works well enough to suit our needs.

Periodically, we have large one-time movements in our numbers that throws the forecasting off. For example, someone finds 500GB of old backups that aren't needed anymore and deletes them. Good for them for reclaiming the space! However our forecasts are now skewed way off by this large drop in one month. We have always just accepted that a drop like this takes 9-10 months to make its way out of the models, but that can be a really long time if we are entering capital expense planning season.

I'm wondering if there is a way to handle these one-time variances such that the forecasted values aren't impacted as much (e.g. the slope of the line doesn't change as dramatically), but they are taken into account (e.g. a one-time change in the y-value associated with a particular point in time). Our first attempts at tackling this have yielded some ugly results (e.g. exponential growth curves). We do all of our processing in SQL Server if that matters.

  • $\begingroup$ Excellent question. Just a quick clarification. Would you like to predict these events, or, once they happen, adjust the model predictions given your new information? $\endgroup$ – Matthew Drury Feb 11 '16 at 20:38
  • 2
    $\begingroup$ Right, it's not clear if you are trying to "smooth" out these rare events like the 500GB example so they don't impact your results as much or if you are trying to account more them since you want to capture when bitch adjustments to store are made? The difference is subtle: In the first, you want to almost ignore the new point (rare event), but int he second, you want to emphasize the point (rare event). If it's the former, robust regression is probably a simple method for you since you are already using linear regression. See here: ats.ucla.edu/stat/r/dae/rreg.htm $\endgroup$ – StatsStudent Feb 11 '16 at 20:45
  • $\begingroup$ Also, do you use any software to make your predictions and do you use confidence intervals? $\endgroup$ – StatsStudent Feb 11 '16 at 20:48
  • $\begingroup$ I can add an adjustment after the fact. In fact, most of the time I won't know about a major deviation until I look at the next month's numbers and see a big change. I'm not using any software to make the predictions; just a stored procedure in SQL Server to calculate my regression values. $\endgroup$ – sbrown Feb 11 '16 at 22:12
  • $\begingroup$ Quick reactions: (a) I'd probably first fit a very basic AR(1) to changes in log disk usage? You''d basically be estimating some long term growth rate in disk usage, and how quickly the growth rate in disk usage moves back to that trend after a shock. (aa) You could use other data too and fit a VAR (vector autoregression). (b) throwing out all data > 12 months may not make be the optimal thing to do. (c) regular OLS minimizes the sum of squares. You could use a different penalty function (eg. Huber) which is more robust to outliers. $\endgroup$ – Matthew Gunn Feb 11 '16 at 23:06

Here's a simple suggestion. I don't know whether it works for you and maybe I should have made it as a comment, but it seems you need more privileges to make a comment than to make a reply.

If I understand correctly, the figures you are using are the amounts of storage you are using each month. Probably these usualy increase, and you want to predict what the amount will be at some time in the future if trends continue. Once you realise that your big change has happened (e.g. that 500 GB has been released) can you go back and change the previous months' figures (e.g. delete 500 GB from all of them)? Basically what you would be doing is to adjust the previous months' figures to what they should have been, if you knew then what you know now.

Of course I don't recommend this unless you make sure you can go back to the old figures. But the forecasting you want to do sounds like it could even be done in Excel, in which case you can have as many versions as you want.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.