I would like to be able to predict when I will exhaust a particular resource.
My situation is analogous* to a water tank. Each day zero or more rain will fall, filling up the tank. I can not tell in advance how much rain will fall each day, if any at all. There will never be any outflow from the tank. Each day I can measure how much is in the tank and I know that tank's capacity. Since there is a cost to having spare tanks on hand, just in case, and also to filling a tank when no spare is available, I'd like to use my daily measurements to predict when the tank will be full.
Experience so far suggests the tank has about four year's capacity. It will take us something like three to six months to get a new tanks set up and working depending on what else is going on. It is something we want to plan for in advance. Of course emergency tanks can be rushed through if a deluge is imminent, but the cost is huge and this should be avoided.
I've calculated a linear regression of the daily water levels. This gives me a slope and intercept from which I can calculate the date when the trend line will cross the tank's capacity. This gives me no indication of the faith I can place in this number, however. Is it legitimate to simply use the Confidence Interval formula to similarly calculate when the CI curve crosses the capacity? Is there a better way of achieving this?
The data will look like this (invented values):
*The actual case is a data warehouse computer system. The rows have an auto-incrementing column. As a new row is written the value in this column goes up. It never goes down. There is no equivalent of "draining" the water-tank analogy. If I measure the largest value in this column today, and no rows are added overnight, I will get the same vlaue when I ask tomorrow. If rows were written overnight, tomorrow's measurement will be larger than today's. Even if all rows are removed the auto-increment will not reset to zero.
The column is defined as an integer which has a maximum value of just over two billion. If we reach this limit we cannot add rows to the table, we have system down time, my boss's head explodes, the oceans turn to blood etc. We don't want that to happen. The disks storing this data cost tens of thousands of dollars, however, so we don't want to have the capital outlay any earlier than we have to, obviously. The definition of "have to" is the thrust of my question.
Appendix: My results
I have been able to re-create what my data would have been for the years 2012/13 and 2013/14. One typical sample set looks like this.
Blue and red are the two years' data. Black and dashed are the least-squares regression for each year.
I used 2012/13's data to predict the value at the end of 2013/14. The trend line through the first year is a little steeper than that through the second so I would expect my answer to be a little earlier than what actually occurred. I used three techniques:
- Simple geometry. Take the first and last point of the sample. Draw a straight line through them. Calculate the date this line cuts the upper limit.
- Calculate a line's parameters from a least squares regression. Use these parameters to calculate the date this line cuts the upper limit.
- Use 2012/13's data as the sample set for a Monte Carlo simulation.
Simple geometry was very fast, easy to code and understand.
Least squares was difficult to get right but runs quite quickly. The result is closer to the "real" value, but not by much.
Monte Carlo is easy to code and maintain. It takes a long time to run, however. I take the median-crossing date as the date of resource exhaustion (where green crosses red in this chart):
All three technique produced an answer within three weeks of the "actual" result, which is good enough for my purposes.
In terms of the original question, Monte Carlo is the only technique that indicates how likely I am to run out of resource sooner than predicted. Because it takes so long to execute I will be adopting a hybrid approach. I will run the simple geometric calculation each day as data arrives. If this indicates an exhaustion date which would be of concern (less than one year in the future) I will run a Monte Carlo to see how bad the situation is.