Faith in an extrapolated result

Question

I would like to be able to predict when I will exhaust a particular resource.

My situation is analogous* to a water tank. Each day zero or more rain will fall, filling up the tank. I can not tell in advance how much rain will fall each day, if any at all. There will never be any outflow from the tank. Each day I can measure how much is in the tank and I know that tank's capacity. Since there is a cost to having spare tanks on hand, just in case, and also to filling a tank when no spare is available, I'd like to use my daily measurements to predict when the tank will be full.

Experience so far suggests the tank has about four year's capacity. It will take us something like three to six months to get a new tanks set up and working depending on what else is going on. It is something we want to plan for in advance. Of course emergency tanks can be rushed through if a deluge is imminent, but the cost is huge and this should be avoided.

I've calculated a linear regression of the daily water levels. This gives me a slope and intercept from which I can calculate the date when the trend line will cross the tank's capacity. This gives me no indication of the faith I can place in this number, however. Is it legitimate to simply use the Confidence Interval formula to similarly calculate when the CI curve crosses the capacity? Is there a better way of achieving this?

The data will look like this (invented values):

---

*The actual case is a data warehouse computer system. The rows have an auto-incrementing column. As a new row is written the value in this column goes up. It never goes down. There is no equivalent of "draining" the water-tank analogy. If I measure the largest value in this column today, and no rows are added overnight, I will get the same vlaue when I ask tomorrow. If rows were written overnight, tomorrow's measurement will be larger than today's. Even if all rows are removed the auto-increment will not reset to zero.

The column is defined as an integer which has a maximum value of just over two billion. If we reach this limit we cannot add rows to the table, we have system down time, my boss's head explodes, the oceans turn to blood etc. We don't want that to happen. The disks storing this data cost tens of thousands of dollars, however, so we don't want to have the capital outlay any earlier than we have to, obviously. The definition of "have to" is the thrust of my question.

---

Appendix: My results

I have been able to re-create what my data would have been for the years 2012/13 and 2013/14. One typical sample set looks like this.

Blue and red are the two years' data. Black and dashed are the least-squares regression for each year.

I used 2012/13's data to predict the value at the end of 2013/14. The trend line through the first year is a little steeper than that through the second so I would expect my answer to be a little earlier than what actually occurred. I used three techniques:

1. Simple geometry. Take the first and last point of the sample. Draw a straight line through them. Calculate the date this line cuts the upper limit.
2. Calculate a line's parameters from a least squares regression. Use these parameters to calculate the date this line cuts the upper limit.
3. Use 2012/13's data as the sample set for a Monte Carlo simulation.

Simple geometry was very fast, easy to code and understand.

Least squares was difficult to get right but runs quite quickly. The result is closer to the "real" value, but not by much.

Monte Carlo is easy to code and maintain. It takes a long time to run, however. I take the median-crossing date as the date of resource exhaustion (where green crosses red in this chart):

All three technique produced an answer within three weeks of the "actual" result, which is good enough for my purposes.

In terms of the original question, Monte Carlo is the only technique that indicates how likely I am to run out of resource sooner than predicted. Because it takes so long to execute I will be adopting a hybrid approach. I will run the simple geometric calculation each day as data arrives. If this indicates an exhaustion date which would be of concern (less than one year in the future) I will run a Monte Carlo to see how bad the situation is.

Answer 1

Answering your edited question. Take the simplest regression possible:
$\text{Netflow} \sim\alpha + \epsilon$
Intercept only. The $\alpha$ will be your average daily netflow.

Now, let's say that you want to know when will the data storage fill up? Simulate. Data storage will follow this formula:
$Y_{t+1} \sim Y_t + \alpha + \epsilon_t$
You can simulate a path that your data storage will follow by sampling a bunch of $\epsilon$ (either assuming they are normal or resampling bootstrap-like from the residuals).

Simulate a 1000 paths. Each of them will have a simulated day of when the storage is full. Use this distribution of days to manage the overfilling risk. You can also update this distibution daily by redoing regression and simulation as more data comes up.

Answer 2

To expand on my comments, a typical simple "level of dam" or "level of tank" type model might produce results that look vaguely something like this, say:

It's not really appropriate to fit a linear regression to this kind of process, for a variety of reasons (not least, dependence and barrier effects). You need a model that more reasonably describes the process, and will measure the uncertanties appropriately.

I wouldn't have much faith in the predictions of a model that really doesn't describe what's going on. With a good model, you can at least use simulation to get some idea of when the tank may be full or unlikely to be full.

---

Stochastic models of this kind are included under stochastic fluid models, what used to be called 'dam models', but they're also used in computing-related applications and ruin theory in insurance. The books on ruin theory that don't necessarily make much reference to the other literature. In ruin theory the process is "flipped" relative to a dam-type model - increases are continuous and close to linear over time (premiums), and the decreases are jumps (claims); for dam models the jumps are increases and decreases more continuous.

There are more than 40 references at the wikipedia page linked above.

Books on stochastic processes (of which there are many) often have some basic models of this type.

More sophisticated models take into account more of the details, like dependence in increases and decreases and variation over time; the more complex models aren't usually nice algebraically and generally have to be simulated.