I have a family of time series, each of which has a fixed deadline. My goal is to predict the final value (and, ideally, a confidence interval) about 30 days before the deadline.

In order to describe this without too much jargon, you can assume that I put out dozens of urns. Each urn is very prominently marked with its expiration date; it also has a color and a label. The process that I'm trying to model places marbles in the urns until either the urn is full or the urn expires. The process is aware of the deadline; it does not know the contents of or the capacity of the urn. Once most of my urns are either full or expired, I put out another batch. I've done this with well over 1000 urns, and I have the timestamps of every time a marble was added (as well as lots of data describing each urn---most of which is probably useless).

I strongly believe that when there are multiple urns with the same label, they interact with each other; the process seems more likely to put marbles in the one with the earliest deadline.

My data all look roughly exponential, and a typical data set describing the number of marbles in an urn at the end of each day would be something like: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 9, 9, 10, 10, 10, 10, 10, 11, 11, 11, 12, 12, 13, 13, 15, 18, 18, 19, 22, 23, 24, 25, 26, 26, 26, 26, 27, 28, 29, 29, 32, 34, 35, 35, 37, 37, 38, 39, 39, 40, 41, 44, 44, 48, 49, 50, 53, 57.

In this particular instance, my hope would be to take the observed values up to the time where this urn has 22 marbles together with what I've learned from the 1000 previous urns in order to predict that this urn would have, say, between 42 and 72 marbles on its expiration day.

I've tried fitting this as a pure exponential model, and while the KS-test says that it's a good fit, the parameter from the model wasn't good enough at predicting real-world behavior. I've also used exponential smoothing with a multiplicative trend; this generally under-estimated the final value. I got slightly better results (but not good enough) by truncating the leading tail of low values so that the moving average wouldn't be damped too much by the slow start.

What's a good way forecast the final value for such an exponential-seeming time series, given the beginning of the time series and a database full of information about similar processes that ran in the past?

Edit: Almost all of the urns are set out in a batch, but most of the deadlines are distinct. The data above really should have a leading trail of 0s, but I haven't been using that in my model because it hasn't helped so far, and it makes it annoying to take the log of the data.

At this point I would be happy with an answer that helps me get closer to a solution to the simplest cases. Exponential functions grow so fast that small errors turn into big errors very quickly. My intuition is that I'm going to need a strategy that takes the historical information about exponential growth rates and then damps it based on the current observations. Or possibly a linear model of the log of my data and then to include some of these other variables as a categorical predictor?

  • $\begingroup$ I suppose all the deadlines are different, but are the starting time the same for all the urns ? $\endgroup$ – AlainD Jul 18 '18 at 7:19
  • $\begingroup$ Thank you for your comment; I have edited the question. The forecasting literature that I'm familiar with is fairly silent about situations where the endpoint is the most important feature of the time series, so I appreciate any ideas! $\endgroup$ – Andrzej W. Jul 18 '18 at 17:29
  • $\begingroup$ If you have one urn, is its growth linear or exponential ? $\endgroup$ – AlainD Jul 19 '18 at 15:24
  • $\begingroup$ The more the process goes, the less urns remain alive. So the marbles will be distributed over a shrinking number of urns. Therefore the growth of these remaining urn will increase. The exponential shape may simply be a consequence of this fact. You can test it by correlating the growth with the number of urn still alive. $\endgroup$ – AlainD Jul 19 '18 at 15:32
  • $\begingroup$ When I have one urn, it is definitely exponential. I'm assuming that this is some sort of Poisson process. Contrary to your assumption, the data show that the first urn in the sequence tends to get the most marbles and the later urns don't get as many. This suggests a fixed supply of marbles in the "black box" in charge of marble-distribution, but I have no direct access to the "black box." $\endgroup$ – Andrzej W. Jul 19 '18 at 16:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.