33
$\begingroup$

I have a question related to modeling short time-series. It is not a question if to model them, but how. What method would you recommend for modeling (very) short time-series (say of length $T \leq 20$)? By "best" I mean here the most robust one, that is the least prone to errors due the fact of limited numbers of observations. With short series single observations could influence the forecast, so the method should provide a cautious estimate of errors and possible variability connected to the forecast. I am generally interested in univariate time-series but it would be also interesting to know about other methods.

$\endgroup$
  • $\begingroup$ What is the time unit? Can you post the data? $\endgroup$ – Dimitriy V. Masterov Jan 26 '15 at 21:09
  • 8
    $\begingroup$ Whatever assumptions you make - concerning seasonality, stationarity, &c. - a short time series will give you the chance to detect only the most flagrant violations; so assumptions should be well-founded in domain knowledge. Do you need to model or just to make forecasts? The M3 competition compared various "automatic" forecasting methods on series from a variety of domains, some as short as 20. $\endgroup$ – Scortchi Jan 26 '15 at 21:47
  • 5
    $\begingroup$ +1 to @Scortchi's comment. Incidentally, out of the 3,003 M3 series (available in the Mcomp package for R), 504 have 20 or fewer observations, specifically 55% of the yearly series. So you could look up the original publication and see what worked well for yearly data. Or even dig through the original forecasts submitted to the M3 competition, which are available in the Mcomp package (list M3Forecast). $\endgroup$ – Stephan Kolassa Jan 27 '15 at 12:13
  • $\begingroup$ Hi, I won't add anything to the answer, but just share something about the question which I hope it can help others understanding the problem here: when you say robust one, that is the least prone to errors due to the fact of limited number of observations . I believe robustness is an important concept in stats and here it's crucial since having so few data any modeling fit will strongly depend on the assumptions of the model itself or outliers. With robustness you make this constraints less strong, not allowing assumption to limit your results. I hope this helps. $\endgroup$ – Tommaso Guerrini Jun 19 '17 at 9:26
  • 2
    $\begingroup$ @TommasoGuerrini robust methods do not make less assumptions, they make different assumptions. $\endgroup$ – Tim Jun 19 '17 at 9:38
30
$\begingroup$

It is very common for extremely simple forecasting methods like "forecast the historical average" to outperform more complex methods. This is even more likely for short time series. Yes, in principle you can fit an ARIMA or even more complex model to 20 or fewer observations, but you will be rather likely to overfit and get very bad forecasts.

So: start with a simple benchmark, e.g.,

  • the historical mean
  • the historical median for added robustness
  • the random walk (forecast the last observation out)

Assess these on out-of-sample data. Compare any more complex model to these benchmarks. You may be surprised at seeing how hard it is to outperform these simple methods. In addition, compare the robustness of different methods to these simple ones, e.g., by not only assessing average accuracy out-of-sample, but also the error variance, using your favorite error measure.

Yes, as Rob Hyndman writes in his post that Aleksandr links to, out-of-sample testing is a problem in itself for short series - but there really is no good alternative. (Don't use in-sample fit, which is no guide to forecasting accuracy.) The AIC won't help you with the median and the random walk. However, you could use time-series cross-validation, which AIC approximates, anyway.

$\endgroup$
  • $\begingroup$ Just discovered your answer (+1). I made another comment, in case that you're interested and would like to clarify. $\endgroup$ – Aleksandr Blekh Jan 27 '15 at 11:52
17
$\begingroup$

I am again using a question as an opportunity to learn more about time series - one of the (many) topics of my interest. After a brief research, it seems to me that there exist several approaches to the problem of modeling short time series.

The first approach is to use standard/linear time series models (AR, MA, ARMA, etc.), but to pay attention to certain parameters, as described in this post [1] by Rob Hyndman, who does not need an introduction in time series and forecasting world. The second approach, referred to by most of the related literature that I have seen, suggest using non-linear time series models, in particular, the threshold models [2], which include threshold autoregressive model (TAR), self-exiting TAR (SETAR), threshold autoregressive moving average model (TARMA), and TARMAX model, which extends TAR model to exogenous time series. Excellent overviews of the non-linear time series models, including threshold models, can be found in this paper [3] and this paper [4].

Finally, another IMHO related research paper [5] describes an interesting approach, which is based on Volterra-Weiner representation of non-linear systems - see this [6] and this [7]. This approach is argued to be superior to other techniques in the context of short and noisy time series.

References

  1. Hyndman, R. (March 4, 2014). Fitting models to short time series. [Blog post]. Retrieved from http://robjhyndman.com/hyndsight/short-time-series
  2. Pennsylvania State University. (2015). Threshold models. [Online course materials]. STAT 510, Applied Time Series Analysis. Retrieved from https://onlinecourses.science.psu.edu/stat510/node/82
  3. Zivot, E. (2006). Non-linear time series models. [Class notes]. ECON 584, Time Series Econometrics. Washington University. Retrieved from http://faculty.washington.edu/ezivot/econ584/notes/nonlinear.pdf
  4. Chen, C. W. S., So, M. K. P., & Liu, F.-C. (2011). A review of threshold time series models in finance. Statistics and Its Interface, 4, 167–181. Retrieved from http://intlpress.com/site/pub/files/_fulltext/journals/sii/2011/0004/0002/SII-2011-0004-0002-a012.pdf
  5. Barahona, M., & Poon, C.-S. (1996). Detection of nonlinear dynamics of short, noisy time series. Nature, 381, 215-217. Retrieved from http://www.bg.ic.ac.uk/research/m.barahona/nonlin_detec_nature.PDF
  6. Franz, M. O. (2011). Volterra and Wiener series. Scholarpedia, 6(10):11307. Retrieved from http://www.scholarpedia.org/article/Volterra_and_Wiener_series
  7. Franz, M. O., & Scholkopf, B. (n.d.). A unifying view of Wiener and Volterra theory and polynomial kernel regression. Retrieved from http://www.is.tuebingen.mpg.de/fileadmin/user_upload/files/publications/nc05_%5B0%5D.pdf
$\endgroup$
  • 4
    $\begingroup$ +1 Would you mind writing out the references for these papers in your answer? We have been finding out recently that many links to papers rot after a while, eventually making them almost useless unless the paper authors, titles, etc. are also mentioned in the text. $\endgroup$ – whuber Jan 26 '15 at 23:11
  • 2
    $\begingroup$ @whuber: Thanks. No problem at all, I will update my answer with references this evening. $\endgroup$ – Aleksandr Blekh Jan 26 '15 at 23:49
  • 2
    $\begingroup$ +1 for the link to Rob Hyndman's post. (However, I am tempted to -1 for the complex models. I'd be extremely wary of using threshold or any other nonlinear time series methods on time series of less than 20 observations. You are almost certain to overfit, which goes directly counter to the OP's requirement of a robust method.) $\endgroup$ – Stephan Kolassa Jan 27 '15 at 11:25
  • 3
    $\begingroup$ [2,3,4] do not mention short time series, and look at the plots in [2]: >120 observations. [4] concentrates on finance, where you have enormously more than 20 observations. [5] writes about "short time series, typically 1,000 points long" (p. 216). I see no way to reliably and robustly fit a TAR or similar model, or any of the more complex ones you link to, with <20 observations. (BTW: I also do some inferential statistics on the side, and with fewer than 20 observations, you really can't estimate more than the mean and one more parameter.) $\endgroup$ – Stephan Kolassa Jan 27 '15 at 11:55
  • 4
    $\begingroup$ You're welcome ;-) I guess the takeaway is that "short" is very context-dependent: for sensor reading series or in finance, 1000 data points is "short" - but in supply chain management, 20 monthly observations is almost normal, and "short" will only start at 12 or fewer observations. $\endgroup$ – Stephan Kolassa Jan 27 '15 at 12:15
10
$\begingroup$

No, There is no best univariate extrapolation method for a short time series with $T \leq 20$ series. Extrapolation methods need lots and lots of data.

Following qualitative methods work well in practice for very short or no data:

  • Composite forecasts
  • Surveys
  • Delphi method
  • Scenario building
  • Forecast by analogy
  • Executive opinion

One of the best methods that I know that works very well is the use of structured analogies (5th in the list above) where you look for similar/analogous products in the category that you are trying to forecast and use them to forecast short term forecasting. See this article for examples, and SAS paper on "how to" do this using of course SAS. One limitation is that forecast by analogies will work only of you have good analogies otherwise you could rely on judgemental forecasting. Here is another video from Forecastpro software on how to use a tool like Forecastpro to do forecasting by analogy. Choosing an analogy is more art than science and you need domain expertise to select analogous products/situations.

Two excellent resources for short or new product forecasting:

  • Principle of Forecasting by Armstrong
  • New Product forecasting by Kahn

The following is for illustrative purpose.I just finished reading Signal and Noise by Nate Silver, in that there is a good example on US and Japanese(analogue to US market) housing market bubble and prediction. In the chart below if you stop at 10 data points and use one of the extrapolation methods (exponential smooting/ets/arima...) and see where it takes you and where the actual ended. Again the example I presented is much more complex than simple trend extrapolation. This is just to highlight the risks of trend extrapolation using limited data points. In addition if your product has seasonal pattern, you have to use some form of analogous products situation to forecast. I read an article I think in Journal of Business research that if you have 13 week of product sales in pharmaceuticals, you could predict data with greater accuracy using analogous products.

enter image description here

$\endgroup$
  • $\begingroup$ Thanks for pointing out a different approach! And I agree, Nate Silvers book is great. $\endgroup$ – Tim Jan 27 '15 at 17:09
5
$\begingroup$

The assumption that the number of observations is critical came from an off-handed comment by G.E.P. Box regarding the minimum sample size to identify a model. A more nuanced answer as far as I am concerned is that the problem/quality of model identification is not solely based upon the sample size but the ratio of signal to noise that is in the data. If you have a strong signal to noise ratio you need less observations. If you have low s/n then you need more samples to identify. If your data set is monthly and you have 20 values it is not possible to empirically identify a seasonal model HOWEVER if you think the data might be seasonal then you could start the modelling process by specifying an ar(12) and then do model diagnostics (tests of significance) to either reduce or to augment your structurally deficient model

$\endgroup$
5
$\begingroup$

With very limited data, I would be more inclined to fit the data using Bayesian techniques.

Stationarity can be a bit tricky when dealing with Bayesian time series models. One choice is to enforce constraints on parameters. Or, you could not. This is fine if you just want to look at the distribution of the parameters. However, if you want to generate the posterior predictive, then you might have a lot of forecasts that explode.

The Stan documentation provides a few examples where they put constraints on the parameters of time series models to ensure stationarity. This is possible for the relatively simple models they use, but it can be pretty much impossible in more complicated time series models. If you really wanted to enforce stationarity, you could use a Metropolis-Hastings algorithm and throw out any coefficients that are improper. However, this requires a lot of eigenvalues to be calculated, which will slow things down.

$\endgroup$
0
$\begingroup$

The problem as you wisely pointed out is the "overfitting" caused by fixed list-based procedures. A smart way is to try and keep the equation simple when you have a negligible amount of data. I have found after many moons that if you simply use an AR(1) model and leave the rate of adaption ( the ar coefficient) to the data things can work out reasonably well. For example if the estimated ar coefficient is close to zero this means that the overall mean would be appropriate . if the coefficient is near +1.0 then this means that the last value (adjusted for a constant is more appropriate . If the coefficient is close to -1.0 then the negative of the last value (adjusted for a constant) would be the best forecast. If the coefficient is otherwise it means that a weighted average of the recent past is appropriate.

This is precisely what AUTOBOX starts with and then discards anomalies as it fine tunes the estimated parameter when a "small # of observations" is encountered.

This is an example of the "art of forecasting" when a pure data driven approach might be inapplicable.

Following is an automatic model developed for the 12 data points without concern for anomalies. enter image description here with Actual/Fit and Forecast here enter image description here and residual plot here enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.