Currently, I am working with a lot of time series data. A lot of my time series data have a lot of leading zeros. For example,

ts = [0, 0, 0, 0, 0, 0, 256, 129, 345, ...]

All, of the time series I am investigating are on a monthly cadence. My overall goal is to get the best forecast after fitting with Prohet, ARIMA, etc.

My first thought with these time series that have many zeros, was to remove them and then fit a model. However, this does not always yield the best results in terms of RMSE or MAPE. So my questions are:

  1. Is it customary to remove leading zeros from a time series?
  2. Are there any types of analysis or tests that can determine when to remove leading zeros from a time series?

I have done some searching online but there is not a lot of information on this topic that I could find on my own. Any comments or resources would be greatly appreciated.

  1. No, this is not customary.

    Your time series may be an : demand is often zero and sometimes nonzero. In this case, leading zeros would just represent bona fide zero demand. Removing them would bias your forecast upward.

    Conversely, your leading zeros might result from padding a series with zeros, and represent an artifact of data munging. In this case, you should of course remove them.

    You need to understand where your data comes from and what has been done with it in order to know what to do with it.

  2. If your series starts with a "long" string of zeros and there are "few" zeros after, then you probably have the second one of the two cases above, and you could remove the zeros with a certain level of confidence. Appropriate values for "long" and "few" will depend on your time series, especially on its time granularity and what it actually represents. You should also remove the zeros if they are physically impossible or at least highly implausible, e.g., if they represent humidity measures taken someplace wet.

Bottom line: subject matter expertise is pretty much indispensable here.

Incidentally, you write that "this does not always yield the best results in terms of RMSE or MAPE". Be aware that the RMSE and the MAPE are usually minimized by different point forecasts. See Why use a certain measure of forecast error (e.g. MAD) as opposed to another (e.g. MSE)?. I have an upcoming invited commentary on the M4 forecasting competition, which should soon appear in the International Journal of Forecasting and which discusses this further. If you are interested, I can send you the manuscript.

| cite | improve this answer | |
  • $\begingroup$ Thank you @Stephan Kolassa for the response. That makes sense, and with certainty, I can say that the series that have zero values at the beginning of the series are because the data was not observed for that period. That manuscript sounds also like a great read for me because I am still trying to understand what measures are appropriate and which others are not when it comes to fitting time series. $\endgroup$ – RDizzl3 Jun 17 '19 at 20:48
  • $\begingroup$ If you know that zeros don't make sense, that is great - then you know you can remove them. If you want that manuscript, send me an email at Stephan dot Kolassa at sap dot com (and ping me here in a couple of days if I don't answer, in case your mail ends up in my spam folder). $\endgroup$ – Stephan Kolassa Jun 17 '19 at 20:50

My approach ( given that it is not intermittent demand data ) is to reverse forecast. Take your time series and arrange it from last to first (without tailing zeroes) and model it to obtain predicted values for the unrecorded past.

See https://stats.stackexchange.com/search?q=user%3A3382+reverse+forecasting . Additionally some researches use the term backcasting to fill in /estimate values that went unrecorded. Also see hindcast What is the proper name for a backward forecast?

| cite | improve this answer | |
  • $\begingroup$ Aren't you implicitly re-interpreting the zeros as missing data? $\endgroup$ – whuber Jun 18 '19 at 13:55
  • $\begingroup$ yes because they would be inconsistent with the prediction and thus flagged as exceptional.. The size (complement) of that estimated intervention/pulse is then the estimate of the missing value. $\endgroup$ – IrishStat Jun 18 '19 at 14:03
  • $\begingroup$ Thank you -- that makes sense. I presume by "flagged as exceptional" you mean "identified a priori as a sequence of initial values that might not be characteristic of the remaining time series and whose presence could adversely affect the results of the analysis." But if that's the case, why would backcasting be of any use at all when the stated purpose is "to get the best forecast"? Isn't your proposal tantamount to saying "just drop the initial zeros"? $\endgroup$ – whuber Jun 18 '19 at 14:13
  • $\begingroup$ In the example where the replacement values are estimated (from the artificial first step) one would have to actually redefine their original data ( going forward) with the replacement values estimated from the backcast replacing the 0's. Thus in the second step the data would be cleaner as the 0's would be replaced with estimates from the artificial backcast. and would now generally be non-zeroes. $\endgroup$ – IrishStat Jun 18 '19 at 14:22
  • 1
    $\begingroup$ ok I will do so and add it to my response. $\endgroup$ – IrishStat Jun 18 '19 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.