Suppose you have a set of observations that occur at regular intervals in time, but containing regular gaps during which there is no data, not because it is hidden or missing, but because the phenomena does not occur at those times. For example, you might have hourly or minutely solar insolation data in some location, but only when the sun is above the horizon. An even better example would be hourly or monthly prices of some security that is traded only for certain hours of the day, and only on weekdays by local time to the exchange. Outside of those hours and days, there are no transactions, so there genuinely is no price. I want to fit a time series regression model to this data.

Clearly there are plenty of possible approaches to estimates dealing with this. One could interpolate and impute, at the risk of finding your interpolation method rather than the true relationships. You could take the least common multiple of the various periodicities -- e.g. 24 hours per day times 7 days per week = 168 hours -- hand that to some time series algorithm with a lot of degrees of freedom, maybe something designed to detect and estimate seasonality, and hope for the best. You could treat the missing observations as zeros and throw in a lot of Fourier terms (though these observations are exactly zero, so I question whether they should be treated as measurements). You could excise the nonexistent hours, and estimate on, say, an eight-hour day and a five-day week. To me, the last of these seems cleanest, but the cost is high, as other temporally-organized phenomena that influence our observed values may not respect the same limitations. I suppose we could add day-of-the-week and hour-of-the-day dummies and their interaction terms, were it not for multicollinearity. And so forth.

(I have found a lot of previous questions that address this question in some way, e.g. How to handle non existent (not missing) data? and How to train the Model in Time Series with Holiday, Weekends, but none that seem to address my concerns exactly.

AmI right in thinking that this is a theoretically important distinction? In cases like this, with intervals in which the data is non-existent and not just missing, if you want to fit and forecast some time-series model, are there one or more techniques that are particularly appropriate -- best practice methods for data of this type? These are not, for example, unevenly-spaced series -- the periods of non-existence are quite regular. I think that, excluding e-commerce, most data that represents economic activity at daily frequency or higher is of this type.

I have been trying to do this using the the R tsibble/fable framework, in which every observation has a particular time stamp, but I am asking for answers that are good based on theory or practical experience, not necessarily for a particular software implementation.

My background is in econometrics, but I have not previously used data with less than monthly frequency.

  • 1
    $\begingroup$ The price of a security is only observable at transaction time but the value of a security "exists" continuously at all times. In that application domain, by far the most common way of treating this feature is by viewing a time series as discretely sampled from a continuous-time process, which is modeled by, for example, an Ito process. $\endgroup$
    – Chris Haug
    Aug 30, 2019 at 21:14
  • $\begingroup$ @ChrisHaug Thanks, Chris! Interesting. I have never worked with a continuous time model, but I see how it could have advantages for data like this, especially high-frequency data or transactional data. If my data set is hourly, does the problem become more tractable by embedding the data in a continuous time model? I'd think that you would have to face all the same problems with missing or non-existent hours, whether you regard them as intervals or points. $\endgroup$
    – andrewH
    Sep 1, 2019 at 14:32
  • $\begingroup$ As for tsibble/fable, we are considering how to represent structural missingness in this issue: github.com/tidyverts/tsibble/issues/18 Modelling data with structural missingness will be supported in fable once the data structure is supported. $\endgroup$ Oct 17, 2019 at 6:44
  • $\begingroup$ Thanks, @Mitchell O'Hara-Wild! That is good to know. $\endgroup$
    – andrewH
    Oct 30, 2019 at 2:43


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.