I'm thinking about writing a framework for time series analysis and obviously the question of scaling for big data sets comes up.

From my experience even for large data sets (larger than the main memory) this will

  • either be handled by scanning in a linear fashion (e.g., for feature extraction)
  • or it can be processed in sliced windows (e.g., sliding window validation).

Could you think of examples (use cases or methods) where an analysis could not be done in this way and instead one needs (random) access to the complete series data?


closed as too broad by Tim, mdewey, John, gung, Peter Flom Dec 15 '16 at 13:20

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    $\begingroup$ What kind of models do you want to implement? Do you ask if any kind of time-series analysis can be conducted in this fashion, obviously not. Please make it more precise, otherwise your question is too broad to be answerable. $\endgroup$ – Tim Dec 13 '16 at 9:24
  • $\begingroup$ Tim: Yes the question is very broad because at first I want to write the underlying structure (API) for such models. If you say not any kinds of models can be constructed in such a way, can you give me counter example? $\endgroup$ – David Dec 13 '16 at 9:48
  • 3
    $\begingroup$ not every. I can't see how could you possibly write "underlying structure" that fitts every possible time-series model (univariate, multivariate, Bayesian, frequentist, simple, complicated, with and without covariates, hierarchical and not, etc.). $\endgroup$ – Tim Dec 13 '16 at 9:57
  • 2
    $\begingroup$ Wouldn't many maximum likelihood (ML) algorithms have to reload data every iteration? (excluding special cases such as where ML is equivalent to ordinary least squares) $\endgroup$ – Matthew Gunn Dec 13 '16 at 9:57

Some times in time series models, you do not want to extract a subset of the time periods because you want to preserve the original structure in the time series. For example, you might be willing to cut out the first half but you might not be willing to cut out 50% of the data randomly chosen because you would then lose the correlation structure. One example is for particle filtering models.

That being said, many time series methods deal with a low number of observations which would not make them "big data".


Not the answer you're looking for? Browse other questions tagged or ask your own question.