I'm thinking about writing a framework for time series analysis and obviously the question of scaling for big data sets comes up.
From my experience even for large data sets (larger than the main memory) this will
- either be handled by scanning in a linear fashion (e.g., for feature extraction)
- or it can be processed in sliced windows (e.g., sliding window validation).
Could you think of examples (use cases or methods) where an analysis could not be done in this way and instead one needs (random) access to the complete series data?