0
$\begingroup$

I'm trying to understand how and at which point can one apply Cross Validation for time series data. If i'm not wrong CV increases generalisation so that our model has less bias in case the data is ordered/distributed in a certain particular way. But time series inherently has a direction (it can only move forward) and if i'm using some unsupervised algorithms on it, how should one apply Cross validation on it if any. I read a related question on it here : (Cross-validation techniques for time series data ) although as opposed to the question above for supervised learning i'm planning on some unsupervised clustering algorithms to group a set of time series together. So basically i have two doubts :

  1. How to approach the problem of CV for my time series problem.
  2. majority of questions related to cv and timeseries and cv used supervised algorithms? So an insight taking into account the clustering technique i would be using would be preferable.
Improve this question
$\endgroup$
2
  • $\begingroup$ 1) What does it mean that you have a "stack" of time series? Properly designing a CV scheme requires accounting for dependencies in the data, so we need more information about how your data is structured. 2) CV for unsupervised learning can be complicated. Certain things work in some cases and not in others. We need more information about what you're trying to do. 3) Are you using CV for model selection or for reporting performance? What criterion are you using to measure performance? $\endgroup$
    – user20160
    Aug 31, 2018 at 8:41
  • $\begingroup$ 1. Stack of time series : The time series are somewhat correlated in the sense that they are behaving similarly, like web server pools processing requests. So yes the time series is correlated. 2. CV process: I understood what you meant. What I'm trying to achieve is cluster all the servers behaving similarly while as in above example processing requests. 3. Yes CV for performance evaluation is my main aim. I'm also really stuck at measuring performance too, as i really don't have any true_labels to be honest. Any help on this could be helpful. $\endgroup$
    – Mr. Confused
    Aug 31, 2018 at 9:43

1 Answer 1

Reset to default
1
$\begingroup$

If i'm not wrong CV increases generalisation so that our model has less bias in case the data is ordered/distributed in a certain particular way.

No, your understanding is wrong here. CV in itself does not help with generalization, it can only help in measuring the generalization abilities of a model.

And even that is only possible if you have a suitable figure of merit that is evaluated. Such figures of merit that measure the performance of predictions are readily availbe for supervised models. For unsupervised models such as cluster analysis it is much more difficult to formulate them. Models can differ in ways that are unimportant to the generalization ability and/or they can differ in ways that are important, and only the latter differences are of interest to us. You may also say that of all ways how a model can differ, we are only interested in a subset and the question is how to catch that subset.

Predictive models have one easy to capture type of important differences: differences in the prediction, and the usual figures of merit track exactly those differences. For your cluster analysis, you'll have to specify what differences are important and what differences are unimportant/benign. This is something that needs to be done in close relation to the application (data generating processes, characteristics of your data, and the question(s) you want to answer).

Using a clustering technique pretty much implies that you do not have a ground truth to compare the clustering with. This is what limits the general possibility of having figures of merit similar to the ones we have for predicitive models. However, if you do have something like a ground truth, you may derive a figure of merit based on that ground truth.

Ensemble models can help with generalization in certain situations and they are linked to resampling validation (such as CV) in that they also use a resampling step.

Time series cross validation: one crucial question that makes a lot of difference in how to set up the splitting is whether you are looking along time series.

  1. For prediction that is training on past events and then predict future developments of the same time series. An example in process analysis would be to train a model predicting current (or future) concentration based on past readings of some sensors.
  2. Or, whether you look across a large number of time series, e.g. series of sensor readings for past runs of a batch process against the achieved final product concentrations and then want to predict final product concentration for unknown batches based on the respective full time series of sensor readings for the batch in question.

Sliding window cross validation is for the first scenario, the 2nd scenario uses each time series as its own case and then "normal" cross validation.

Correlation and stack of time series. The crucial question here is not whether there's a similarity in the time series but what causes the similarities. If there's an underlying factor causing similarity (i.e. you have several pools of servers in your data and the server data is more similar with each pool than compared to a server of another pool), you need to take that into account for CV splitting. If on the other hand the similarity is not caused by other factors but is just the phenomenon you're modeling, that is fine.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks cbeleites for your answer and particularly about explaining about what CV really does. You are right i don't have any ground truths and that's why i went with the clustering approach, and the only metric that i can seem to use right out of the box is silhouette score. I understood your approach of determining the differences that really matter and then maybe using them for improving on our base algorithms. $\endgroup$
    – Mr. Confused
    Sep 4, 2018 at 5:38
  • $\begingroup$ Also my use case here is not to predict some future value but just study the behaviour of lets say 5 servers (lets say their memory utilisation in 1 min ticks) running together and then find whether any of them is behaving abnormally by clustering the similar behaviours together and others not in that group. sliding window CV i get but what is normal "CV". Is it like ensemble average i.e. CV across multiple signals? Should we do that and how to do that ? I know it's asking too much but can you provide an example or a more complete description. It will be a lot of help. Thanks. $\endgroup$
    – Mr. Confused
    Sep 4, 2018 at 5:38
  • $\begingroup$ 3. In your third point where you mentioned "correlation and stack of time series" you mentioned that the underlying similarity has to be taken into account if an underlying factor causes similarity. But how would we judge that?. And i get that. $\endgroup$
    – Mr. Confused
    Sep 4, 2018 at 5:46
  • $\begingroup$ 4. Also when you say that this has to be taken into account, does that mean all the series with let's say high similarity to each other to be first grouped together , then CV and clustering applied only on that particular group to find the abnormal behaviour (in my case) and so on to the second group of similar time series stack and then third and so on. Please correct me if i'm wrong. $\endgroup$
    – Mr. Confused
    Sep 4, 2018 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.