5
$\begingroup$

We are extracting features from EEG, which is a time dependent signal.

We have signals of 10,000 datapoints over 64 channels, and we extract 10 features per timestamp per channel, so at the end we have a feature data set of 64x10 features for each time step.

We are applying PLS to regress against hand movements (X,Y,Z).

We are using cross validation to verify the results, but some of our colleagues are pointing out that using interleaved sampling from the datasets feels like cheating.

Their suggestion is to use 2/3rds of contiguous data for training and 1/3rd for testing.

I would like to hear the overall opinion on this setting, my intuition tells me that since it is time dependent data, just dividing the datasets like that would yield bad results (which it does)

$\endgroup$
4
$\begingroup$

You want your evaluation to tell you something useful about your system's performance. Using a specific, held-out test set is nice because it tells you how the system will perform on totally new data. On the other hand, it's hard-to-impossible to perform meaningful inference (i.e., "In general, is my system better than this other one?") with only a single data point and one rarely has nearly enough data to create multiple test partitions. Cross validation (and similar techniques) try to estimate the generalization error by dividing up the data into multiple training and test sets and evaluating on each of them. This is nice because you get multiple estimates of its generalization ability, which allows you to do some proper statistical comparisons. However, these are only valid if the cross-validation folds are set up appropriately.

For a "classic" machine learning problem with one set of observations per subject (e.g., the Fischer Iris or Pima Indian data sets), it's hard to mess up cross-validation. Divide your data into $k$ folds, use $k-1$ of them for training and test on the last one. Lather, rinse, and repeat until each fold has been used as the test set.

This is probably less advisable for a data set like yours, where there are multiple (64x10x?) observations from each subject. If the observations are correlated across time, space/sensor, and within subjects (as yours almost certainly are), then the model might "learn" some of these associations, which would boost its performance when data from the same/nearby time points, sensors, and subjects appear in the test set, and thus provide an over-optimistic estimate of your system's ability to generalize.

The most conservative approach would be to stratify your cross-validation by subject: all the data from a given subject goes into the same fold (e.g., subjects 1-5 are in fold 1, 6-10 in fold 2, etc) and then cross-validation proceeds as normal. If you're willing to assert that temporally-distant parts of the signal are independent, I guess you could try stratifying that way too, but I'd find that considerably less satisfying. What I would not do is dump all data into a big list, regardless of time point, sensor, or subject, and then randomly divide that list into folds. I can almost guarantee that will overstate your generalization error!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.