0
$\begingroup$

Does anybody know in which cases --of learning and predicting--, it is better to use "validation test" or something else, instead of "k-fold cross validation" to assess the performance of the algorithm in predicting the target classes? Here is my case and problem: Yes, I know that if the data is less, CV works well, but this case --with limited samples-- is a bit different. Because the order of the data as the test set is important for us, so that we can not apply CV, since CV gives you the average performance of all folds. More in detail: I have the following data set with 4 features and 10 samples;

            f1,f2,f3,f4,class
       S1  
       S2
       S3
       S4
       S6
       S7
       S8
       S9
       S10

Every sample referees to an activity that accrued in a specific date (E.g, S1 logged in the first week, S2 logged in the second week, and so on). It means an activity that logged in sample (S1) is much different from the last sample (S10), on the other-hand S1 and S2 or S3 are not much different. So in this case, I used S1 to S8 as the training set, and S9 and S10 as the test set to validate the classification performance. Since, the samples and values change over time, I could not use CV to validate the performance. So my questions is, can I rely on this statement and my way to design the training and test is reliable?

Improve this question
$\endgroup$
6
  • $\begingroup$ By "validation test" do you mean measuring performance a single, held-out validation set rather than repeated over k folds? $\endgroup$
    – user20160
    Commented Sep 21, 2018 at 10:40
  • $\begingroup$ Yes, E.g., 70% of the data as training and 30% as the test set, considering if we have around 10 samples --of course they are few--. $\endgroup$
    – nahid khosh
    Commented Sep 21, 2018 at 10:45
  • 1
    $\begingroup$ A test set is preferred over CV, when possible. So if you have a big enough data set and can afford to split your data then you should do so, otherwise, if you do not have a large enough sample, CV works well. $\endgroup$
    – user2974951
    Commented Sep 21, 2018 at 10:49
  • 1
    $\begingroup$ essentially its a computational issue. if it takes too long to train you use single split of data (eg enormous Neural networks). Equally if its too long because you have so much data, then a single split will still be reliable. [andrew ng makes this statement in coursera.org/learn/deep-neural-network] $\endgroup$
    – seanv507
    Commented Sep 21, 2018 at 10:50
  • $\begingroup$ Yes, I agree that if the data is less, CV works well, but this case --with limited samples-- is a bit different. I have updated my question with more detailed information. $\endgroup$
    – nahid khosh
    Commented Sep 21, 2018 at 12:36

1 Answer 1

Reset to default
3
$\begingroup$

  • single test set vs. cross validation has been discussed before.

  • As for your data, you essentially have a time series. This leads to dependence along your data. Thus, a single random split or a cross validation will not achieve independence between training and test sets.

    Dealing with that will depend on how your model will be used.

    • in case you'd train on your S1-S10 data, and then start predicting S11,S12, and so on: look into sliding windows cross validation for time series. For these models, you can actually use the temporal correlation to help with your prediction.
    • If you do not want or cannot use that temporal correlation, you may be able to obtain test data after finding out the temporal length of correlation and obtain data that is far enough away in time from all your training data so that training and test data are not correlated any more.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.