I have a training set with 140 instances and no separate cross validation set. The data set contains 7 measurements from each of 20 objects, hence the 140 instances. Each of 7 measurements have the same label.

My problem is the choice of cross validation (CV) scheme. I wanted to do 20-fold CV. But this will almost always be biased since the random subsets will have different object measurements mixed. Instead I decided to "leave one object out" each time, i.e., at each run of 20-fold CV, a validation set contains one of the 7 instances and and the remaining 133 will be the training set. Do you think this is a fair CV scheme?

  • $\begingroup$ Can you explain what your data set looks like and what your model is a little better? Do you have 20 observations with seven variables, or 140 observations with some sort of correlation structure? $\endgroup$ Commented Dec 28, 2012 at 21:03
  • $\begingroup$ Dataset = { {obj1measurement1, f1, f2, ..., fn,label1}, {obj1measurement2, f1, f2, ..., fn,label1}, ..., {obj1measurement7, f1, f2, ..., fn,label1}, ..., {obj20measurement7, f1, f2, ..., fn,label7} } where f' s refer to features and label' s are from {0,1}. $\endgroup$
    – Zoran
    Commented Dec 28, 2012 at 21:10
  • $\begingroup$ This will be used for a classification task where there is also a 28-instance test set. $\endgroup$
    – Zoran
    Commented Dec 28, 2012 at 21:18
  • $\begingroup$ It sounds to me like you have 20 observations. For your cross-validation, take out one observation at a time, and fit to the remaining 19. $\endgroup$ Commented Dec 29, 2012 at 0:52
  • $\begingroup$ I dont want to do this. Measurements are done in a time line and can be informative individually. Since data is small, I want to use all of them. Anyways, eventually, I applied both k-fold CV and the above mentioned approach and compared their results. Thanks for your time. $\endgroup$
    – Zoran
    Commented Dec 29, 2012 at 10:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.