I have a training set with 140 instances and no separate cross validation set. The data set contains 7 measurements from each of 20 objects, hence the 140 instances. Each of 7 measurements have the same label.

My problem is the choice of cross validation (CV) scheme. I wanted to do 20-fold CV. But this will almost always be biased since the random subsets will have different object measurements mixed. Instead I decided to "leave one object out" each time, i.e., at each run of 20-fold CV, a validation set contains one of the 7 instances and and the remaining 133 will be the training set. Do you think this is a fair CV scheme?

  • $\begingroup$ Can you explain what your data set looks like and what your model is a little better? Do you have 20 observations with seven variables, or 140 observations with some sort of correlation structure? $\endgroup$ – Jonathan Christensen Dec 28 '12 at 21:03
  • $\begingroup$ Dataset = { {obj1measurement1, f1, f2, ..., fn,label1}, {obj1measurement2, f1, f2, ..., fn,label1}, ..., {obj1measurement7, f1, f2, ..., fn,label1}, ..., {obj20measurement7, f1, f2, ..., fn,label7} } where f' s refer to features and label' s are from {0,1}. $\endgroup$ – Zoran Dec 28 '12 at 21:10
  • $\begingroup$ This will be used for a classification task where there is also a 28-instance test set. $\endgroup$ – Zoran Dec 28 '12 at 21:18
  • $\begingroup$ It sounds to me like you have 20 observations. For your cross-validation, take out one observation at a time, and fit to the remaining 19. $\endgroup$ – Jonathan Christensen Dec 29 '12 at 0:52
  • $\begingroup$ I dont want to do this. Measurements are done in a time line and can be informative individually. Since data is small, I want to use all of them. Anyways, eventually, I applied both k-fold CV and the above mentioned approach and compared their results. Thanks for your time. $\endgroup$ – Zoran Dec 29 '12 at 10:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.