I have a training set with 140 instances and no separate cross validation set. The data set contains 7 measurements from each of 20 objects, hence the 140 instances. Each of 7 measurements have the same label.
My problem is the choice of cross validation (CV) scheme. I wanted to do 20-fold CV. But this will almost always be biased since the random subsets will have different object measurements mixed. Instead I decided to "leave one object out" each time, i.e., at each run of 20-fold CV, a validation set contains one of the 7 instances and and the remaining 133 will be the training set. Do you think this is a fair CV scheme?