Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc.

I'm at the point where I'm trying to do some feature reduction work/parameter optimization and concerned about over-fitting by using the same CV setup to find parameters and then get error measures.

My question is this: if I create a separate CV indexing setup where row 1 = CV set 5, row 2 = CV set 3,..., row6 = CV set 1, etc. does this function similarly to having a separate validation set or do I need to explicitly have a 50/25/25 train/validation/test setup?

share|improve this question
    
You could use bootstrap for the error rate estimation. –  Michael Chernick Aug 24 '12 at 15:55

1 Answer 1

up vote 7 down vote accepted

It is best to treat CV as a method for estimating the generalization performance of a method for modeling the data (rather than of the model) and if your method includes feature selection then you need to perform the feature selection step independently in each fold of the cross-validation; otherwise you will end up with an optimistically biased performance estimate.

Re-partitioning the dataset to get a different cross-validation estimate (which seems to be what you are suggesting) will end up with a biased performance estimate as the test data in each fold of the performance evaluation CV will have been used to optimize the model by minimizing the feature selection CV estimate.

The simple rule is that the test data for performance estimation cannot have been used in any way (no matter how indirectly) in tuning any aspect of the model. I find nested cross-validation is a good approach for most problems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.