I have a smallish dataset ~ 1500 rows X 500 columns. I've been using a standard 5 fold CV setup where row 1 = CV set1, row2 = CV set2, ... row 6 = CV set1,etc.
I'm at the point where I'm trying to do some feature reduction work/parameter optimization and concerned about over-fitting by using the same CV setup to find parameters and then get error measures.
My question is this: if I create a separate CV indexing setup where row 1 = CV set 5, row 2 = CV set 3,..., row6 = CV set 1, etc. does this function similarly to having a separate validation set or do I need to explicitly have a 50/25/25 train/validation/test setup?