Can one use k-fold cv and holdout analysis together? I would like to start by saying i have just started using cross-validation, so please bear with me if the questions seems very trivial.
I am reviewing someones work where the person has used k-fold cross validation and holdouts concurrently.
What the person has done is split the data into 5-folds for model selection. I follow the approach upto this point. Next what he claims to have done is use holdout analysis but what he has really done is iterate the process of building the model on each fold 10 times. I am not sure if this can be called as holdout analysis, but as stated above i am very new to this.
So is this approach right? or more importantly is it wrong, because if the 10 iterations are just redundant then i am ok with that as long as it is not fundamentally wrong.
Another quick doubt, is there a case where k-fold is used along with holdout analysis because based on my understanding they are two ways of validating a model or selecting a model.
 A: It isn't voodoo - there is a specific purpose behind each approach.  If you understand the purpose and how it leads to the approach, then it makes sense (or shows the lack thereof) for variations on the theme.
The goal of the process is to get "generalization without over-fitting".  That means you fit the model to the signal, but not the noise.  If you fit this samples noise, then your model is actually worse on the next sample because the noise is random - nobody knows what it is going to be next time around.
The purpose of hold-out is to have a way to test for over-fitting.  If the training error and validation or test error suddenly diverge, then you have a problem.
The problem with holdout is that you are not accessing all of the information.  What if you kept training-important data in the hold-out set?  An answer to that is cross-validation.  It gives the same generalization - the same not over-fitting to the noise - that you get from the holdout, but it gives every piece of data a chance to speak once or twice.
CV is meant to over-come the weakness of the holdout: use all the data to fit the model and still get a signal for over-fit based on not fitting some portion of the data.
Combining them means the actual data was trained on less than the hold-out set.  It makes for weaker models.
