Is it valid to assign observations partially to both test and train in n-fold cross validation?

Question

I'm looking to use n-fold cross validation for selecting meta-parameters for fitting a model to a dataset. However, dropping observations entirely from the learning-set while fitting the model to each of the folds may create problems for the model fitting.

I was wondering whether it would be valid to perform the cross-validation on a weighted basis, e.g. the observations selected for the fold would receive a weight of say 0.95 and the ones excluded from the fold would receive a weight of say 0.05. When assessing predictive performance one then would weight prediction errors according to the complement of these weights, e.g. (1-0.95) for those 'participating heavily' and (1-0.05) for those 'participating lightly'.

Is there any merit in this intuition?

Such a process is similar to Fay's Balanced Repeated Replication (Survey sampling variance estimation). I have never encountered it in relation to cross-validation. Any literature pointers are appreciated.

Answer 1

I doubt this is a good idea. The core idea behind CV is that it is a simulation of testing model on unseen data; any leak of information from local test sets to local trains will ruin the guarantee that the performance on local test is not a result of overfitting, thus making the result unreliable.

If your model has a problem with reduced train set, it is quite possible that you have caught some general instability and you should look for some better solution.