3
$\begingroup$

I'm looking to use n-fold cross validation for selecting meta-parameters for fitting a model to a dataset. However, dropping observations entirely from the learning-set while fitting the model to each of the folds may create problems for the model fitting.

I was wondering whether it would be valid to perform the cross-validation on a weighted basis, e.g. the observations selected for the fold would receive a weight of say 0.95 and the ones excluded from the fold would receive a weight of say 0.05. When assessing predictive performance one then would weight prediction errors according to the complement of these weights, e.g. (1-0.95) for those 'participating heavily' and (1-0.05) for those 'participating lightly'.

Is there any merit in this intuition?

Such a process is similar to Fay's Balanced Repeated Replication (Survey sampling variance estimation). I have never encountered it in relation to cross-validation. Any literature pointers are appreciated.

$\endgroup$

1 Answer 1

10
$\begingroup$

I doubt this is a good idea. The core idea behind CV is that it is a simulation of testing model on unseen data; any leak of information from local test sets to local trains will ruin the guarantee that the performance on local test is not a result of overfitting, thus making the result unreliable.

If your model has a problem with reduced train set, it is quite possible that you have caught some general instability and you should look for some better solution.

$\endgroup$
1
  • $\begingroup$ Some more intuition: Statistical learning theory leverages the fact that w.h.p your iid training and test sets will be representative of the distribution from which they were drawn. Hence low training error and high test error implies a poor classifier rather than some unlucky pick of the test points. Therefore, you should trust the data and not try to massage the test error. $\endgroup$ Jul 15, 2011 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.