I was wondering how to asses which sample in the data, during K fold cross-validation drives the bias that may be observed in the results.

My training data consists of 40 samples. And I try to classify these samples based on some class that i know using PLSDA. I perform 10 fold stratified cross-validation a 100 times and identify the optimal parameter.

I find that there is some bias based on how the data is split, and would like to understand which are the samples in the test-fold, that drive the bias.

For this, I keep track of how my data is split across ten folds, and for each test fold, I also keep track of the error. I then count the number of times a patient was in a test fold that gave me the least error or the greatest error when compared to other folds. And try to come up with a metric based on this frequency as a measure of contribution to bias...

But, each fold consists of 4 patients, and the final metric I get may not reflect the individual contribution. Any thoughts on this would be helpful.

  • $\begingroup$ I'd consider the changes in model due to diffent splits an indicator of variance, i.e. the models are unstable. $\endgroup$ – cbeleites unhappy with SX Mar 10 '15 at 9:29

If you consider a leave-one-patient-out jackknifing scheme instead of leaving out 4 patients, you can easiy calculate Cook's Distance which is a measure of leverage for each of the patients.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.