I was wondering how to asses which sample in the data, during K fold cross-validation drives the bias that may be observed in the results.
My training data consists of 40 samples. And I try to classify these samples based on some class that i know using PLSDA. I perform 10 fold stratified cross-validation a 100 times and identify the optimal parameter.
I find that there is some bias based on how the data is split, and would like to understand which are the samples in the test-fold, that drive the bias.
For this, I keep track of how my data is split across ten folds, and for each test fold, I also keep track of the error. I then count the number of times a patient was in a test fold that gave me the least error or the greatest error when compared to other folds. And try to come up with a metric based on this frequency as a measure of contribution to bias...
But, each fold consists of 4 patients, and the final metric I get may not reflect the individual contribution. Any thoughts on this would be helpful.