When I am using k-folding to split my labelled data (labelled as signal or background) and train k classifiers on it, I believe I am not allowed to assume that the distributions of the classifier output variable on, for example, signal, is the same among the k test samples of the k folds.
I am not talking about statistical fluctuations here, but about the fact that there is no assurance on how the classifier output variable is scaled.
Is that right? Or is there such an assurance for, say, scikit learn?
So, now I have my data set with the labelled data and the classifier output written in one column. Depending on the row the right one of the k classifiers was used to calculate a classifier output on that event.
I then apply a cut on the classifier output variable in my labelled data. Because the data is labelled, I can measure the background and signal efficiency of that cut. I am applying the same cut on all data points, no matter which classifier was used to determine the classifier output. This of course is not optimal unless the output distributions of the k classifiers are similar.
Now, I also have some unlabeled data which was therefore not used in the k-folding.
I am not sure which of the k classifiers to use to determine the classifier output on unlabeled data. It is tempting to just use any of them, for example the first one.
But because the output distributions of the k classifiers might differ, I can not a priory assume that the signal or background efficiency of applying my cut on the response of the first classifier only is the same as the efficiency that I estimated on my unlabeled data, where I also used the k-1 other classifiers.
I was thinking I could randomly split my unlabeled data into k equal data sets and and use a different one of the k classifiers on each. This way I am mirroring what happened on the labelled data, which should lead to the same signal and background efficiency, even if the classifier output distributions differ among the k classifiers.
Do you agree that this is the right approach?
Is there a best practice that I am not aware of?
EDIT: I should probably mention that I am talking about the "k-folding" method as it is used in High Energy Physics. The accepted answer explains what is meant by this in the first paragraph.