How to apply classifiers from k-folding to data not used in the k-folding?

Question

When I am using k-folding to split my labelled data (labelled as signal or background) and train k classifiers on it, I believe I am not allowed to assume that the distributions of the classifier output variable on, for example, signal, is the same among the k test samples of the k folds.

I am not talking about statistical fluctuations here, but about the fact that there is no assurance on how the classifier output variable is scaled.

Is that right? Or is there such an assurance for, say, scikit learn?

So, now I have my data set with the labelled data and the classifier output written in one column. Depending on the row the right one of the k classifiers was used to calculate a classifier output on that event.

I then apply a cut on the classifier output variable in my labelled data. Because the data is labelled, I can measure the background and signal efficiency of that cut. I am applying the same cut on all data points, no matter which classifier was used to determine the classifier output. This of course is not optimal unless the output distributions of the k classifiers are similar.

---

Now, I also have some unlabeled data which was therefore not used in the k-folding.

I am not sure which of the k classifiers to use to determine the classifier output on unlabeled data. It is tempting to just use any of them, for example the first one.

But because the output distributions of the k classifiers might differ, I can not a priory assume that the signal or background efficiency of applying my cut on the response of the first classifier only is the same as the efficiency that I estimated on my unlabeled data, where I also used the k-1 other classifiers.

I was thinking I could randomly split my unlabeled data into k equal data sets and and use a different one of the k classifiers on each. This way I am mirroring what happened on the labelled data, which should lead to the same signal and background efficiency, even if the classifier output distributions differ among the k classifiers.

Do you agree that this is the right approach?

Is there a best practice that I am not aware of?

EDIT: I should probably mention that I am talking about the "k-folding" method as it is used in High Energy Physics. The accepted answer explains what is meant by this in the first paragraph.

Answer 1

(answer was updated to mention HEP)

Given that question author's background I'm assuming this question is about high energy physics (HEP).

In HEP we need unbiased quality estimation and we want to make use of all available data (since for many analyses amount of available data is very limited) at the same time. That is why in HEP we often use k-folding scheme:

1. divide data into k folds
2. train k classifiers on k-1 folds of data
3. predict i-th fold of data by i-th classifier, which is not trained on this i-th fold

Then using the same data, which now are marked, we estimate quality (we want to preserve as much as possible statistics for estimation, that is why i.e. we don't divide data into training and test parts). After that we need to apply the model to data from collider to compare quality: expected and observed (we test hypothesis if we have in data from collider searched decay). The answer below should be considered in the context of HEP above problem.

If you use stable models and have enough data samples then the distributions of the classifiers' outputs for signal and background events will be very similar across all folds (assuming you use the same parameters for all classifiers in the folding scheme).

During new data prediction, which didn't take part in the training process, it is a bad idea just to use any of the classifiers from folding scheme. In this case efficiency for chosen threshold will be overestimated or underestimated (due to the reasons you named). There are two ways of predicting new data:

1. Predict a sample by all classifiers in the folding scheme, take the average. This gives you reliable stable prediction for a new sample.
2. Take a random trained classifier from the folding scheme set of the classifiers. This is equivalent to divide new data into k-folds and predict each fold by only one classifier, like the training sample.

In my practice in rare decays analyzes to preserve data as much as possible we use folding scheme and predict new data the second way. In this case the same model is applied to the training and test sample and thus we will have correct quality estimation on the test sample for different measures like efficiency (thus single threshold is chosen for all classifiers).

Answer 2

Is there a best practice that I am not aware of?

There is at least a standard practice that you seem not be aware of:

$k$-fold cross validation results (if used for validation as opposed for training purposes, see below) are used as an approximation to measuring the generalization error of the one model fit using the whole data as training set. So that one model is the one you use for predicting new cases.

You may want to read up on the fundamentals of model validation, and how cross validation fits into that framework.

---

I believe I am not allowed to assume that the distributions of the classifier output variable on, for example, signal, is the same among the k test samples of the k folds.

In the case of a model fully trained to output the labels you'd be wrong. However, you seem to be talking here about some intermediate scores, on which further calculations (thresholding) are done. For such intermediate scores, it depends on the particular algorithm you use to produce them whether they will/should already be calibrated to some numerical (?) range with interpretable meaning or not.
It may help to think whether there are (and which) changes to these internals of the model to which prediction is invariant (such as the flipping/mirror invariance of PCA)

I then apply a cut on the classifier output variable in my labelled data. Because the data is labelled, I can measure the background and signal efficiency of that cut. I am applying the same cut on all data points, no matter which classifier was used to determine the classifier output. This of course is not optimal unless the output distributions of the k classifiers are similar.

This means that you are using your cross validation as part of the model training process: to determine the optimal cut point. In that case, a) you need to do an independent validation of the model you get by this optimization process (e.g. by nested cross validation), and b) I'd put it the other way round: in order for your approach of for finding the threshold to work, you need to use a classifier that produces calibrated intermediate output.