3
$\begingroup$

When I am using k-folding to split my labelled data (labelled as signal or background) and train k classifiers on it, I believe I am not allowed to assume that the distributions of the classifier output variable on, for example, signal, is the same among the k test samples of the k folds.

I am not talking about statistical fluctuations here, but about the fact that there is no assurance on how the classifier output variable is scaled.

Is that right? Or is there such an assurance for, say, scikit learn?

So, now I have my data set with the labelled data and the classifier output written in one column. Depending on the row the right one of the k classifiers was used to calculate a classifier output on that event.

I then apply a cut on the classifier output variable in my labelled data. Because the data is labelled, I can measure the background and signal efficiency of that cut. I am applying the same cut on all data points, no matter which classifier was used to determine the classifier output. This of course is not optimal unless the output distributions of the k classifiers are similar.


Now, I also have some unlabeled data which was therefore not used in the k-folding.

I am not sure which of the k classifiers to use to determine the classifier output on unlabeled data. It is tempting to just use any of them, for example the first one.

But because the output distributions of the k classifiers might differ, I can not a priory assume that the signal or background efficiency of applying my cut on the response of the first classifier only is the same as the efficiency that I estimated on my unlabeled data, where I also used the k-1 other classifiers.

I was thinking I could randomly split my unlabeled data into k equal data sets and and use a different one of the k classifiers on each. This way I am mirroring what happened on the labelled data, which should lead to the same signal and background efficiency, even if the classifier output distributions differ among the k classifiers.

Do you agree that this is the right approach?

Is there a best practice that I am not aware of?

EDIT: I should probably mention that I am talking about the "k-folding" method as it is used in High Energy Physics. The accepted answer explains what is meant by this in the first paragraph.

$\endgroup$

2 Answers 2

1
$\begingroup$

(answer was updated to mention HEP)

Given that question author's background I'm assuming this question is about high energy physics (HEP).

In HEP we need unbiased quality estimation and we want to make use of all available data (since for many analyses amount of available data is very limited) at the same time. That is why in HEP we often use k-folding scheme:

  1. divide data into k folds
  2. train k classifiers on k-1 folds of data
  3. predict i-th fold of data by i-th classifier, which is not trained on this i-th fold

Then using the same data, which now are marked, we estimate quality (we want to preserve as much as possible statistics for estimation, that is why i.e. we don't divide data into training and test parts). After that we need to apply the model to data from collider to compare quality: expected and observed (we test hypothesis if we have in data from collider searched decay). The answer below should be considered in the context of HEP above problem.

If you use stable models and have enough data samples then the distributions of the classifiers' outputs for signal and background events will be very similar across all folds (assuming you use the same parameters for all classifiers in the folding scheme).

During new data prediction, which didn't take part in the training process, it is a bad idea just to use any of the classifiers from folding scheme. In this case efficiency for chosen threshold will be overestimated or underestimated (due to the reasons you named). There are two ways of predicting new data:

  1. Predict a sample by all classifiers in the folding scheme, take the average. This gives you reliable stable prediction for a new sample.
  2. Take a random trained classifier from the folding scheme set of the classifiers. This is equivalent to divide new data into k-folds and predict each fold by only one classifier, like the training sample.

In my practice in rare decays analyzes to preserve data as much as possible we use folding scheme and predict new data the second way. In this case the same model is applied to the training and test sample and thus we will have correct quality estimation on the test sample for different measures like efficiency (thus single threshold is chosen for all classifiers).

$\endgroup$
3
  • $\begingroup$ sorry to -1, but strategy 1 yields the prediction of a so called aggregated classifier which is not the same as the "normal" (unaggregated) classifier and has quite different behavior in practice as well (unless you are using it in a situation where aggregation is not needed). Strategy 2 is also non-standard and also likely to be worse than the standard approach of doing predictions using a model trained on all data that was used for the cross validation. $\endgroup$
    – cbeleites
    Commented Jul 23, 2016 at 7:13
  • $\begingroup$ and has quite different behavior in practice as well - which is said in my answer. Standard technique (cross-validate and retrain on the whole sample) is not applicable in particle physics (this wasn't said explicitly in the question, but author is physicist), because to measure different physics parameters we need unbiased quality estimation (not lower bound estimation, as in the standard technique). This is done reliably and on the whole sample using the first approach. I will update my answer. $\endgroup$ Commented Jul 24, 2016 at 16:36
  • $\begingroup$ The question was indeed coming from a High Energy Physics (HEP) background. I guess I should have made that clear in the question. $\endgroup$ Commented Jul 25, 2016 at 7:52
2
$\begingroup$

Is there a best practice that I am not aware of?

There is at least a standard practice that you seem not be aware of:

$k$-fold cross validation results (if used for validation as opposed for training purposes, see below) are used as an approximation to measuring the generalization error of the one model fit using the whole data as training set. So that one model is the one you use for predicting new cases.

You may want to read up on the fundamentals of model validation, and how cross validation fits into that framework.


I believe I am not allowed to assume that the distributions of the classifier output variable on, for example, signal, is the same among the k test samples of the k folds.

In the case of a model fully trained to output the labels you'd be wrong. However, you seem to be talking here about some intermediate scores, on which further calculations (thresholding) are done. For such intermediate scores, it depends on the particular algorithm you use to produce them whether they will/should already be calibrated to some numerical (?) range with interpretable meaning or not.
It may help to think whether there are (and which) changes to these internals of the model to which prediction is invariant (such as the flipping/mirror invariance of PCA)

I then apply a cut on the classifier output variable in my labelled data. Because the data is labelled, I can measure the background and signal efficiency of that cut. I am applying the same cut on all data points, no matter which classifier was used to determine the classifier output. This of course is not optimal unless the output distributions of the k classifiers are similar.

This means that you are using your cross validation as part of the model training process: to determine the optimal cut point. In that case, a) you need to do an independent validation of the model you get by this optimization process (e.g. by nested cross validation), and b) I'd put it the other way round: in order for your approach of for finding the threshold to work, you need to use a classifier that produces calibrated intermediate output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.