Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
To plot the ROC curve for the test set using k-fold cross-validation, the common approach is to plot k different ROC curves, one for each fold, and then obtain their average, as suggested in sklearn website.
I want to follow a different approach: Let's concatenate the predictions obtained for the test sets in different folds, and plot an ROC plot on the whole set. Does this approach make sense?
Your approach is sensible as well if the predicted scores are on the same scale for all $k$ surrogate models.
However, not all classifiers that predict continuous scores do also predict them on a common scale across models. If the scores do not share a common scale, you need to think in detail whether and how pooling is appropriate.
In sklearn terms,
decision_function() output always allows to generate per-surrogate-model ROCs, but not necessarily a pooled ROC.predict_proba() output always allows generating both a pooled ROC and per-surrogate-model ROCs.To illustrate the difference: consider doing an LDA with 2 classes. (You could consider SVM with or without Platt scaling instead)
Posterior probabilities predicted by the k surrogate models can easily be pooled into a single ROC.
The posterior probability threshold for classification directly corresponds to a threshold for the predicted LD score for each surrogate model. However, between the surrogate models the direction of the LD axis can flip* without affecting the predicted posterior probability. Pooling these predicted scores would not be sensible, although they can be used to generate ROCs for each surrogate model separately.
You may post-process such scores to align them onto a common axis using the characteristics of the classifier at hand without that axis having the meaning of posterior probability. E.g. for the LDA, you may flip the LD axis (and fix the origin if needed). The result of that would be equivalent LDA models that allow pooling their LD scores into a single ROC, but the LD scores are not probabilities.
* Depending on implementation details, the LD axis origin may also shift without affecting the predicted posterior probability.
Update wrt comment
My concern is that, in a small sample size setting, the predicted probabilities obtained from the different surrogate models cannot be concatenated, since they are coming from models trained on different training subsets.
One of the main underlying assumptions of cross validation (or out-bootstrap in all its flavors) is that the training sets are sufficiently similar to the whole data set that each of the surrogate models can be used as approximation to the model built on the whole data set. There assumption that allows pooling the results of the surrogate models is slightly different: the surrogate models are considered sufficiently similar to each other.
In other words, if you think you cannot pool the CV results here, that implies that you think the underlying assumptions of the CV are so severely violated that the CV results cannot be used at all (except maybe establishing this violation).
If the training is unstable wrt the (slight) changes in the training subsets, that uncertainty generates additional variance in the predictions. While it depends on the actual figure of merit/measure of predictive performance, most of them penalize such variance, and correctly so: if the training is unstable on $\frac{k-1}{k}$ of the data, it is likely still unstable when training the final model on the whole data set. This additional variance is part of what could happen for the predictions of the whole-data model.
