2
$\begingroup$

I am trying to construct confusion matrices in R with CARET package for repeated K-fold cross-validation, specifically, 10-fold cross-validation with 10 repeats.

I realized there was already a similar question asked on this topic here: How is the confusion matrix reported from K-fold cross-validation?

However, the answer to the above question does not really deal with repeated cross-validation. The solution I have come up with is to use the average of the repeated predicting probability of an observation, i.e. for each repetition, each data point is being predicted once so the final prediction of that data point is obtained by looking at the average predicting probability from all 10 repetitions.

Does this make any logical sense? Also, if this is a sensible way of doing things, how should the confidence interval of the accuracy be constructed? Furthermore, how to compare model performances based on this?

I also just realized a potential concern of the method described above.

Suppose we have a total of $N$ observations and $n$ repetitions of K-fold cross-validation. We also assume we are working with binary classification where the positive case is denoted by 1 and the negative case is denoted by 0. Multilabel classification can be done in an analogous fashion. Let $y_i$ denote the observed label of the $i$-th observation, $i = 1,..., N$, and $\hat{p}_{ij}$ be the predicted probability of the $i$-th observation being the positive case for the $j$-th repetition, $j = 1,...,n$. The final predicted label of the $i$-th observation is then $$ \displaystyle \hat{y}_i = \mathbf{1}\{n^{-1}\sum_{j=1}^{n}\hat{p}_{ij}>1/2\}, $$ i.e. the final label is the average predicted probability of each observation from all repetitions after thresholding.

Consequently, the overall estimated accuracy (from the confusion matrix) will then be $$ \displaystyle \text{Acc} = N^{-1}\sum_{i = 1}^{N} \hat{y}_i. $$ This form of "ensemble" is for the purpose of ease of interpretation and the construction of sensible confusion matrices. However, there could be drawback to this approach as we would also like to have an estimate of the variance of the overall accuracy. "Collapsing" each repetition and obtaining one final label will make it very difficult to obtain an estimate of the variance. To estimate the variance, we will need to consider the overall accuracy of each repetition, i.e. for the $j$-th repetition $$ \displaystyle \text{Acc}_j = N^{-1}\sum_{i = 1}^{N}\mathbf{1}\{\hat{p}_{ij}>1/2\}, $$ and the estimated variance will then be $$ \displaystyle \text{Var} = \sum_{j = 1}^{n}(\text{Acc}_j-\overline{\text{Acc}})^2/(n-1), $$ where $$ \displaystyle \overline{\text{Acc}} = n^{-1}\sum_{j = 1}^{n}\text{Acc}_j. $$ Clearly, $$ \displaystyle \text{Acc} \neq \overline{\text{Acc}}. $$ However, we would hope that these two quantities are relatively close. Ideally, one would want to show that mathematically these two quantities all converge to the true accuracy in a limiting situation. Can this be proven in any way?

$\endgroup$
0
$\begingroup$

The way I usually do this is to stack the test folds (so you get one test prediction for the entire dataset) and ensemble the train folds. If your classifier generates probabilities, this is often done by averaging the probabilities and then thresholding to get the labels. Otherwise, with a binary classifier, it’s common to use a majority rule (and with 10-fold cross-validation, you’ll have 9 train predictions per dataset entry, so majority rule will give no ties).

You compare models with their test set evaluation metric, and you use the train and test evaluation metrics to guard against overfitting.

Confidence intervals for prediction are commonly obtained through bootstrapping, but you can also do it when cross-validating. See for example https://lagunita.stanford.edu/c4x/HumanitiesScience/StatLearning/asset/cv_boot.pdf

$\endgroup$
  • $\begingroup$ Hi @Michael, thanks for your reply. Regarding your first paragraph, I am not exactly sure what you meant. What do you mean by stacking test folds and ensembling train folds? Also, for your information, I am not splitting my dataset into a training set and a test set. I am using the entire data to train the models and trying to get a sense of performance from the validation data. That is why I try to construct confusion matrices from cross-validation. $\endgroup$ – davidolohowski Jun 2 '18 at 3:08
  • $\begingroup$ Hi @DavidLeigh, 10-fold cross-validation splits the data set into 10 “folds”, and then iterates through those folds, each time 1 fold is the “test” set and the other 9 folds are the “train” set. Here’s an example of R code that does some of what I described and computes the confidence interval using the standard deviation. milanor.net/blog/… $\endgroup$ – Michael Brundage Jun 2 '18 at 3:40
  • $\begingroup$ Hi @Michael, I understand very well what cross-validation does and how it works. I also understand how to construct confusion matrix from cross-validated data. The confusion I had with your method is how come there are 9 predictions? My understanding is that each fold is used to construct a confusion matrix and the total of all 10 fold is the confusion matrix for the whole dataset. And from what you are saying, it seems you are trying to use training data to construct confusion matrix which I think is not correct. Please explain if I misunderstood what you meant. $\endgroup$ – davidolohowski Jun 2 '18 at 4:21
  • $\begingroup$ And my main problem is regarding the issue of constructing confusion matrices with repetition. I just need to know if it is legitimate to construct a confusion matrix by doing what I described. $\endgroup$ – davidolohowski Jun 2 '18 at 4:23
  • $\begingroup$ Hi @DavidLeigh, there is some confusion here :) with each iteration, you get both a train prediction (on the same data used to train the model) and a test prediction (on the held out fold). The test predictions are non-overlapping and cover your data once. The train predictions partially overlap and cover your data 9 times (k-1, for k-fold validation). $\endgroup$ – Michael Brundage Jun 2 '18 at 4:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.