I'm getting into ML, working through the book Machine Learning with R, the Tidyverse and MLR. Early on the concept of cross validation is introduced as a means to gauge the ability of my model to work with previously unseen data. Specifically, k-fold CV and leave-one-out CV are introduced using a kNN task on the diabetes dataset from the mclust package.

The performance measure I'm using for CV is the average accuracy (fraction of test set which was correctly classified). Repeatedly evaluating my model performance wrt. k-fold CV returns different results as the data set is shuffled every time prior to CV.

Now, what I don't understand is why I get variations in model performance when using leave-one-out CV. As I perceive it:

  • the randomness introduced by shuffling the dataset is of no consequence here as every single sample serves as test set exactly once and the order of samples is irrelevant for the average accuracy ( 1 + 0 + 1 == 0 + 1 + 1 ).
  • the "training" and evaluation process of kNN is deterministic (ranking according to Eucl. distance)

Thus I'd naively expect zero variation when using leave-one-out CV. Where's my error?

Here's a MWE to reproduce my observation. Run it repeatedly and notice that the accuracy changes everytime.


# Configure mlr to shut up

# Load data
data(diabetes, package="mclust")
diab <- diabetes %>% as_tibble

# Configure task & learner
diab_task <- makeClassifTask(data = diab, target = "class")
diab_learner_knn <- makeLearner("classif.knn", par.vals=list("k" = 2))

# Leave-one-out CV
LOO <- makeResampleDesc(method = "LOO")
LOOCV <- resample(learner=diab_learner_knn, task=diab_task, resampling=LOO, measure=list(acc))
  • $\begingroup$ In the theoretical limit, results of different LOOCV converge. The theoretical regularity does not hold true for finite samples of data which will and do show differences between iterations. $\endgroup$
    – user234562
    Jul 27 '20 at 12:30
  • $\begingroup$ @user332577 I'm afraid I don't understand what you mean by 'theoretical limit'. The dataset is trivially small (145 observations) which allows me to compute the exhaustive LOOCV (equivalent to a 145-fold CV) on the full dataset. And by my above reasoning I'd expect to get the same result each and every time, which I don't. $\endgroup$
    – korolev
    Jul 27 '20 at 15:12
  • $\begingroup$ My understanding of LOOCV is that it's an iterative algorithm where the underlying data changes with each iteration. Have you 'fixed' those changes to be the same with each pass, drawn in the same sequence? In addition, are there outliers in your data? These could be compounding different results. $\endgroup$
    – user234562
    Jul 29 '20 at 12:23

You need to set the random seed to fix random initializations of the algorithm to get the same results:

LOOCV = resample(learner=diab_learner_knn, task=diab_task, resampling=LOO, measure=list(acc))
  • 1
    $\begingroup$ May I kindly ask you to point out to me where there is randomness in the computation of the accuracy? I recognize that the dataset is reshuffled but this is irrelevant for a LOOCV, no? $\endgroup$
    – korolev
    Jul 28 '20 at 11:48
  • $\begingroup$ I believe that this is in the initialisation of clusters in the k means implementation. $\endgroup$ Jul 28 '20 at 15:25
  • $\begingroup$ But I am not using k-means :( $\endgroup$
    – korolev
    Jul 29 '20 at 7:47
  • $\begingroup$ I'm sorry, I mean knn of course. In particular, ties are broken randomly by default. $\endgroup$ Jul 29 '20 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.