# Why does the accuracy of leave-one-out CV change between runs for my kNN task?

I'm getting into ML, working through the book Machine Learning with R, the Tidyverse and MLR. Early on the concept of cross validation is introduced as a means to gauge the ability of my model to work with previously unseen data. Specifically, k-fold CV and leave-one-out CV are introduced using a kNN task on the diabetes dataset from the mclust package.

The performance measure I'm using for CV is the average accuracy (fraction of test set which was correctly classified). Repeatedly evaluating my model performance wrt. k-fold CV returns different results as the data set is shuffled every time prior to CV.

Now, what I don't understand is why I get variations in model performance when using leave-one-out CV. As I perceive it:

• the randomness introduced by shuffling the dataset is of no consequence here as every single sample serves as test set exactly once and the order of samples is irrelevant for the average accuracy ( 1 + 0 + 1 == 0 + 1 + 1 ).
• the "training" and evaluation process of kNN is deterministic (ranking according to Eucl. distance)

Thus I'd naively expect zero variation when using leave-one-out CV. Where's my error?

Here's a MWE to reproduce my observation. Run it repeatedly and notice that the accuracy changes everytime.

library(tidyverse)
library(mlr)

# Configure mlr to shut up
configureMlr(show.info=F)

data(diabetes, package="mclust")
diab <- diabetes %>% as_tibble

LOOCV$aggr  • In the theoretical limit, results of different LOOCV converge. The theoretical regularity does not hold true for finite samples of data which will and do show differences between iterations. – user234562 Jul 27 '20 at 12:30 • @user332577 I'm afraid I don't understand what you mean by 'theoretical limit'. The dataset is trivially small (145 observations) which allows me to compute the exhaustive LOOCV (equivalent to a 145-fold CV) on the full dataset. And by my above reasoning I'd expect to get the same result each and every time, which I don't. Jul 27 '20 at 15:12 • My understanding of LOOCV is that it's an iterative algorithm where the underlying data changes with each iteration. Have you 'fixed' those changes to be the same with each pass, drawn in the same sequence? In addition, are there outliers in your data? These could be compounding different results. – user234562 Jul 29 '20 at 12:23 ## 1 Answer You need to set the random seed to fix random initializations of the algorithm to get the same results: set.seed(123) LOOCV = resample(learner=diab_learner_knn, task=diab_task, resampling=LOO, measure=list(acc)) LOOCV$aggr