I'm getting into ML, working through the book Machine Learning with R, the Tidyverse and MLR. Early on the concept of cross validation is introduced as a means to gauge the ability of my model to work with previously unseen data. Specifically, k-fold CV and leave-one-out CV are introduced using a kNN task on the diabetes dataset from the
The performance measure I'm using for CV is the average accuracy (fraction of test set which was correctly classified). Repeatedly evaluating my model performance wrt. k-fold CV returns different results as the data set is shuffled every time prior to CV.
Now, what I don't understand is why I get variations in model performance when using leave-one-out CV. As I perceive it:
- the randomness introduced by shuffling the dataset is of no consequence here as every single sample serves as test set exactly once and the order of samples is irrelevant for the average accuracy ( 1 + 0 + 1 == 0 + 1 + 1 ).
- the "training" and evaluation process of kNN is deterministic (ranking according to Eucl. distance)
Thus I'd naively expect zero variation when using leave-one-out CV. Where's my error?
Here's a MWE to reproduce my observation. Run it repeatedly and notice that the accuracy changes everytime.
library(tidyverse) library(mlr) # Configure mlr to shut up configureMlr(show.info=F) # Load data data(diabetes, package="mclust") diab <- diabetes %>% as_tibble # Configure task & learner diab_task <- makeClassifTask(data = diab, target = "class") diab_learner_knn <- makeLearner("classif.knn", par.vals=list("k" = 2)) # Leave-one-out CV LOO <- makeResampleDesc(method = "LOO") LOOCV <- resample(learner=diab_learner_knn, task=diab_task, resampling=LOO, measure=list(acc)) LOOCV$aggr