KNN parameter tuning with cross validation: score draw

Question

I'm trying to use the KNN method for binary classification. When trying to find the best 'k' parameter (the amount of neighbours that the algorithm looks at) I train a model on my training set and look at its accuracy on a seperate validation set I got with my data. This validation set only has 12 samples, which causes a draw in accuracy for 3 k's (1,3,5).

Now I'm looking for a way to choose one of these 3 k's for the definitive model. I had the following approach in mind: for the 3 k's, I do K-fold cross validation for a certain K on the training set and then look which one has the best average accuracy here. Is this a decent approach, or are there better options? I also thought of just picking a random k (1, 3 or 5), because the 'validation procedure' tells me that I can choose any of the 3.

Answer 1

This problem occurs when you have a small test set, which can cause multiple models to tie, by achieving the same number of correct predictions.

The method you said first should do. Because in CV each model sees each training sample once, I'd consider it unlikely for your 3 models to have the same accuracy. If this persists, it is safe to choose at random (I'd go for 3 because it is the middle element)

Answer 2

Occam's principle suggest's that you should go for the simplest model possible. So you should go for that one. But to get a better idea of the model's generalization, i would suggest you to use nested cross validation.