K nearest neighbors with nested cross validation

Question

I'm working on a binary classification problem on this dataset, using the k-nn algorithm.

For the performance evaluation and the parameter tuning (i.e. the choosing of k) I'm using the nested cross validation.

I split my dataset in 5 equal sized fold, and then I performed a cross validation for every training set/fold (i.e. I took from the fold the training set, on which I split it in 5 equal sized fold) for the k tuning. I've taken a specified set of values for the k tuning (1, 3, 5, 7, 9, 11, 13, etc)

For every nested cross validation I've taken the best k and I used it for the current fold evaluation. I've drawn an example schema for explanation:

I got, for example, these results (cross validation with 5 fold):

• First fold, best k = 11, accuracy = 0.785:
• Second fold, best k = 11, accuracy = 0.776:
• Third fold, best k = 11, accuracy = 0.786:
• Fourth fold, best k = 11, accuracy = 0.791:
• Fifth fold, best k = 9, accuracy = 0.793:

With an overall performance of 0.7853669 (accuracies mean)

Now, because I don't get for every fold the same best k (selection done with the inner cross validation), which k I need to use for my final model (the one which I will use for real classification)?

• makes it sense to use the mean of the best k?

• Or need I to do on all the dataset a inner cross validation for the final k selection? And saying that the expected performance will be the one evaluated with the nested cross validation?

Answer 1

I found out the response to my question

need I to do on all the dataset a inner cross validation for the final k selection? And saying that the expected performance will be the one evaluated with the nested cross validation?

• short answer: yes.

• long answer: The nested cross validation is needed to evaluate a process of learning and hyper parameters tuning, which means that at the end, if I want to select a k to use for my final model, I need to use my process of inner cross validation done on the different training sets obtained by the external cross validation split. The expected performance of this final model is what you evaluated with nested cross-validation earlier.

Answer 2

To put it more clearly, the outer loop of the nested cross validation in any 2-layer/nested cross validation, is used to give an "unbiased" estimate of the model's generalization error (expected value of test error with infinite test data, mathematicians please don't be mad! ) . This value of generalisation error, in the case of a knn classification algorithm happens to be linked trivially or linearly, to the accuracy of the model (because you define how you want to count the error, usually i would say, error=+1 for every misclassified sample) in the case of a knn, and thus gives you an idea of how well the model generalises, or whether or not to use this model or not in the future. Go to this link--> Nested cross validation for model selection for better analysis of what all things could be analysed from 2-layer/nested cross validation. So in layman's terms, nested CV will tell me, if i was to take this model out in the market, how well would it perform, and to tune the final hyperparameter, which is k, i just do the inner CV on all data, and churn out a K!