0
$\begingroup$

I am trying to find the best k for my kNN algorithm. I am using K Fold CV and Stratified K fold CV, with K = 5, 10 & 20.

I run a K-Fold CV on my training dataset, for each k [1-50]. In the end, I have 50 cross-validated accuracy values (% of correctly classified labels). I only run the K-Fold CV once.

Then, I do the same thing, but instead of using K-Fold CV, I use Stratified K-Fold CV. In the end, I once again have 50 cross-validated accuracy values (% of correctly classified labels). I only run the Stratified K-Fold CV once.

Now, here are my questions:

  • If I am doing both K-Fold CV and Stratified K-Fold CV, which one should I choose? I am going with Stratified, as each fold has same proportions of classes, so it should be less biased and have less variance, but I am not sure.
  • Which K should I choose? My dataset is small (like 150 items), so should I choose the results from K = 10? As K = 5 will have higher bias, but lower variance and K = 20 will have lower bias, but higher variance, so I feel K = 10 is a good balance.
  • I have calculated the standard error for each K/k combo, for K = 5, 10 and 20, and k = 1 - 50, so in total I have 50 SE values for each K. I used the formula of sample standard deviation/sqrt(K) to do calculate the SE. But I am not sure how to use the value. For example, in Stratified K-Fold CV, the k with the highest cross-validated accuracy has the lowest SE.

In the end, for K-Fold CV, I have one k value for K = 5, 10 and 15 all three of them, that gives the highest cross-validated accuracy and lowest SE. For Stratified K-Fold CV, I have one k value for K = 5, 10 and 15 all three of them, that gives the highest cross-validated accuracy and lowest SE. The difference is that the best k obtained in K-Fold CV is different from the best k obtained from the Stratified K-Fold CV. Do I need to run my K-Fold CV multiple times and calculated the average cross-validated accuracy and average standard error? Does that even make sense i.e average of averages, as the cross-validated accuracy itself is an average of accuracies of the individual K folds.

When I test on the independent test data I have, the best k value given by K-Fold CV return 100% accuracy, where as the best k given by Stratified K-Fold returns 93% accuracy.

So I am a bit confused as to which value should I choose.

$\endgroup$

1 Answer 1

0
$\begingroup$

Your k in the kNN algorithm is a hyper parameter. You should not use the same K-folds to find out which k works best, and then report that same result. You should use a nested K-fold CV. For example, if your K= 10, then divide the dataset into 10 folds, take 9 fold for training, and 1 fold for testing. Then divide that 9 folds into 10-fold again. Use 9-folds to train, and the other one to test which value of k works best. Then train on all the 10 folds using that k, and test it on the initial test fold you made. This way you will avoid bias, and an over-optimistic result. For an unbalanced dataset, I would suggest using stratified K-fold.

$\endgroup$
5
  • $\begingroup$ I was following this great article: sebastianraschka.com/blog/2016/… Can I not use Stratified K fold CV on a balanced dataset? What about repeated K-Fold CV? $\endgroup$
    – coderrio
    Oct 26, 2017 at 22:25
  • $\begingroup$ Are you saying I need to change the hyper parameter in each fold when doing normal K-Fold CV? $\endgroup$
    – coderrio
    Oct 26, 2017 at 22:52
  • $\begingroup$ In a nested CV, you use the inner 10 folds to train and test for different k-s to find out which gives you the best performance, then use that k to train on the whole inner dataset, and then test just once on your outer test set to report the performance. I think you need to clear your concept of nested CV, then it will be easier for you to decide what to do. You can use stratified K-fold for balanced dataset too. $\endgroup$
    – nafizh
    Oct 26, 2017 at 23:15
  • $\begingroup$ Complicated. Can't I just use a Stratified K-Fold CV on my entire training dataset? Run it for multiple values of k and find the best k and use it + my entire training dataset to test the mode's performance on my independent dataset? $\endgroup$
    – coderrio
    Oct 27, 2017 at 0:45
  • $\begingroup$ If you are finding the best k using just your training set, then basically you are already doing a nested CV. So, what you described is right, if I understand it correctly. $\endgroup$
    – nafizh
    Oct 27, 2017 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.