1. How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score?
  2. Do I need to plot AUC score vs k on both train & cross-validation data?

$k$ in $k$-NN is a hyperparameter to tune. You can try rules of thumb, like $k=\sqrt{n}$, where $n$ is a sample size, but better plot $k$ vs error metric as computed on validation set. The choice of the metric will depend on your needs, it can be any metric, or combination of metrics, that is appropriate for your problem (e.g. precision, recall, AUC).

Since it is quite primitive algorithm, that simply does majority votes (or averages in $k$-NN regression) given the nearest neighbors, it will overfit with smaller values of $k$, so this is something you should consider. On another hand, plotting training set error in here would show you exactly this relation and would lead to pretty obvious conclusions, so while people usually plot it, it isn't that helpful as with more advanced algorithms.

  • $\begingroup$ If a KNN classifier returns the most highly-voted class rather than a score, wouldn't it be the case that an ROC curve doesn't exist (so no AUC)? $\endgroup$
    – user20160
    Jan 24 '19 at 11:45
  • $\begingroup$ @user20160 you can calculate fraction of each class among the $k$ nearest neighbors and you'd have the score (same thing happens in decision trees). But I'm not the greatest fan of AUC/ROC, so I probably wouldn't use it either. $\endgroup$
    – Tim
    Jan 24 '19 at 11:58
  • $\begingroup$ That seems good. I will try this. $\endgroup$ Jan 24 '19 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.