K-NN optimal value of 'K'

1. How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score?
2. Do I need to plot AUC score vs k on both train & cross-validation data?

$$k$$ in $$k$$-NN is a hyperparameter to tune. You can try rules of thumb, like $$k=\sqrt{n}$$, where $$n$$ is a sample size, but better plot $$k$$ vs error metric as computed on validation set. The choice of the metric will depend on your needs, it can be any metric, or combination of metrics, that is appropriate for your problem (e.g. precision, recall, AUC).
Since it is quite primitive algorithm, that simply does majority votes (or averages in $$k$$-NN regression) given the nearest neighbors, it will overfit with smaller values of $$k$$, so this is something you should consider. On another hand, plotting training set error in here would show you exactly this relation and would lead to pretty obvious conclusions, so while people usually plot it, it isn't that helpful as with more advanced algorithms.
• @user20160 you can calculate fraction of each class among the $k$ nearest neighbors and you'd have the score (same thing happens in decision trees). But I'm not the greatest fan of AUC/ROC, so I probably wouldn't use it either. – Tim Jan 24 '19 at 11:58