am trying to learn KNN by working on Breast cancer dataset provided by UCI repository. The Total size of dataset is 699 with 9 continuous variables and 1 class variable.

I tested my accuracy on cross-validation set. For K =21 & K =19. Accuracy is 95.7%.

from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=21)
neigh.fit(X_train, y_train) 
y_pred_val = neigh.predict(X_val)
print accuracy_score(y_val, y_pred_val)

But for K= 1, I am getting Accuracy = 97.85% K = 3, Accuracy = 97.14

I read

Choice of k is very critical – A small value of k means that noise will have a higher influence on the result. A large value make it computationally expensive and kinda defeats the basic philosophy behind KNN (that points that are near might have similar densities or classes ) .A simple approach to select k is set k = n^(1/2). (Here.)

Which value of K should I consider for my model. Can you guys elaborate the logic behind it?

Thanks in advance!

  • 1
    $\begingroup$ You would consider k as a hyperparameter of your algorithm and chose the best possible hyperparameter by crossvalidation. There is no valid rule of thumb for chosing k, since the ideal k varies depending on domain and data. $\endgroup$ – Nikolas Rieble Dec 22 '16 at 9:44

Nikolas is right. The way to go about it is to do something like cross validation with different Ks, and chose the k that minimizes the cross validation error.

  • $\begingroup$ Hi @filipe I tested accuracy on cross-validation data and these results are tested on that data only. $\endgroup$ – Rahul Saxena Dec 22 '16 at 10:44
  • $\begingroup$ Hi Rahul. In that case, I guess it stands to reason, that K=1 is the value that leads to the best generalization accuracy. However, maybe another concern is the issue of class imbalance. If you have a class imbalance (one of the classes is far more frequent than others), then "Accuracy" may not be the best error metric. $\endgroup$ – f.g. Dec 22 '16 at 13:46

Given that its cancer data (which inherently has class imbalance), F1 score (not accuracy) will be the correct metric to use. Here's wiki for more on F1 score. It basically lets you chose how you want to relatively weigh Precision and Recall.

Small changes to K in training set can lead to large changes in decision boundary so you'll need to cross validate and see if the results generalize well.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.