2
$\begingroup$

If I got an accuracy of 95% in the training set, and 80% accuracy in the test set, what could be the explanation for that? it sounds like a pretty basic question, but I just can't put it into words correctly.

It's supposed to be like that, I know that far. I interpret it like this:

" There is overfitting in the KNN model, the model learned the training data too well, and now it uses the features it learned in the training data in a false way. The model is familiar with the training data more, after all, it was trained on it, and the test data it has never seen before so it's only natural to get these results ".

What do you think of this explanation?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

If you use too small values for k (e.g. k=1) you get decision boundaries which are very erratic and that leads to overfitting. It means that if k is too small the decision boundaries depend highly on the sample that you have. If you had another sample (by chance) the decision boundaries would look very different.

Below are two pictures from the book "The Elements of Statistical Learning Data Mining, Inference, and Prediction" by Hastie, Tibshirani and Friedman illustrating this. Choosing larger k leads to more smooth decision boundaries and less variance in the model.

Whether a gap of 95% to 80% is reasonable depends on the sample size and the dimensionality of your data set I would say. In general though the gap will widen for smaller k and get smaller for larger k, but never disappear.

Smooth decision boundary Erratic decision boundary

Improve this answer
$\endgroup$
2
  • $\begingroup$ Please could you add the name(s) of the author(s) and, if available, a link to the book you reference - see stats.stackexchange.com/help/referencing for information about how to provide references. $\endgroup$
    – Lynn
    Commented Jul 17, 2022 at 12:01
  • $\begingroup$ Thanks for the suggestion. I've added the names of the authors. $\endgroup$
    – TuemmlerTanne11
    Commented Jul 17, 2022 at 20:28
-2
$\begingroup$

The explanation for this situation: your testing data contains features which are not in the training data (so the model see the new features as unknown and make wrong prediction)

You can use one (or both) of the methods bellow:

1-Split the data again but with random_state >80 according to your dataset

2-Resize the training and the testing set

if you have a large data you must use another algorithm KNN is bad for large data

Improve this answer
$\endgroup$
3
  • $\begingroup$ But is it always the case? For example, I split the data into 80% train and 20% test. It is the same data with the same features but the accuracy in the training is higher. For example consider this: classifying a dog playing in the snow to be a wolf, because almost all wolf pictures are in the snow. That's what I mean, they're the same features but the classifications are wrong. Why would this happen in your opinion? $\endgroup$
    – CORy
    Commented Jul 16, 2022 at 20:51
  • $\begingroup$ In case classifying a dog playing in the snow, this depend on the training test if you have a lot of images that contains wolf playing in snow so the classifier will make a wrong prediction with dog playing in snow, to avoid this you have to put a dog and a wolf playing in snow in the training set this maybe will help the classifier to find the right class. You can think like this you, you teach a children that wolfs always be in snow and dogs be in roads and farms and..., so when the children see from a far place a dog in snow he will say that is a wolf. $\endgroup$
    – Levent Özdemir
    Commented Jul 17, 2022 at 7:20
  • $\begingroup$ if you have 4 observations in the test set and you get %75 accuracy this meaning that the classifier predict 3 true and 1 false (this depend on the metric you use). if you have a different special features you have to select the train and the test sets manually, don't use train_test_split this will make it randomly and maybe all the special features will be in the test set so the classifier will make a wrong prediction. $\endgroup$
    – Levent Özdemir
    Commented Jul 17, 2022 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.