4
$\begingroup$

I am trying to reach a stable model for my classification problem and I am using KNN for classification. In order to improve the accuracy I want to use bootstrap aggregation. In MATLAB I found a function "fitensemble". To my surprise in this function you cannot choose your learner as "KNN" and your ensemble learning method as "Bag". The only allowable ensemble learning method for KNN is "random subspace". I wonder if it is always like this and why? Am I missing something considering that I am quite new in this area.

Thanks in advance

$\endgroup$

2 Answers 2

4
$\begingroup$

You can use bootstrap aggregation (bagging) with any classifier! In fact, if you're crazy, you can bootstrap aggregate your bootstrap aggregates!

However, the author of the fitensemble function chose not to implement bagging for knn, which is too bad.

I don't know much about matlab, but you could try the ipredknn functions from the ipred package in R, which will allow you to bag your knn models.

$\endgroup$
1
  • $\begingroup$ In matlab, you can use create a KNN Ensemble an train it with subspace method. It's like bagging. $\endgroup$
    – hh32
    Oct 28, 2016 at 18:56
4
$\begingroup$

Late answer, but there's a reason why you generally don't want to bag a KNN classifier when "k" is small. Randomly sampling the training set with replacement gives you a new data set with about 62% unique training points -- this means that 62% of the time, a bootstrapped nearest neighbor classifier will give the exact same result.

So by the time we get to the aggregation step, the classifier outputs are super correlated, and a majority-vote step doesn't help much at all. On the other hand, it stands to reason that bagging may help when k is not small.

For the reason described above, kNN classifiers tend to do better by randomly sampling feature, rather than training, space. This is what the random subspace technique does. Google it!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.