2
$\begingroup$

While reading through KNN in detail, I was checking if there is a way to improve/penalize KNN? I didn't find any concrete/easily understandable solutions.

$\endgroup$
2
  • $\begingroup$ What do you mean by "improve"? $\endgroup$ Apr 24 at 9:31
  • $\begingroup$ One way of regularization that I can imagine is in the selection and weighing of the parameters that are used in the model. $\endgroup$ Apr 24 at 9:32

1 Answer 1

2
$\begingroup$

Good question. There are many forms of regularization: $\ell_1$, $\ell_2$, early stopping, dropout, etc. $k$NN is a nonparametric model, it doesn't have parameters to penalize, drop, or way to stop training earlier.

The main point of regularization is preventing overfitting. The way to prevent overfitting in $k$NN is to increase $k$ as it leads to averaging over many points instead of memorizing the training set as in $k=1$. It also makes the result smoother, because of averaging, which is another common consequence of regularization. A similar approach is used in decision trees where we restrict depth, minimal node size, or prune them, as means of regularization.

You can also use penalized distance metrics and in $k$NN regression more robust aggregation function, e.g. median, to make it less susceptible to outliers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.