While reading through KNN in detail, I was checking if there is a way to improve/penalize KNN? I didn't find any concrete/easily understandable solutions.
Good question. There are many forms of regularization: $\ell_1$, $\ell_2$, early stopping, dropout, etc. $k$NN is a nonparametric model, it doesn't have parameters to penalize, drop, or way to stop training earlier.
The main point of regularization is preventing overfitting. The way to prevent overfitting in $k$NN is to increase $k$ as it leads to averaging over many points instead of memorizing the training set as in $k=1$. It also makes the result smoother, because of averaging, which is another common consequence of regularization. A similar approach is used in decision trees where we restrict depth, minimal node size, or prune them, as means of regularization.
You can also use penalized distance metrics and in $k$NN regression more robust aggregation function, e.g. median, to make it less susceptible to outliers.