# K-Nearest neighbour vs Gaussian classifier

I'm implementing an algorithm for doing dimensionality reduction: I'm going from a high dimennsional space (images) to a L dimensional space (where L is the number of classes). On this space I'm applying a classifier. In order to test my algorithm, I'm applying it on the MNIST dataset. The two different classifier I'm using are Gaussian classifier and K nearnest neighbours.

By Gaussian Classifier I'm using this, where I have K different multivariate gaussian: one for each class label l_k. The learning part of the Gaussian Classifier is learning the shape of the gaussian (mean and covariance matrix) on the training set, and then it label a point $x$ in the test set according to the maximum probability: $$P(l_k|x) = \frac{p(x|l_k)p(l_k)}{\sum_{i=1}^K{p(x|l_i)}}$$

Surprisingly, in all the experiment I did so far, have seen the kNN outperform Gaussian classifier! By better performance I simply mean the ratio of correctly classified sample vs mis-classified ones.

def evaluate_me(y_pred, test_labels):
r=0
w=0
for i in range(len(test_labels)):
if y_pred[i] == test_labels[i]:
r+=1
else:
w+=1
print("Tested ", len(test_labels), " digits")
print("Correct: ", r, "wrong: ", w, "error rate: ", float(w)*100/(r+w), "%")
print("Got correctly ", float(r)*100/(r+w), "%")
return

print("Trasforming training set")
Y_train = dim_red.transform(X=x_train)

print("Transforming test test")
Y_test= dim_red.transform(X=x_test)

print("Done")

knn_test_labels = classify_knn(Y_train, Y_test, train_labels)
gc_test_labels = classify_gaussian(Y_train, Y_test, train_labels)

print("kNN evaluation")
evaluate_me(knn_test_labels, test_labels)
print("Gaussian evaluation")
evaluate_me(gc_test_labels, test_labels)


Do you have any explanation for this? To me it's sounds strange, since I expect the GC to use more information on the dataset than the plain KNN, since it takes care of different variance on the multiple dimenisons, and the a prori probability of each class.

In particular, can you think of a dataset that perform better with KNN than GC

• What constitutes better performance? How are classifiers used for dimensionality reduction. What are the results from your code? May 19, 2017 at 14:54
• Could you elaborate on the Gaussian classifier? Is this a mixture of gaussians? If yes, please elaborate on how it classifies. For example KNN classifies in accordance to the closest centroid. May 19, 2017 at 17:22