I'm implementing an algorithm for doing dimensionality reduction: I'm going from a high dimennsional space (images) to a L dimensional space (where L is the number of classes). On this space I'm applying a classifier. In order to test my algorithm, I'm applying it on the MNIST dataset. The two different classifier I'm using are Gaussian classifier and K nearnest neighbours.

By Gaussian Classifier I'm using this, where I have K different multivariate gaussian: one for each class label l_k. The learning part of the Gaussian Classifier is learning the shape of the gaussian (mean and covariance matrix) on the training set, and then it label a point $x$ in the test set according to the maximum probability: $$ P(l_k|x) = \frac{p(x|l_k)p(l_k)}{\sum_{i=1}^K{p(x|l_i)}}$$

Surprisingly, in all the experiment I did so far, have seen the kNN outperform Gaussian classifier! By better performance I simply mean the ratio of correctly classified sample vs mis-classified ones.

def evaluate_me(y_pred, test_labels):
    for i in range(len(test_labels)):
        if y_pred[i] == test_labels[i]:
    print("Tested ", len(test_labels), " digits")
    print("Correct: ", r, "wrong: ", w, "error rate: ", float(w)*100/(r+w), "%")
    print("Got correctly ", float(r)*100/(r+w), "%")

print("Trasforming training set")
Y_train = dim_red.transform(X=x_train)

print("Transforming test test")
Y_test= dim_red.transform(X=x_test)


knn_test_labels = classify_knn(Y_train, Y_test, train_labels)
gc_test_labels = classify_gaussian(Y_train, Y_test, train_labels)

print("kNN evaluation")
evaluate_me(knn_test_labels, test_labels)
print("Gaussian evaluation")
evaluate_me(gc_test_labels, test_labels)

Do you have any explanation for this? To me it's sounds strange, since I expect the GC to use more information on the dataset than the plain KNN, since it takes care of different variance on the multiple dimenisons, and the a prori probability of each class.

In particular, can you think of a dataset that perform better with KNN than GC

  • $\begingroup$ What constitutes better performance? How are classifiers used for dimensionality reduction. What are the results from your code? $\endgroup$ May 19, 2017 at 14:54
  • $\begingroup$ Could you elaborate on the Gaussian classifier? Is this a mixture of gaussians? If yes, please elaborate on how it classifies. For example KNN classifies in accordance to the closest centroid. $\endgroup$
    – Alex R.
    May 19, 2017 at 17:22


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.