1
$\begingroup$

I'm performing a classification task using KNN and PCA to pre-process the data.

The dataset contains 101 continuous variables and the column of the labels (here the link to download the data filebin.net/fy2238g063hhijsf)

> dim(train)
[1]  33 102
> unique(train[,1])
[1] Crete      Peloponese Other     
Levels: Crete Other Peloponese

I've applied PCA on the dataset as below:

> train.pca <- prcomp(train[,-1],center = TRUE,scale. = TRUE)
> summary(train.pca)
Importance of components:
                          PC1    PC2    PC3    PC4     PC5    PC6     PC7    PC8     PC9    PC10    PC11
Standard deviation     6.2499 5.5934 4.3538 2.5857 1.53128 1.1457 0.88391 0.5223 0.37085 0.27148 0.20914
Proportion of Variance 0.3867 0.3098 0.1877 0.0662 0.02322 0.0130 0.00774 0.0027 0.00136 0.00073 0.00043
Cumulative Proportion  0.3867 0.6965 0.8842 0.9504 0.97360 0.9866 0.99433 0.9970 0.99840 0.99912 0.99956
                          PC12    PC13    PC14    PC15    PC16    PC17    PC18    PC19    PC20    PC21
Standard deviation     0.10622 0.09947 0.07317 0.06188 0.05741 0.04700 0.04026 0.03675 0.03154 0.03029
Proportion of Variance 0.00011 0.00010 0.00005 0.00004 0.00003 0.00002 0.00002 0.00001 0.00001 0.00001
Cumulative Proportion  0.99967 0.99977 0.99982 0.99986 0.99989 0.99991 0.99993 0.99994 0.99995 0.99996
                          PC22    PC23    PC24    PC25    PC26    PC27    PC28    PC29    PC30    PC31
Standard deviation     0.02676 0.02422 0.02301 0.01986 0.01969 0.01836 0.01757 0.01506 0.01304 0.01241
Proportion of Variance 0.00001 0.00001 0.00001 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000
Cumulative Proportion  0.99997 0.99997 0.99998 0.99998 0.99999 0.99999 0.99999 1.00000 1.00000 1.00000
                          PC32      PC33
Standard deviation     0.01016 3.344e-13
Proportion of Variance 0.00000 0.000e+00
Cumulative Proportion  1.00000 1.000e+00

and I chose to keep only 4 principal components. I then calculated the principal components scores for measures in the validation set.

validation.pca <- predict(train.pca,newdata = validation[,-1])

I then tried a few values of the parameter k to fit a KNN model.

> set.seed(1234)
> #tune k using transformed data
> ccr.tnx <-numeric(25)
> for(j in 1:25)
+ {
+   pred.class.tnx<-knn(train.pca$x[,1:4], 
+                   validation.pca[,1:4], 
+                   train[,1], 
+                   k=j)
+   ccr.tnx[j]<-sum((pred.class.tnx==validation[,1]))/length(pred.class.tnx)
+   print(ccr.tnx[j])
+ }
[1] 0.8823529
[1] 0.8823529
[1] 0.7058824
[1] 0.8235294
[1] 0.8235294
[1] 0.8235294
[1] 0.6470588
[1] 0.7647059
[1] 0.7058824
[1] 0.6470588
[1] 0.8235294
[1] 0.7647059
[1] 0.7647059
[1] 0.7058824
[1] 0.7647059
[1] 0.6470588
[1] 0.6470588
[1] 0.7647059
[1] 0.7647059
[1] 0.7058824
[1] 0.6470588
[1] 0.5882353
[1] 0.5294118
[1] 0.7058824
[1] 0.7058824

From the result above I chose to use k=2 with 

> ccr.tnx[2]
[1] 0.8823529

I then tried to fit the model again with k=2 as below

> set.seed(1234)
> pred.class.tnx.2<-knn(train.pca$x[,1:4], 
+                     validation.pca[,1:4], 
+                     train[,1], 
+                     k=2)
> sum((pred.class.tnx.2==validation[,1]))/length(pred.class.tnx.2)
[1] 0.6470588

But I get a different correct classification rate! How is this possible?

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ Have you tried setting the random seed inside the for loop? $\endgroup$
    – Tim
    Commented Jun 16, 2019 at 11:02
  • $\begingroup$ @Tim you are my saviour! that was the problem, with the seed set inside the loop the results are consistent. $\endgroup$
    – lalessandro
    Commented Jun 16, 2019 at 11:09
  • $\begingroup$ Notice that there is 23% discrepancy between runs with two seeds. Unwillingly you cross-validated your results with different training and validation sets and it shown that there is a big variability in your results, so they are probably not very trustworthy. $\endgroup$
    – Tim
    Commented Jun 17, 2019 at 6:07
  • $\begingroup$ @Tim could you expand a bit on your last point? I'm not entirely sure I understand why I have used different training and validation sets. Following your suggestion now I get the same result within the loop and outside of it. $\endgroup$
    – lalessandro
    Commented Jun 17, 2019 at 6:50
  • $\begingroup$ Sorry, I didn't use this kind of stuff in R for a while. In knn there's not much randomization going on, do I assumed it randomly splits your data. This doesn't seem to be the case, but nonetheless, if changing seed has that profound impact, this suggests problems. $\endgroup$
    – Tim
    Commented Jun 17, 2019 at 7:11

1 Answer 1

Reset to default
2
$\begingroup$

As I said in the comment, the problem with your code was that you used a single seed at the beginning of the for-loop, so at each step after the random number generator got called, the seed incremented. The solution would be either to manually set the seeds at each step, or at each step record the seed that was used.

The bigger problem, not mentioned by you, is however that when you unintentionally used different seeds for different runs of the algorithm, you validated the algorithm's sensitivity to the seeds. $k$-NN algorithm does not really use randomization, the only point where it comes to the algorithm is when breaking the ties. If different seeds give you results that can differ by 23%, then this means that it is widely unstable. This discrepancy means that the algorithm has settled in local minimum, so the recommended solution would usually be to use some kind of regularization (in $k$-NN this would mean using higher number of neighbours $k$). Of course, in some cases, you can treat random seed as a hyperparameter to tune, but since this is a clear signal of overfitting, it should be carefully considered, and used after trying more standard approaches like regularization.

Improve this answer
$\endgroup$
9
  • $\begingroup$ I understand your point now Tim, thanks. What I had in mind though was to calculate the error rate for different values of k on the validation set and for different seeds. Then I was thinking to take the average error rate for a given k using the error rates obtained with different seeds. Finally choosing k with the lowest average error rate. Would you say that it's a reasonable way of going about it? $\endgroup$
    – lalessandro
    Commented Jun 18, 2019 at 19:12
  • $\begingroup$ @alessandro but then you would be using the average of the predictions over different seeds as your final prediction? Otherwise, the method would not be consistent with your usage. On another hand, if you did that, this would be very computationally intensive--why not consider other method? $\endgroup$
    – Tim
    Commented Jun 18, 2019 at 20:19
  • $\begingroup$ I would take the averages over different seeds just to tune the parameter k that minimises the misclassification rate on the validation set. would you agree with that approach? $\endgroup$
    – lalessandro
    Commented Jun 18, 2019 at 20:26
  • $\begingroup$ @alessandro but the average tells you nothing, e.g. if you pick the "worst" seed, the performance would be worst then the average you calculated... $\endgroup$
    – Tim
    Commented Jun 18, 2019 at 20:29
  • $\begingroup$ it makes sense, but what about if I don't set the seed and let R pick one at 'random'? $\endgroup$
    – lalessandro
    Commented Jun 18, 2019 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.