# R prcomp and KNN different correct classification rate

I'm performing a classification task using KNN and PCA to pre-process the data.

The dataset contains 101 continuous variables and the column of the labels (here the link to download the data filebin.net/fy2238g063hhijsf)

> dim(train)
[1]  33 102
> unique(train[,1])
[1] Crete      Peloponese Other
Levels: Crete Other Peloponese


I've applied PCA on the dataset as below:

> train.pca <- prcomp(train[,-1],center = TRUE,scale. = TRUE)
> summary(train.pca)
Importance of components:
PC1    PC2    PC3    PC4     PC5    PC6     PC7    PC8     PC9    PC10    PC11
Standard deviation     6.2499 5.5934 4.3538 2.5857 1.53128 1.1457 0.88391 0.5223 0.37085 0.27148 0.20914
Proportion of Variance 0.3867 0.3098 0.1877 0.0662 0.02322 0.0130 0.00774 0.0027 0.00136 0.00073 0.00043
Cumulative Proportion  0.3867 0.6965 0.8842 0.9504 0.97360 0.9866 0.99433 0.9970 0.99840 0.99912 0.99956
PC12    PC13    PC14    PC15    PC16    PC17    PC18    PC19    PC20    PC21
Standard deviation     0.10622 0.09947 0.07317 0.06188 0.05741 0.04700 0.04026 0.03675 0.03154 0.03029
Proportion of Variance 0.00011 0.00010 0.00005 0.00004 0.00003 0.00002 0.00002 0.00001 0.00001 0.00001
Cumulative Proportion  0.99967 0.99977 0.99982 0.99986 0.99989 0.99991 0.99993 0.99994 0.99995 0.99996
PC22    PC23    PC24    PC25    PC26    PC27    PC28    PC29    PC30    PC31
Standard deviation     0.02676 0.02422 0.02301 0.01986 0.01969 0.01836 0.01757 0.01506 0.01304 0.01241
Proportion of Variance 0.00001 0.00001 0.00001 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000 0.00000
Cumulative Proportion  0.99997 0.99997 0.99998 0.99998 0.99999 0.99999 0.99999 1.00000 1.00000 1.00000
PC32      PC33
Standard deviation     0.01016 3.344e-13
Proportion of Variance 0.00000 0.000e+00
Cumulative Proportion  1.00000 1.000e+00


and I chose to keep only 4 principal components. I then calculated the principal components scores for measures in the validation set.

validation.pca <- predict(train.pca,newdata = validation[,-1])


I then tried a few values of the parameter k to fit a KNN model.

> set.seed(1234)
> #tune k using transformed data
> ccr.tnx <-numeric(25)
> for(j in 1:25)
+ {
+   pred.class.tnx<-knn(train.pca$x[,1:4], + validation.pca[,1:4], + train[,1], + k=j) + ccr.tnx[j]<-sum((pred.class.tnx==validation[,1]))/length(pred.class.tnx) + print(ccr.tnx[j]) + } [1] 0.8823529 [1] 0.8823529 [1] 0.7058824 [1] 0.8235294 [1] 0.8235294 [1] 0.8235294 [1] 0.6470588 [1] 0.7647059 [1] 0.7058824 [1] 0.6470588 [1] 0.8235294 [1] 0.7647059 [1] 0.7647059 [1] 0.7058824 [1] 0.7647059 [1] 0.6470588 [1] 0.6470588 [1] 0.7647059 [1] 0.7647059 [1] 0.7058824 [1] 0.6470588 [1] 0.5882353 [1] 0.5294118 [1] 0.7058824 [1] 0.7058824  From the result above I chose to use k=2 with > ccr.tnx[2] [1] 0.8823529  I then tried to fit the model again with k=2 as below > set.seed(1234) > pred.class.tnx.2<-knn(train.pca$x[,1:4],
+                     validation.pca[,1:4],
+                     train[,1],
+                     k=2)
> sum((pred.class.tnx.2==validation[,1]))/length(pred.class.tnx.2)
[1] 0.6470588


But I get a different correct classification rate! How is this possible?

• Have you tried setting the random seed inside the for loop? – Tim Jun 16 '19 at 11:02
• @Tim you are my saviour! that was the problem, with the seed set inside the loop the results are consistent. – datapipe Jun 16 '19 at 11:09
• Notice that there is 23% discrepancy between runs with two seeds. Unwillingly you cross-validated your results with different training and validation sets and it shown that there is a big variability in your results, so they are probably not very trustworthy. – Tim Jun 17 '19 at 6:07
• @Tim could you expand a bit on your last point? I'm not entirely sure I understand why I have used different training and validation sets. Following your suggestion now I get the same result within the loop and outside of it. – datapipe Jun 17 '19 at 6:50
• Sorry, I didn't use this kind of stuff in R for a while. In knn there's not much randomization going on, do I assumed it randomly splits your data. This doesn't seem to be the case, but nonetheless, if changing seed has that profound impact, this suggests problems. – Tim Jun 17 '19 at 7:11

As I said in the comment, the problem with your code was that you used a single seed at the beginning of the for-loop, so at each step after the random number generator got called, the seed incremented. The solution would be either to manually set the seeds at each step, or at each step record the seed that was used.
The bigger problem, not mentioned by you, is however that when you unintentionally used different seeds for different runs of the algorithm, you validated the algorithm's sensitivity to the seeds. $$k$$-NN algorithm does not really use randomization, the only point where it comes to the algorithm is when breaking the ties. If different seeds give you results that can differ by 23%, then this means that it is widely unstable. This discrepancy means that the algorithm has settled in local minimum, so the recommended solution would usually be to use some kind of regularization (in $$k$$-NN this would mean using higher number of neighbours $$k$$). Of course, in some cases, you can treat random seed as a hyperparameter to tune, but since this is a clear signal of overfitting, it should be carefully considered, and used after trying more standard approaches like regularization.