2
$\begingroup$
set.seed(522) 
model <- train(Class~.,data = network_measures,method = 'knn',
           trControl = trainControl(method = 'LOOCV'),
           preProcess = c('center','scale'),
           tuneGrid = expand.grid(k = 1:30))
k <- model$bestTune[[1]]
accuracy <- model$results[k,2]

Hello!

So as you can see here I am searching for the best k. My dataset is 39 subjects so my k can range from 1 to 38. If I run the script with k = 1:38 I get 82% accuracy with k = 4. If I run the script with k = 1:30 I get 69% accuracy with k = 2. If I run the script with k = 1:32 I get 74% accuracy with k = 4. Is this normal? I would expect to always have 82% accuracy as long as k = 4 was included.

$\endgroup$
4
  • $\begingroup$ There is probably some random selection if there are two neighbours tied in distance. So if you have two records that are the same distance away from the record you are classifying the algorithm has to pick one and this is usually done randomly $\endgroup$
    – astel
    Oct 28 '20 at 22:00
  • $\begingroup$ I see... And is there any way to check for sure if this is the case? Except from finding all distances manually of course $\endgroup$
    – Orestis
    Oct 28 '20 at 22:09
  • $\begingroup$ Within caret I’m not sure. Is there a way for you to see what records got what prediction? You could find the records that get different predictions between the two tests and investigate them $\endgroup$
    – astel
    Oct 28 '20 at 23:20
  • $\begingroup$ Yes finding the predictions for each record is possible. Good idea, thanks! $\endgroup$
    – Orestis
    Oct 29 '20 at 7:54
0
$\begingroup$

Setting a seed at the start is not enough if you are testing different number of k each time. You need to set the seed for each model trained. So if you are testing k = 1:8, your data has n rows, according to the help page (?trainControl) you need:

The list should have ‘B+1’ elements where ‘B’ is the number of resamples, unless ‘method’ is ‘"boot632"’ in which case ‘B’ is the number of resamples plus 1. The first ‘B’ elements of the list should be vectors of integers of length ‘M’ where ‘M’ is the number of models being evaluated. The last element of the list only needs to be a single integer (for the final model).

So I use an example dataset below:

library(caret)
library(MASS)

dat = Pima.tr
# generate seed list for up to k = 2000
set.seed(111)
kseeds=lapply(1:nrow(dat),function(i)sample(2000,k))

Write a function to select the first k seeds

s = function(seeds_list,k){
seeds_list = lapply(seeds_list,"[",1:k)
seeds_list[[length(seeds_list)+1]] = 999
seeds_list
}

We train with k = 5 :

k = 5
model <- train(type~.,data=dat,method="knn",
trControl = trainControl(method = 'LOOCV',
seeds=s(kseeds,k)),tuneGrid = expand.grid(k = 1:k))

model$results
  k Accuracy     Kappa
1 1    0.710 0.3491921
2 2    0.675 0.2784192
3 3    0.710 0.3249534
4 4    0.730 0.3667917
5 5    0.735 0.3761770

Now train with k = 10, you can see the Accuracy are the same now:

k = 10
model <- train(type~.,data=dat,method="knn",
trControl = trainControl(method = 'LOOCV',
seeds=s(kseeds,k)),tuneGrid = expand.grid(k = 1:k))

model$results
    k Accuracy     Kappa
1   1    0.710 0.3491921
2   2    0.675 0.2784192
3   3    0.710 0.3249534
4   4    0.730 0.3667917
5   5    0.735 0.3761770
6   6    0.700 0.3016760
7   7    0.720 0.3433396
8   8    0.710 0.3147448
9   9    0.725 0.3526365
10 10    0.745 0.3951613

And the model will choose the one with best accuracy. As you can also see, there will be differences based on your seed.. So I would actually be quite careful of using loocv here. Cross-validation might give you a better estimate of this uncertainty.

$\endgroup$
1
  • $\begingroup$ Yes, this did the trick! Thank you! $\endgroup$
    – Orestis
    Oct 29 '20 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.