1
$\begingroup$

With a binary classification analysis, I find myself with the exact same accuracy, recall and specificity for both a qda and glm model.

When I apply the same random seed to both classification models I get the same results, same when I apply a different random seed to each.

Anyone know why that is?

here's a reprex

   library(caret)

set.seed(73, sample.kind = "Rounding")

sonar_index <- createDataPartition(Sonar$Class, times = 1, p= 0.2, list = F)
sonar_train <- Sonar[-sonar_index,]
sonar_test <- Sonar[sonar_index,]

#glm
set.seed(73, sample.kind = "Rounding")

sonar_ctrl <- trainControl(method="cv", summaryFunction=twoClassSummary, classProbs=T,
                           savePredictions = T)

sonar_glm <- train(Class ~ V13 + V25, data = sonar_train, method = "glm", trControl = sonar_ctrl)

cm_sonar_glm <- confusionMatrix(predict(sonar_glm, sonar_test), sonar_test$Class)

results_glm <- tibble(model = "GLM",
                      accuracy = cm_sonar_glm$overall["Accuracy"],
                      sensitivity = cm_sonar_glm$byClass["Sensitivity"],
                      specificity = cm_sonar_glm$byClass["Specificity"],
                      f1_score = F_meas(predict(sonar_glm, sonar_test), sonar_test$Class))


#QDA
set.seed(73, sample.kind = "Rounding")

sonar_ctrl <- trainControl(method="cv", summaryFunction=twoClassSummary, classProbs=T,
                           savePredictions = T)

sonar_qda <- train(Class ~ V13 + V25, data = sonar_train, method = "qda", trControl = sonar_ctrl)

cm_sonar_qda <- confusionMatrix(predict(sonar_qda, sonar_test), sonar_test$Class)

results_qda <- tibble(model = "QDA",
                      accuracy = cm_sonar_qda$overall["Accuracy"],
                      sensitivity = cm_sonar_qda$byClass["Sensitivity"],
                      specificity = cm_sonar_qda$byClass["Specificity"],
                      f1_score = F_meas(predict(sonar_qda, sonar_test), sonar_test$Class))

The tables don't have the same result but with the exact same code structure like the one above, on my dataset it comes out as exactly the same

$\endgroup$
6
  • $\begingroup$ can you share the code used to run the training for both models? I think it's very unlikely $\endgroup$ – StupidWolf May 17 '20 at 12:06
  • 1
    $\begingroup$ Thank you for being willing to help, I edited my question with a reprex $\endgroup$ – bolleke May 17 '20 at 12:43
  • $\begingroup$ my first guess is your measure of performance is not very good - e.g. proportion of correct guesses is usually a bad metric. better to use Bernoulli log likelihood $\endgroup$ – probabilityislogic May 17 '20 at 13:10
  • $\begingroup$ there's a bug see cm_sonar_qda <- confusionMatrix(predict(sonar_glm, sonar_test), sonar_test$Class) $\endgroup$ – StupidWolf May 17 '20 at 13:38
  • $\begingroup$ also cars_index is used but I have no idea where that came from $\endgroup$ – StupidWolf May 17 '20 at 13:39
0
$\begingroup$

Never a good thing to repeat the same chunk of code twice. So if we train on accuracy, we set up like what you did:

library(mlbench)
library(caret)

data(Sonar)
set.seed(73, sample.kind = "Rounding")

sonar_index <- createDataPartition(Sonar$Class, times = 1, p= 0.2, list = F)
sonar_train <- Sonar[-sonar_index,]
sonar_test <- Sonar[sonar_index,]

Methods = c("glm","qda")

wrap up the test in a function:

res = lapply(Methods,function(i){

ctrl <- trainControl(method="cv", classProbs=T,
                           savePredictions = T)

fit <- train(Class ~ V13 + V25, 
 data = sonar_train, method = i, trControl = ctrl)

cm = confusionMatrix(predict(fit, sonar_test), sonar_test$Class)

results_glm <- data.frame(model = i,
                accuracy = cm$overall["Accuracy"],
                sensitivity = cm$byClass["Sensitivity"],
             specificity = cm$byClass["Specificity"],
             f1_score = F_meas(predict(fit, sonar_test), 
                        sonar_test$Class))

return(results_glm)
})

And I get the following results:

[[1]]
         model  accuracy sensitivity specificity  f1_score
Accuracy   glm 0.6511628   0.7391304        0.55 0.6938776

[[2]]
         model  accuracy sensitivity specificity f1_score
Accuracy   qda 0.5813953   0.6956522        0.45     0.64
$\endgroup$
6
  • $\begingroup$ Thank you for the tip on the function! I'll definitely use it. However, even when I apply your function to my dataset I still get the exact same results for everything. Is there a mathematical explanation behind this? $\endgroup$ – bolleke May 17 '20 at 14:16
  • $\begingroup$ what is the accuracy you are getting? and is your dataset imbalanced, like do you have 90% of one label $\endgroup$ – StupidWolf May 17 '20 at 14:17
  • $\begingroup$ 0.904 accuracy and the dataset is not imbalanced, aprox. 70-30 $\endgroup$ – bolleke May 17 '20 at 14:23
  • $\begingroup$ you have to check out why those that are predicted wrongly always fail in both models. it could be that they have some extreme values or some reason that leads to that $\endgroup$ – StupidWolf May 17 '20 at 14:49
  • $\begingroup$ hey it's really hard to tell without seeing the data. And obviously you can see from the example above, it really depends on your data $\endgroup$ – StupidWolf May 17 '20 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.