# With two different random seeds for binary classification, I have the exact same result for GLM and QDA, what gives?

With a binary classification analysis, I find myself with the exact same accuracy, recall and specificity for both a qda and glm model.

When I apply the same random seed to both classification models I get the same results, same when I apply a different random seed to each.

Anyone know why that is?

here's a reprex

   library(caret)

set.seed(73, sample.kind = "Rounding")

sonar_index <- createDataPartition(Sonar$Class, times = 1, p= 0.2, list = F) sonar_train <- Sonar[-sonar_index,] sonar_test <- Sonar[sonar_index,] #glm set.seed(73, sample.kind = "Rounding") sonar_ctrl <- trainControl(method="cv", summaryFunction=twoClassSummary, classProbs=T, savePredictions = T) sonar_glm <- train(Class ~ V13 + V25, data = sonar_train, method = "glm", trControl = sonar_ctrl) cm_sonar_glm <- confusionMatrix(predict(sonar_glm, sonar_test), sonar_test$Class)

results_glm <- tibble(model = "GLM",
accuracy = cm_sonar_glm$$overall["Accuracy"], sensitivity = cm_sonar_glm$$byClass["Sensitivity"],
specificity = cm_sonar_glm$$byClass["Specificity"], f1_score = F_meas(predict(sonar_glm, sonar_test), sonar_test$$Class))

#QDA
set.seed(73, sample.kind = "Rounding")

sonar_ctrl <- trainControl(method="cv", summaryFunction=twoClassSummary, classProbs=T,
savePredictions = T)

sonar_qda <- train(Class ~ V13 + V25, data = sonar_train, method = "qda", trControl = sonar_ctrl)

cm_sonar_qda <- confusionMatrix(predict(sonar_qda, sonar_test), sonar_test$Class) results_qda <- tibble(model = "QDA", accuracy = cm_sonar_qda$$overall["Accuracy"], sensitivity = cm_sonar_qda$$byClass["Sensitivity"], specificity = cm_sonar_qda$$byClass["Specificity"], f1_score = F_meas(predict(sonar_qda, sonar_test), sonar_test$$Class))  The tables don't have the same result but with the exact same code structure like the one above, on my dataset it comes out as exactly the same • can you share the code used to run the training for both models? I think it's very unlikely May 17 '20 at 12:06 • Thank you for being willing to help, I edited my question with a reprex May 17 '20 at 12:43 • my first guess is your measure of performance is not very good - e.g. proportion of correct guesses is usually a bad metric. better to use Bernoulli log likelihood May 17 '20 at 13:10 • there's a bug see cm_sonar_qda <- confusionMatrix(predict(sonar_glm, sonar_test), sonar_test$Class) May 17 '20 at 13:38
• also cars_index is used but I have no idea where that came from May 17 '20 at 13:39

Never a good thing to repeat the same chunk of code twice. So if we train on accuracy, we set up like what you did:

library(mlbench)
library(caret)

data(Sonar)
set.seed(73, sample.kind = "Rounding")

sonar_index <- createDataPartition(Sonar$Class, times = 1, p= 0.2, list = F) sonar_train <- Sonar[-sonar_index,] sonar_test <- Sonar[sonar_index,] Methods = c("glm","qda")  wrap up the test in a function: res = lapply(Methods,function(i){ ctrl <- trainControl(method="cv", classProbs=T, savePredictions = T) fit <- train(Class ~ V13 + V25, data = sonar_train, method = i, trControl = ctrl) cm = confusionMatrix(predict(fit, sonar_test), sonar_test$Class)

results_glm <- data.frame(model = i,
accuracy = cm$$overall["Accuracy"], sensitivity = cm$$byClass["Sensitivity"],
specificity = cm$$byClass["Specificity"], f1_score = F_meas(predict(fit, sonar_test), sonar_test$$Class))

return(results_glm)
})


And I get the following results:

[[1]]
model  accuracy sensitivity specificity  f1_score
Accuracy   glm 0.6511628   0.7391304        0.55 0.6938776

[[2]]
model  accuracy sensitivity specificity f1_score
Accuracy   qda 0.5813953   0.6956522        0.45     0.64

• Thank you for the tip on the function! I'll definitely use it. However, even when I apply your function to my dataset I still get the exact same results for everything. Is there a mathematical explanation behind this? May 17 '20 at 14:16
• what is the accuracy you are getting? and is your dataset imbalanced, like do you have 90% of one label May 17 '20 at 14:17
• 0.904 accuracy and the dataset is not imbalanced, aprox. 70-30 May 17 '20 at 14:23
• you have to check out why those that are predicted wrongly always fail in both models. it could be that they have some extreme values or some reason that leads to that May 17 '20 at 14:49
• hey it's really hard to tell without seeing the data. And obviously you can see from the example above, it really depends on your data May 17 '20 at 14:49