0
$\begingroup$

I am working on a simple binary classification problem, where I compare various models. I have posted the mean accuracies for 5-fold cross validation for the models below (the area-under-curve values behave similarly). The accuracies were calculated on a test set.

My question is: Can I make any assumptions why the accuracies differ so much between models (52% to 88%)? Is there any common denominator between the good/bad performing models?

Model Accuracy
Naive Bayes 0.52
Logistic Regression 0.53
Linear SVM 0.53
QDA 0.54
AdaBoost 0.63
Random Forest 0.64
RBF SVM 0.85
Nearest Neighbors 0.88
Deep Neural Network 0.88
$\endgroup$
1
  • 1
    $\begingroup$ Looks like complex, no-linear models work better. Perhaps the data do not fit linear models. $\endgroup$
    – Kota Mori
    Sep 10, 2021 at 11:04

1 Answer 1

2
$\begingroup$

This is very general question and there are many factors that can effect model accuracy and the differences between the models. In order to answer about your specific problem, many details are missing (problem types, hyper parameters of the models, dataset size, train\test split etc)

I'll explain about some key factors that can affect the results in the general case.

  • generalization: Maybe some models generalize better. You can check it by measuring the performance on the train set and see if it changes the result. If it does, you can improve generalization. Each model can have different ways to improve generalization (e.g. early stopping, regularization, dropout etc)

  • Is the problem linearly separable? - If not, linear models like logistic regression, linear SVM, will not perform well. In this case you either use non linear models or to create non-linear features and feed them to linear models. Note that this, by itself, can't explain the poor performance of non linear models like random forest.

  • Model capacity - if the learnt function is complex, maybe you need large model capacity. Again different for each method\model (e.g. number of layers in deep NN, number of trees and trees depth in random forest etc)

  • Hyper parameters - In many cases you can improve accuracy dramatically by tuning parameters for each model. e.g. number of trees in random forest, regularization term coefficient, learning rate, number of estimators in adaboost etc.

If i have to guess, with further tuning you can improve the accuracy of your non linear models like ada boost and random forest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.