0
$\begingroup$

I'm currently analyzing the same data using lasso, elasticnet and random forest classifiers. I'm just wondering if there is a best way to compare these three models(eg. AUROC, accuracy, mcnemar's test), or I should compare multiple features and trying to figure out if they all favour the same model?

$\endgroup$
1
0
$\begingroup$

In machine learning, there is no one algorithm that’s always better than others which is as per the “No free lunch theorem”. Therefore, one has to try with different set of classifiers and choose the best one.

Now it is also important to statistically compare the classifiers to know whichone is the best one. One way is to carry out one-way ANOVA followed by post hoc tests using Tukey-Kramer method for multiple comparison analysis to determine which pairs of means are significant and which are not. This generates a graph, similar to box plot where two means are significantly different if their intervals are disjoint, and are not significantly different if their intervals overlap.

For more info, refer to this research paper

More useful links:

http://www.cse.iitd.ac.in/~sumantra/publications/ijmi16_nonmotor.pdf http://machinelearningmastery.com/compare-machine-learning-algorithms-python-scikit-learn/

$\endgroup$
5
  • $\begingroup$ Is there a good reason to use ANOVA instead of a non-gaussian assumption (non-parametric) version like Chi-square / Friedman-test? I rarely find classifier output metrics to be even nearly normally distributed. $\endgroup$
    – hirschme
    Aug 6 '18 at 22:52
  • $\begingroup$ What means would you be comparing? $\endgroup$
    – rolando2
    Aug 7 '18 at 12:35
  • 1
    $\begingroup$ @hirschme They could also be used I guess. Take atleast 100 different partitions of the data and evaluate the classifier metrics. In this case, generally, the metrics are likely to follow normal distribution. If not, then it shows that the characteristics of the training data is changing drastically with partitions. $\endgroup$
    – prashanth
    Aug 8 '18 at 7:41
  • $\begingroup$ @rolando2 means of all the classifier metrics that you want to compare. It could be accuracy, AUC, sensitivity and/or specificity. $\endgroup$
    – prashanth
    Aug 8 '18 at 7:42
  • $\begingroup$ You could add these new explanations to your answer. $\endgroup$
    – rolando2
    Aug 8 '18 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.