I am conducting a classification problem. Usually AUC is used to check the effectiveness of a model when performing logistic regression model. However, if I fit the same data set by using Random forest and logistic regression models, which measures should be used to check the accuracy of the two models? Are there any measures like "BIC" can be used to compare the two models? Thanks.
2 Answers
$\begingroup$
$\endgroup$
4
Those are two separate issues, AUC can be used to compare arbitrarily many algorithms. AUC is also not more linked to logistic regression than to random forests or any other algorithm. Just test both algorithms on the same data-set splits (hold-out, cross validation, bootstrap or whatever it is you do) and compare the numbers.
AUC has other drawbacks like being limited to binary classification and being less intuitive than say accuracy, but that's data-set related and thus also independent of the number and type of algorithms to compare.
There are many measures you can use, AUC is one of them.
-
$\begingroup$ Thank you for the quick answering. My question is, if the AUCs calculated from the two models are the same, but the estimated outputs (e.g., y-hat for regression model) are totally different, then which one is better? $\endgroup$– LiamCommented Nov 1, 2017 at 13:33
-
$\begingroup$ You didn't say that. If your algorithms have (as good as) the same average performance, you have the discretion to choose the algorithm with the lower variance of performance, the algorithm that is computationally simpler or conceptually simpler etc. You could also have an ensemble of algorithms and determine the label by majority vote. Your choice! But that is also true for all performance metrics, not just for AUC. $\endgroup$ Commented Nov 1, 2017 at 13:37
-
$\begingroup$ +1, however, the AUC is not ultimately limited to binary classification (Hand & Till, 2001); & although the AUC is less immediately intuitive to people who have never completed any statistical or machine learning coursework, it shouldn't be any less intuitive than accuracy to people who have (eg, see: here, &/or here). $\endgroup$ Commented Nov 1, 2017 at 16:33
-
$\begingroup$ @Liam, if your models have comparable overall performance, but yield different outcomes for individual cases, you might explore how to combine them. Eg, you might train a 3rd model that uses the outputs of the 1st 2 to make a final prediction. $\endgroup$ Commented Nov 1, 2017 at 16:38
$\begingroup$
$\endgroup$
1
You can also use other measures like the brier score and logarithmic loss which look how well the probability estimations are done. Or of course the missclassification case, which has the drawback that a specific threshold has to be set. I always check these three measures together with the AUC. In R you can easily calculate them with the mlr package: http://mlr-org.github.io/mlr-tutorial/devel/html/measures/index.html
-
$\begingroup$ Thank you very much! Your answer is exactly what I want. $\endgroup$– LiamCommented Nov 10, 2017 at 15:12