2
$\begingroup$

I am conducting a classification problem. Usually AUC is used to check the effectiveness of a model when performing logistic regression model. However, if I fit the same data set by using Random forest and logistic regression models, which measures should be used to check the accuracy of the two models? Are there any measures like "BIC" can be used to compare the two models? Thanks.

$\endgroup$
1
$\begingroup$

Those are two separate issues, AUC can be used to compare arbitrarily many algorithms. AUC is also not more linked to logistic regression than to random forests or any other algorithm. Just test both algorithms on the same data-set splits (hold-out, cross validation, bootstrap or whatever it is you do) and compare the numbers.

AUC has other drawbacks like being limited to binary classification and being less intuitive than say accuracy, but that's data-set related and thus also independent of the number and type of algorithms to compare.

There are many measures you can use, AUC is one of them.

$\endgroup$
  • $\begingroup$ Thank you for the quick answering. My question is, if the AUCs calculated from the two models are the same, but the estimated outputs (e.g., y-hat for regression model) are totally different, then which one is better? $\endgroup$ – Liam Nov 1 '17 at 13:33
  • $\begingroup$ You didn't say that. If your algorithms have (as good as) the same average performance, you have the discretion to choose the algorithm with the lower variance of performance, the algorithm that is computationally simpler or conceptually simpler etc. You could also have an ensemble of algorithms and determine the label by majority vote. Your choice! But that is also true for all performance metrics, not just for AUC. $\endgroup$ – David Ernst Nov 1 '17 at 13:37
  • $\begingroup$ +1, however, the AUC is not ultimately limited to binary classification (Hand & Till, 2001); & although the AUC is less immediately intuitive to people who have never completed any statistical or machine learning coursework, it shouldn't be any less intuitive than accuracy to people who have (eg, see: here, &/or here). $\endgroup$ – gung Nov 1 '17 at 16:33
  • $\begingroup$ @Liam, if your models have comparable overall performance, but yield different outcomes for individual cases, you might explore how to combine them. Eg, you might train a 3rd model that uses the outputs of the 1st 2 to make a final prediction. $\endgroup$ – gung Nov 1 '17 at 16:38
1
$\begingroup$

You can also use other measures like the brier score and logarithmic loss which look how well the probability estimations are done. Or of course the missclassification case, which has the drawback that a specific threshold has to be set. I always check these three measures together with the AUC. In R you can easily calculate them with the mlr package: http://mlr-org.github.io/mlr-tutorial/devel/html/measures/index.html

$\endgroup$
  • $\begingroup$ Thank you very much! Your answer is exactly what I want. $\endgroup$ – Liam Nov 10 '17 at 15:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.