Determine by AUC and RMSE if Logistic Regression or Decision Tree is a better model?

Question

I am using Linear Regression and Decision Tree to predict whether an e-mail is spam or no spam. I have built both models and got different values regarding AUC and RMSE.

Can I determine by AUC and RMSE which model is better? Is there an unconditional model-leader?

For example, I have

for Logistic Regression:

• AUC = 89.87%
• RMSE = 34.75%
• Accuracy = 84.86 %

for Decision Tree:

• AUC = 89.73%
• RMSE = 32.99%

I would be very, very grateful about some advice.

Answer 1

If these are the only metrics suggested and I have to choose one over the other, I'd go with RMSE. I'm not sure AUROC (I guess that by AUC you mean AUROC) is suitable here. Plus, your AUC scores are practically the same while the gap in RMSE is a bit more solid. However, we have very little information on your problem and models to draw conclusions.

If you're predicting whether an email is a ham or spam, I assume that:

1. Your data is imbalanced (most of your emails aren't spam).
2. You care mostly about Precision. It's more important to you that every email identified as spam would indeed be spam. Otherwise, you're passing a lot of important emails to the spam folder.

So I think AUROC won't be a good fit here. There's a rule of thumb that on imbalanced datasets AUROC tends to favor models that assign high positive probabilities, i.e., it gives better scores to models with a high rate of false positives.

In conclusion, I'd start with calculating the confusion matrix and calculate the rates for Precision, Recall etc. I'd also suggest you look at the PRAUC metric.

Answer 2

In a classification task, along with accuracy, it is better to also check the other metrics like sensitivity, specificity, precision, etc. Since, in this case we could overlook a few spams classified as hams over any important ham i classified as spam. So, the threshold of logistic regression should be chosen accordingly. If all other metrics of Logistic regression and decision tree are comparable, then the simplest model has to be chosen, in this case logistic regression, as it is a linear model.

I'm fairly new to Data Science, and this is what I feel is to be done. Let me know if I'm wrong or missing something.

Answer 3

RMSE is a monotonic transformation of the Brier score, a strictly proper scoring rule. Therefore, any model that has a lower RMSE also has a lower Brier score and is better in the sense of having a lower Brier score.

Brier score has an advantage over the accuracy in that the Brier score is threshold-independent, while accuracy depends on an additional choice of threshold to bucket the predicted probabilities into categories. Brier score also has an advantage over AUC in that it considers the calibration of the predicted probabilities in addition to the separation of the predictions for the two categories.

A spam detector is likely to be part of an entire software pipeline that includes decisions about what to do with emails. Do they go to the inbox? Do they go straight to spam? Do they go to the inbox with "[SUSPECTED SPAM]" added to the subject line? There are decisions to be made about what will make customers happy, and customers might not all have the same preferences (might want to be quite liberal about what gets through if you are emailing with a customer who spells quite poorly and sends messages with many typos). Consequently, there is more to the idea of what is "better" than just the model predictions, though having a good handle on the spam probabilities seems like a good start.

Shamelessly, I will link to a question of mine about proper scoring rules in the context of spam/ham emails.