How to compare performance of regression and classification?

Question

I built a linear regression model and evaluated it with respect to R-Squared and RMSE (the latter cross-validated).

Then, I built a logistic regression classifier on the same data. It answers the same question, but discretized of course. I wanted to test whether this simplification of the problem increased prediction quality. The classifier is evaluated with respect to accuracy (cross-validated).

How do I compare the performance of the two models?

Can I simply compare the ratios $1-RMSE/RMSE_{baseline}$ and $Acc/Acc_{baseline}-1$. That feels kind of wrong, though.

Answer 1

Imagine this: You are trying to predict age of a population using some features. It does not work that well. Then, you are reducing the complexity of the problem. You only try to predict whether age is above or below 20. This works well, using the same features. I simply want to quantify the improvement gained by this simplification.

So you simply would have two models, one that says that the age is a numeric value $\hat y$ (regression), and the other that says that the age is some constant depending weather it is below, or above certain threshold (classification). To choose the optimal constants, you would simply take the conditional mean for the age, given being above, or below, the threshold. Now you can simply compare both outcomes using same metric for comparing regression models (e.g. RMSE, MAE).

I guess, in vast majority of cases this would tell you that no matter how bad the regression model is, it is still better then predicting only two constants. But if you think about it, on the end of the day, this is what the classification model will give you.

Now, if you'd agree with me that using classifier leaves you with two conditional means as constants approximating the continuous variable, another thing follows. An algorithm that conditional on some variables makes a binary split (you also said in the comments that you actually don't have any prespecified threshold) and predicts two conditional means is a very simple regression tree (see here for explanation how decision trees work). Usually, you would use much more complicated regression trees, that make more splits, and so get more accurate. Even more, usually you wouldn't use a single tree, but rather a random forest of many trees, trained on different subsets of data, that made multiple different splits and then aggregate the outputs. So, have you tried random forest? It is simple, yet pretty powerful algorithm, that would be doing all the "classification" part for you, but better.

But the general answer is that in most cases you can't compare classification to regression. Both approaches give you different kind of outcomes, I can't think of situation where they would be equivalent. In terms of real-life examples, say that you used your algorithm to predict age of your customers, and based on this send them targeted marketing campaigns. With more precise predictions about age you would be able to send them the age-specific campaigns. In this case the more accurate you are, the better for you. On another hand, you could quantify this and check how much better your business does if you have campaigns for precise age vs two age groups (in terms of some business metric like clicks, purchases etc). Based on this, you would also know how much better would you be if using regression vs classification. Same would apply if the classification task was something completely different, say classifying "send the campaign vs not (irrelevant of age)", or sending age specific campaign based on regression predictions for age, here also you should rather undertake an A/B test and simply check what pays better (or gives you more clicks etc). Saying this differently, to answer your question you need to consider also what for you want to use the outputs of the algorithms and check which algorithm works better for the task.

Answer 2

I would choose the metric depending on my problem. Ie, if my problem is to predict the age of a person, I'd choose RMSE, if my problem is to predict young or old, I'd choose accuracy.

After choosing the metric, you need to be able to use that metric with the models. Ie, if your problem is to predict young or old, then it's supposed that you have a threshold to determine the labels used to train the LR, so you can apply what @EdM had mentioned.

IMHO, if you compare two models that perform different tasks, you can't conclude that one is better than the other, because are doing different things.

Let me know if I misunderstood something.