How to compare performance of regression and classification?

Question

I built a linear regression model and evaluated it with respect to R-Squared and RMSE (the latter cross-validated).

Then, I built a logistic regression classifier on the same data. It answers the same question, but discretized of course. I wanted to test whether this simplification of the problem increased prediction quality. The classifier is evaluated with respect to accuracy (cross-validated).

How do I compare the performance of the two models?

Can I simply compare the ratios $1-RMSE/RMSE_{baseline}$ and $Acc/Acc_{baseline}-1$. That feels kind of wrong, though.

Answer 1

Imagine this: You are trying to predict age of a population using some features. It does not work that well. Then, you are reducing the complexity of the problem. You only try to predict whether age is above or below 20. This works well, using the same features. I simply want to quantify the improvement gained by this simplification.

So you simply would have two models, one that says that the age is a numeric value $\hat y$ (regression), and the other that says that the age is some constant depending weather it is below, or above certain threshold (classification). To choose the optimal constants, you would simply take the conditional mean for the age, given being above, or below, the threshold. Now you can simply compare both outcomes using same metric for comparing regression models (e.g. RMSE, MAE).

I guess, in vast majority of cases this would tell you that no matter how bad the regression model is, it is still better then predicting only two constants. But if you think about it, on the end of the day, this is what the classification model will give you.

Now, if you'd agree with me that using classifier leaves you with two conditional means as constants approximating the continuous variable, another thing follows. An algorithm that conditional on some variables makes a binary split (you also said in the comments that you actually don't have any prespecified threshold) and predicts two conditional means is a very simple regression tree (see here for explanation how decision trees work). Usually, you would use much more complicated regression trees, that make more splits, and so get more accurate. Even more, usually you wouldn't use a single tree, but rather a random forest of many trees, trained on different subsets of data, that made multiple different splits and then aggregate the outputs. So, have you tried random forest? It is simple, yet pretty powerful algorithm, that would be doing all the "classification" part for you, but better.

But the general answer is that in most cases you can't compare classification to regression. Both approaches give you different kind of outcomes, I can't think of situation where they would be equivalent. In terms of real-life examples, say that you used your algorithm to predict age of your customers, and based on this send them targeted marketing campaigns. With more precise predictions about age you would be able to send them the age-specific campaigns. In this case the more accurate you are, the better for you. On another hand, you could quantify this and check how much better your business does if you have campaigns for precise age vs two age groups (in terms of some business metric like clicks, purchases etc). Based on this, you would also know how much better would you be if using regression vs classification. Same would apply if the classification task was something completely different, say classifying "send the campaign vs not (irrelevant of age)", or sending age specific campaign based on regression predictions for age, here also you should rather undertake an A/B test and simply check what pays better (or gives you more clicks etc). Saying this differently, to answer your question you need to consider also what for you want to use the outputs of the algorithms and check which algorithm works better for the task.

Answer 2

I would choose the metric depending on my problem. Ie, if my problem is to predict the age of a person, I'd choose RMSE, if my problem is to predict young or old, I'd choose accuracy.

After choosing the metric, you need to be able to use that metric with the models. Ie, if your problem is to predict young or old, then it's supposed that you have a threshold to determine the labels used to train the LR, so you can apply what @EdM had mentioned.

IMHO, if you compare two models that perform different tasks, you can't conclude that one is better than the other, because are doing different things.

Let me know if I misunderstood something.

Answer 3

Short answer: If you're sure you really want to strip a numeric outcome to a binary simplification, then I suggest that you establish a cutoff to determine what is a "positive" versus "negative" case on the regression data and then use AUCROC (area under the ROC curve) to compare the regression and classification models.

Longer answer

First of all, it is highly questionable if you should strip the rich information of a numeric outcome variable into a simplistic binary outcome. You did not give any application context, so it is difficult to know why you would want to do this. In most real-life situations, such simplification would lead to poor decisions. Retaining the data in its more detailed (numeric) state gives the opportunity to make more nuanced and informed decisions.

That said, if you are sure that you want to go ahead, then I do not recommend that you ever use accuracy to evaluate classification models. While it is certainly the most intuitive classification measure, it has many shortcomings and just about every other classification measure that exists improves on some of accuracy's shortcomings. It is helpful as a first step to learn about how to evaluate classification models, but it should never be used beyond that in practice.

The primary reason why accuracy is an inadequate measure is that it forces you to pick a threshold cutoff that determines what is a "positive" and what is a "negative" case. This is a serious problem because we cannot know the optimal threshold in advance since it varies with every dataset, every model applied to each dataset, and every real-life application of a model based on a dataset. Force-fitting a default cutoff (usually 0.5) is almost always an inappropriate choice. (Exactly the same problem exists with precision, recall, specificity, F1, etc. These measures are often useful for making decisions after we have selected a model but they do not provide good guidance on how to select a model in the first place.)

The AUCROC (area under the ROC curve), or just AUC for short, resolves this issue by essentially evaluating the average ability to discriminate between positive and negative cases across all possible thresholds. So, it gives a very robust measure of the all-round quality of a classification model. (It is not perfect without some occasional refinements, but that is beyond the scope of this answer.)

Although AUC is typically calculated based on probabilities (ranging from 0 to 1), that is not its essence. At its core, AUC is a ranking algorithm. It evaluates how frequently a model scores or ranks positive cases higher than negative cases. These scores or ranks do not need to be probabilities. They can also be regular numeric values, such as the predictions of a regression algorithm.

Here is the procedure that I propose:

1. You must determine what you consider to be "TRUE" or "positive" based on your numeric data. You did not give any application context, but if your analysis is to have any practical usefulness, you must carefully select a threshold value across which a critical decision is to be made. Outcome values above one cutoff will lead to one decision and values below the cutoff will lead to the second decision.

   Despite my caveat above of making a threshold decision, this is an unavoidable first step if you want to reduce numeric data to a binary simplification. However, this is not being used to evaluate the quality of models--it is needed if you want to determine what is "true" or "false" for your specific application.

2. Code your outcome data as TRUE or FALSE based on your practical decision threshold.

3. Train your classification models on the TRUE/FALSE version of the outcome data and train the regression models on the original numeric version of the outcome data.

4. Calculate AUCROC for all models:

   • The actual or truth values will be the TRUE/FALSE version of the outcome data.
   • For the classification models, the predictions will be the predicted probabilities of these classification models.
   • For the regression models, if your AUC software allows any numeric data, then enter the numeric predictions as they are as the prediction input for the AUC calculation. If not, then simply scale the values from 0 to 1 so that they look like probabilities. The scaling algorithm does not matter as long as it maintains the rank order of predicted values--that's all the AUCROC algorithm cares about. (For example, subtracting the minimum and dividing by the maximum should work just fine.)

The AUC values across models will then be directly comparable with a very specific interpretation: they show how well each model (whether classification or regression) discriminates cases that are true or false according to the specific threshold you set. If you change your mind and decide to adjust the threshold, then you would need to retrain all your models and compare them again. When the data changes, then the models are no longer valid. Otherwise similar models might perform differently with different criteria of true or false.