# How to compare performance of regression and classification?

I built a linear regression model and evaluated it with respect to R-Squared and RMSE (the latter cross-validated).

Then, I built a logistic regression classifier on the same data. It answers the same question, but discretized of course. I wanted to test whether this simplification of the problem increased prediction quality. The classifier is evaluated with respect to accuracy (cross-validated).

How do I compare the performance of the two models?

Can I simply compare the ratios $1-RMSE/RMSE_{baseline}$ and $Acc/Acc_{baseline}-1$. That feels kind of wrong, though.

• It sounds like you're trying to compare 2 different types of things, a continuous versus a yes/no predictor. What do you hope to gain by this comparison?
– EdM
Jun 6, 2015 at 15:45
• Yes, that's what I'm trying to do. Imagine this: You are trying to predict age of a population using some features. It does not work that well. Then, you are reducing the complexity of the problem. You only try to predict whether age is above or below 20. This works well, using the same features. I simply want to quantify the improvement gained by this simplification. Jun 8, 2015 at 13:30
• I what you actually care about is age greater than versus less than 20, have you considered comparing the accuracy of the logistic classifier against the ability of the linear regression model to make that classification?
– EdM
Jun 8, 2015 at 13:40
• Good idea, but not applicable in my case. I don't have a specifc age threshold, it's more like "old" vs. "young". What I'm looking for is whether there is a more general approach. For example, image the regression model has RMSE=0.7 with a baseline of 0.8 and the classifier achieves an accuracy of 90% versus a baseline of 10%. Clearly, intuition suggests that the classifier is superior. I'm looking for a more formal/mathematical way to state this. Jun 9, 2015 at 7:24

Imagine this: You are trying to predict age of a population using some features. It does not work that well. Then, you are reducing the complexity of the problem. You only try to predict whether age is above or below 20. This works well, using the same features. I simply want to quantify the improvement gained by this simplification.

So you simply would have two models, one that says that the age is a numeric value $$\hat y$$ (regression), and the other that says that the age is some constant depending weather it is below, or above certain threshold (classification). To choose the optimal constants, you would simply take the conditional mean for the age, given being above, or below, the threshold. Now you can simply compare both outcomes using same metric for comparing regression models (e.g. RMSE, MAE).

I guess, in vast majority of cases this would tell you that no matter how bad the regression model is, it is still better then predicting only two constants. But if you think about it, on the end of the day, this is what the classification model will give you.

Now, if you'd agree with me that using classifier leaves you with two conditional means as constants approximating the continuous variable, another thing follows. An algorithm that conditional on some variables makes a binary split (you also said in the comments that you actually don't have any prespecified threshold) and predicts two conditional means is a very simple regression tree (see here for explanation how decision trees work). Usually, you would use much more complicated regression trees, that make more splits, and so get more accurate. Even more, usually you wouldn't use a single tree, but rather a random forest of many trees, trained on different subsets of data, that made multiple different splits and then aggregate the outputs. So, have you tried random forest? It is simple, yet pretty powerful algorithm, that would be doing all the "classification" part for you, but better.