Comparing two regressions models by looking at $R^2$

I recently found out that there are some situations where we can not use $R^2$ to compare goodness of fit between two regression models.

Let

$Y = \beta_1 X_1 + \beta_2 X_2 + u \\ ln(Y) = \beta_1 X_1 + \beta_2 X_2$

be two regressions. I want to compare these two by looking at their $R^2$. At first glance I thought that since $R^2$ explains us how close the explained variation is to the total variation, it could be a good way of measuring. But for some reason I can not use $R^2$ in this case because their "ESS" are different. I didn't get why that's a reason.

Are $u$ the residuals; is the second model also supposed to have residuals?

Choosing whether to log-transform the response variable should be based on the type of relationship you expect between the explanatory and response variable. The explained variance ($R^2$) can be low or high despite the relationship being logarithmic.

$R^2$ is not a model selection criterium because it does not take into account the degrees of freedom used by the model (i.e. the number of parameters). However, since these models have the same number of parameters, this would not be a problem. More important is that they describe something completely different:

At first glance I thought that since $R^2$ explains us how close the explained variation is to the total variation

This is true, $R^2$ is the amount of variance in the response variable that is explained by the model. However, the $R^2$ of the first model describes the variance explained of $Y$, which is something else than in the second model ($\ln(Y)$). This is probably why you found that they cannot be compared this way: ESS from your question is Explained Sum of Squares.

• +1 for most of your answer. However, the last paragraph could be misunderstood. Neither AIC nor adjusted $R^2$ can be used to compare models with transformed dependent variables (here: "plain vanilla" $Y$ vs. $\ln Y$). I'd recommend cross-validation for this, of course with $\ln Y$ back-transformed. – Stephan Kolassa Oct 11 '17 at 7:31
• @StephanKolassa While the answer is correct, it fails to answer the question. There's no point to any of those metrics. Whether the dependent variable should be log-transformed depends on the problem, not those numbers. – HelloWorld Oct 11 '17 at 13:02
• I have removed the part about model selection criteria to avoid confusion. – Frans Rodenburg Oct 11 '17 at 13:21
• what about metrics that are invariant to monotone transformations? e.g. AUC or it's equivalent for regression – rep_ho Oct 12 '17 at 5:42

If the only two choices are either a log transformation or no transformation of the dependent variable, then to get the AIC values on an equal footing and allow for a comparison one could perform the linear regression on the untransformed dependent variable and on the log-transformed dependent variable multiplied by the geometric mean.

First (using R for this example) generate some data with a model that's linear in the logs of the dependent variable:

set.seed(12345)
x = c(1:10)
y = exp(1 + 0.2*x + 0.2*rnorm(10))


Fit a linear regression on $y$ given $x$:

lm.y = lm(y ~ x)
AIC(lm.y)
# 41.72051


Now fit a linear regression on the geometric mean times the log of $y$ given $x$:

lm.logy = lm(I(mean(log(y))*log(y)) ~ x)
AIC(lm.logy)
# 9.189641


The difference in the AIC values is around 32 which would suggest that taking the logs would produce a better model. (Not necessarily a good model, but a better model.)

If the choice of the transformation is on some unknown power (rather than just "knowing" that either a log transformation or no transformation are the only two options), then that power should be estimated simultaneously with the regression parameters. See Box-Cox transformations.