Comparing two regressions models by looking at $R^2$

Question

I recently found out that there are some situations where we can not use $R^2$ to compare goodness of fit between two regression models.

Let

$\begin{aligned}Y &= \beta_1 X_1 + \beta_2 X_2 + u \\ \ln(Y) &= \beta_1 X_1 + \beta_2 X_2\end{aligned}$

be two regressions. I want to compare these two by looking at their $R^2$. At first glance I thought that since $R^2$ explains us how close the explained variation is to the total variation, it could be a good way of comparing models. But for some reason I can not use $R^2$ in this case because their "ESS" are different. I didn't get why that's a reason.

Answer 1

Are $u$ the residuals; is the second model also supposed to have residuals?

Choosing whether to log-transform the response variable should be based on the type of relationship you expect between the explanatory and response variable. The explained variance ($R^2$) can be low or high despite the relationship being logarithmic.

About R-squared
$R^2$ is not a model selection criterium because it does not take into account the degrees of freedom used by the model (i.e. the number of parameters). However, since these models have the same number of parameters, this would not be a problem. More important is that they describe something completely different:

At first glance I thought that since $R^2$ explains us how close the explained variation is to the total variation

This is true, $R^2$ is the amount of variance in the response variable that is explained by the model. However, the $R^2$ of the first model describes the variance explained of $Y$, which is something else than in the second model ($\ln(Y)$). This is probably why you found that they cannot be compared this way: ESS from your question is Explained Sum of Squares.

Answer 2

If the only two choices are either a log transformation or no transformation of the dependent variable, then to get the AIC values on an equal footing and allow for a comparison one could perform the linear regression on the untransformed dependent variable and on the log-transformed dependent variable multiplied by the geometric mean.

First (using R for this example) generate some data with a model that's linear in the logs of the dependent variable:

set.seed(12345)
x = c(1:10)
y = exp(1 + 0.2*x + 0.2*rnorm(10))

Fit a linear regression on $y$ given $x$:

lm.y = lm(y ~ x)
AIC(lm.y)
# 41.72051

Now fit a linear regression on the geometric mean times the log of $y$ given $x$:

lm.logy = lm(I(mean(log(y))*log(y)) ~ x)
AIC(lm.logy)
# 9.189641

The difference in the AIC values is around 32 which would suggest that taking the logs would produce a better model. (Not necessarily a good model, but a better model.)

If the choice of the transformation is on some unknown power (rather than just "knowing" that either a log transformation or no transformation are the only two options), then that power should be estimated simultaneously with the regression parameters. See Box-Cox transformations.