4
$\begingroup$

I am working on building a regression model to predict housing sales price using house features (Ames housing dataset). And I prepared feature set in two ways:

Case 1.

I performed Box-Cox transformation on all the numerical features and performed one-hot encoding on the categorical features.

Case 2.

I used the numerical feature set as it is and performed one-hot encoding on the categorical features.

I applied linear regression on both feature sets and got $R^2$ much, much lower in Case 1 (where numerical features were transformed). In Case 2 it was always more than 0.85 for any train test combination and in Case 1 it was never more than 0.5.

Since linear regression assumes the features to be conditionally independent and normal distributed and Box-Cox helps to make features near normally distributed I thought performing Box-Cox will boost my $R^2$. Can anyone help me understand why that plummeted for the model applied to Box-transformed data?

$\endgroup$
7
  • 2
    $\begingroup$ Your understanding of linear regression, unfortunately, is quite wrong. There is no assumption that features should be independent or normal. Maybe the untransformed variant contains strong outliers that can trigger leverage effects, which can affect the R-squared. Working with a clean train/valid/test strategy could reveal what is going on. $\endgroup$
    – Michael M
    Commented Jun 5, 2022 at 21:03
  • 1
    $\begingroup$ If you transformed the response variable, then the $R^2$ values just aren't comparable. Not even remotely. That's because they compare reductions in the variances of two different sets of data. $\endgroup$
    – whuber
    Commented Jun 6, 2022 at 0:33
  • $\begingroup$ @MichaelM Thank you for the insight. I did further research and discovered that the normality of the features is not an assumption. In fact, I should have figured that while performing one hot encoding (that it will obviously not result in a perfect or near normally distributed dummy variables all the time). However does not independence of the features is one of the assumptions in a ideal case linear regression? That is what I make of the lack of multi-collinearity assumption from this article: en.wikipedia.org/wiki/Linear_regression#Assumptions $\endgroup$ Commented Jun 7, 2022 at 5:47
  • $\begingroup$ @whuber Appreciate your input! By response variable do you mean the dependent variable or output variable (or the Y in Y = B*X). In my case the dependent variable was in consistent format in both the cases. $\endgroup$ Commented Jun 7, 2022 at 5:52
  • 1
    $\begingroup$ @whuber, yes in both cases the response variable used in model was log of actual response variable. But I get your point. $\endgroup$ Commented Jun 8, 2022 at 4:27

1 Answer 1

1
$\begingroup$

As was mentioned in the comments, there is no normality assumption about the features. It is common to assume a normal error term in linear regression, as this means that the OLS solution for the coefficient estimates is equivalent to maximum likelihood estimation of the regression coefficients and that the usual standard errors are correct, but even that does not require normality of the features.

It is a common misconception that the features should be normally distributed. I think this comes from the normal-errors assumption being abstract, so people make the mistake of assuming normal features.

Consequently, when you transform the features, there is no reason to expect that being normal gets you closer to some ideal that should result in higher performance. You aren’t assured of worse performance, either, but if you transform features that are roughly linearly related to $y$, you might wind up with a relationship between the transformed feature and $y$ that is not linear, and your linear model will miss such a relationship.

EDIT

A simulation shows this in action.

library(MASS)
set.seed(2024)
N <- 1000
x1 <- rchisq(N, 1) # Non-normal feature
x2 <- rchisq(N, 1) # Non-normal feature
e <- rnorm(N)
Ey <- x1 - x2
y <- Ey + e 
t1 <- (x1)^(1/3) # Cube root is the Box-Cox transformation for chi-squared
t2 <- (x2)^(1/3) # https://stats.stackexchange.com/a/565868/247274
L1 <- lm(y ~ x1 + x2) # Fit to original features
L2 <- lm(y ~ t1 + t2) # Fit to transformed features
summary(L1)$r.squared # 0.7854019
summary(L2)$r.squared # 0.5932263
par(mfrow = c(2, 2))
plot(x1, y)
plot(x2, y)
plot(t1, y)
plot(t2, y)
par(mfrow = c(1, 1))

In the plots below, notice that transforming the features leads to a curved relationship between the features and the outcome that will not be captured by a linear regression unless such curvature is explicitly included in the model, such as with splines (or cubing the t features back to the original features that have a linear relationship with y).

Plot the tansformed and untransformed features

$\endgroup$
1
  • $\begingroup$ thank you for making it clear. $\endgroup$ Commented Jun 13, 2022 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.