1
$\begingroup$

I am working on building a regression model to predict housing sales price using house features (Ames housing dataset). And I prepared feature set in two ways

Case 1.

I performed boxcox transformation on all the numerical features and performed one hot encoding on the categorical features.

Case 2.

I used the numerical feature set as it is and performed one hot encoding on the categorical features.

I applied linear regression on both feature sets and got r2 score much much lower in Case 1 (where numerical features were transformed). In Case 2 it was always more than 0.85 for any train test combination and in Case 1 it was never more than 0.5.

Since linear regression assumes the features to be conditionally independent and normal distributed and box cox helps to make features near normally distributed I thought performing box cox will boost my r2 score. Can anyone help me understand why did the r2 score plummeted for the model applied on box-cox transformed data?

$\endgroup$
7
  • 1
    $\begingroup$ Your understanding of linear regression, unfortunately, is quite wrong. There is no assumption that features should be independent or normal. Maybe the untransformed variant contains strong outliers that can trigger leverage effects, which can affect the R-squared. Working with a clean train/valid/test strategy could reveal what is going on. $\endgroup$
    – Michael M
    Jun 5 at 21:03
  • $\begingroup$ If you transformed the response variable, then the $R^2$ values just aren't comparable. Not even remotely. That's because they compare reductions in the variances of two different sets of data. $\endgroup$
    – whuber
    Jun 6 at 0:33
  • $\begingroup$ @MichaelM Thank you for the insight. I did further research and discovered that the normality of the features is not an assumption. In fact, I should have figured that while performing one hot encoding (that it will obviously not result in a perfect or near normally distributed dummy variables all the time). However does not independence of the features is one of the assumptions in a ideal case linear regression? That is what I make of the lack of multi-collinearity assumption from this article: en.wikipedia.org/wiki/Linear_regression#Assumptions $\endgroup$ Jun 7 at 5:47
  • $\begingroup$ @whuber Appreciate your input! By response variable do you mean the dependent variable or output variable (or the Y in Y = B*X). In my case the dependent variable was in consistent format in both the cases. $\endgroup$ Jun 7 at 5:52
  • 1
    $\begingroup$ @whuber, yes in both cases the response variable used in model was log of actual response variable. But I get your point. $\endgroup$ Jun 8 at 4:27

1 Answer 1

0
$\begingroup$

As was mentioned in the comments, there is no normality assumption about the features. It is common to assume a normal error term in linear regression, as this means that the OLS solution for the coefficient estimates is equivalent to maximum likelihood estimation of the regression coefficients and that the usual standard errors are correct, but even that does not require normality of the features.

It is a common misconception that the features should be normally distributed. I think this comes from the normal-errors assumption being abstract, so people make the mistake of assuming normal features.

Consequently, when you transform the features, there is no reason to expect that being normal gets you closer to some ideal that should result in higher performance. You aren’t assured of worse performance, either, but if you transform features that are roughly linearly related to $y$, you might wind up with a relationship between the transformed feature and $y$ that is not linear, and your linear model will miss such a relationship.

$\endgroup$
1
  • $\begingroup$ thank you for making it clear. $\endgroup$ Jun 13 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.