1
$\begingroup$

I'm working on a regression problem where the dependent variable is highly skewed (See below). My idea is to use a Box-Cox transformation to make the problem a bit more well-behaved. Is it good practice to transform the dependent variable on the whole dataset (before splitting it into train and test sets)? In addition, when evaluating the model, should I use the transformed or untransformed values to calculate R2, RMSE etc?

enter image description here

$\endgroup$
1
  • $\begingroup$ If you determine the form of the transform (or even whether to transform) by examining the data before you split them, then you are cheating: the test set no longer serves as an adequate test of your model. And--as you can find by reading many related posts on regression and transformations--the distribution of the dependent variable is usually not a good indicator of whether or how to transform, because what matters is the pattern of distributions conditional on the explanatory variables. $\endgroup$
    – whuber
    Mar 22, 2022 at 16:37

1 Answer 1

1
$\begingroup$

In classical regression, there are no normality assumptions about the distribution of predictors in the regression or even the response (Y). All assumptions are based on residuals. It is assumed that the residuals should be approximately normal. Transforming the predictors or outcome might alter the distribution of residuals. Look for the influential observations using Cook's distance to see if any of your extreme values have high influence. Transformations are good whenever u have a meaningful interpretation, for instance, the log transform and reciprocal are often useful depending on context. Now, in the context of machine learning, transforming entire data before the split will cause data leakage. As a consequence, u will have unreliable overoptimistic performance on the test data.

$\endgroup$
2
  • $\begingroup$ One more thing, $R^2$ is misleading measure since it increases whenever u have high number of predictors. Use adjusted $R^2$ to evaluate your model. $\endgroup$
    – ForestGump
    Mar 25, 2022 at 16:03
  • $\begingroup$ Technically the normality assumption is on the error distribution, which the residual distribution estimates. For ordinary least squares to give the best linear unbiased estimates, all you need is equal uncorrelated error distributions with mean 0. The normality assumption helps with parametric statistical tests. You can avoid assumptions about residuals by using proportional odds logistic ordinal regression, which can work with continuous outcomes. $\endgroup$
    – EdM
    Mar 25, 2022 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.