I'm working on a regression problem where the dependent variable is highly skewed (See below). My idea is to use a Box-Cox transformation to make the problem a bit more well-behaved. Is it good practice to transform the dependent variable on the whole dataset (before splitting it into train and test sets)? In addition, when evaluating the model, should I use the transformed or untransformed values to calculate R2, RMSE etc?
In classical regression, there are no normality assumptions about the distribution of predictors in the regression or even the response (Y). All assumptions are based on residuals. It is assumed that the residuals should be approximately normal. Transforming the predictors or outcome might alter the distribution of residuals. Look for the influential observations using Cook's distance to see if any of your extreme values have high influence. Transformations are good whenever u have a meaningful interpretation, for instance, the log transform and reciprocal are often useful depending on context. Now, in the context of machine learning, transforming entire data before the split will cause data leakage. As a consequence, u will have unreliable overoptimistic performance on the test data.