1
$\begingroup$

I'm working with the famous diamonds dataset and the target value is non-normal:

enter image description here

After applying the Box-Cox transform, the shape of the histogram is closer to a normal distribution but the quantile plot is far from it:

enter image description here

Also, all normality tests fail to reject non-normality with tiny p-values. This happens after I have filtered the outliers in my data using the IQR criterion.

Is there a way to fix the normality of my target variable? Why is the Box-Cox transform failing?

$\endgroup$
3
  • 2
    $\begingroup$ You have included the regression tag. Linear regression makes no assumption that $y$ is normal. When we make a normality assumption in linear regression, the assumption is about the error term, not the pooled/marginal distribution of $y$. You might be transforming for no reason. $\endgroup$
    – Dave
    Jul 11, 2022 at 11:40
  • $\begingroup$ I would take a log transform because your data is positive and right skewed and then run my regression. If you want to use a model that assumes normality like OLS then you should check that assumption on the residuals not on the target variable. So as Dave said you might not need any transformation at all. $\endgroup$
    – Amin Shn
    Jul 11, 2022 at 11:50
  • 1
    $\begingroup$ Related to both the two comments above; a log transformation of prices of diamonds may be useful for reasons other than conditional normality (which you can't assess from your plot, since you're looking at the marginal distribution, whose appearance may be non-normal when the assumptions hold). Such a transformation may well help improve linearity, heteroskedasticity and make the distribution around the conditional mean less skew, though my first thought may well have been to do something else than transform the data. $\endgroup$
    – Glen_b
    Jul 12, 2022 at 3:26

1 Answer 1

2
$\begingroup$

Quick points about the issues you raised:

  1. There is no guarantee that the Box-Cox transformation will transform your data to look more like a normal distribution.
  2. Do not remove "outliers" because they do not fit your model, change your model.
  3. Often times there is nothing you can do (no transformation) that can make a variable "normal".

More importantly:

  1. Why are you trying to transform your variable into a normal distribution?
  2. Why are you removing "outliers"?
  3. Your data is bounded from below by 0 (or close to 0, so it appears), so there is no way to make this normal in any way, shape, or form.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.