Yeo-Johnson does not increase normality

Question

I have used Box-Cox Yeo-Johnson transformation to make my skewed data columns less skewed and more normal so that I can remove outliers.

e.g. originally most of my columns have a 'skewness' of 400! After applying Box Cox they reduce to -36.965404. This is a huge difference and is still somewhat skewed.

I then apply quantile based method to remove outliers (by column) and a lot of the data is removed (50%) so this method doesn't seem appropriate.

 def remove_outlier_by_Col(df,col,low_q,hi_q):
        low = low_q
        high = hi_q
        quant_df = df.quantile([low, high])
        df = df[(df[col] > quant_df.loc[low, col]) & (df[col] < quant_df.loc[high, col])]
        return df

I am doing this to minimize the effect the 'outliers' have on xgboost but I am having trouble deciding how to treat these outliers when my distribution is heavily skewed.

I have thought about simply Winsorizing, but is this appropriate when data is skewed?

Can somebody please advise what is best thing to do in this situation!

Before Yeo-Johnson transformation on one column:

After Yeo-Johnson on the same column:

Answer 1

The criteria here should depend on the goals of the project, including what modelling or other analyses are intended next. Otherwise guidelines might include

1. Outliers are likely to be genuine, and so in general should be included in any analysis, yet not so that results are highly distorted by a small fraction of extreme outliers.

2. Any transformations should be easy to report (as exactly what you did) and as far as possible easy to interpret and discuss. (Any report that Box-Cox or Yeo-Johnson method was used is, for example, not informative without knowing the parameter values estimated or chosen.)

3. Other ideal conditions aside, approximately symmetric distributions are easier to handle than highly asymmetric distributions.

4. Other ideal conditions aside, approximately normal distributions are easier to handle than others.

On #3 and #4 I note that even ideally it is not marginal distributions of outcomes or predictors that are particularly important, but conditional distributions of outcomes given the predictors.

The minimal information to try out a transformation would be values of (selected) order statistics or quantiles, such as minimum and maximum and at least median and quartiles too, and ideally more such summaries. The elementary but fundamental principle is that quantile of transformed variable $=$ transform of quantile of original variable. (Small print that may bite occasionally is that median and other quantiles may be calculated by some kind of interpolation between original data points, but no more on that from me here.)

If it is not obvious otherwise, it is vital to know the possible support of any variable, including whether zeros or negative values are possible, as that affects which transformations are possible or even convenient.

Note that knowing mean and SD is not especially helpful in choosing a transformation.

The report here makes some comparisons possible. Although not explained in detail, the data have a flavour like profits and losses for a range of firms, such that negative and positive values are both possible and seen and a few values are very large indeed. Transformations that preserve sign -- which arguably helps a great deal with #2 above -- include inverse hyperbolic sinh (often asinh() or some such in software) and neglog (under that or some other name), namely

$$\text{sign}(x) \log(1 + |x|)$$

which behaves like $-\log(-x)$ for $x \ll 0$ and like $\log(x)$ for $x \gg 0$ and is $0$ at $x = 0$ and differentiable throughout.

Trying out these transformations I calculated Bowley-like skewness measures of the form $(U - 2M + L) / (U - L)$ for median $M$, first for $(U, L) = $ (maximum, minimum) and then for $(U, L) = $ (upper quartile, lower quartile). Spelling this out,

$$[(U - M) - (M - L)] / (U - L)$$

approaches $1$ if $M \approx L$ (extreme positive skewness) and it approaches $-1$ if $U \approx M$ (extreme negative skewness). Naturally it is 0 if (and only if) $(U - M) = (M - L)$.

The display here is negligent of how many decimal places are worth thinking about. Nor do units of measurement concern us. This tableau is (minimum, lower quartile, median, upper quartile, maximum) as reported above in the question (YJ = Yeo-Johnson) and also for asinh and neglog.

  +---------------------------------------------+
  | original          YJ       asinh     neglog |
  |---------------------------------------------|
  | -3057.04   -217.7435   -8.718349   -8.02553 |
  | 380.4398    -.185922    6.634477   5.943953 |
  | 871.7755   -.1274842    7.463679   6.771678 |
  | 2478.533   -.0058352     8.50857   7.815825 |
  | 5.15e+08    188.3655    20.75313   20.05998 |
  +---------------------------------------------+

Next we have those skewness measures:

                 (E)       (Q) 
original       1.0000    0.5316
YJ            -0.0717    0.3510
asinh         -0.0981    0.1151
neglog        -0.0537    0.1156


(E) skewness based on extremes 
(Q) skewness based on quartiles 

Morals:

A. The Yeo-Johnson transformation (unstated parameters) evidently involves translation as well as powering as sign is not respected.

B. The asinh and neglog transformations work as well if not better than the Yeo-Johnson, while preserving sign of the original values. The main deal is pulling in far tails but making the middle of the distribution more symmetrical is always welcome.