0
$\begingroup$

I have used Box-Cox Yeo-Johnson transformation to make my skewed data columns less skewed and more normal so that I can remove outliers.

e.g. originally most of my columns have a 'skewness' of 400! After applying Box Cox they reduce to -36.965404. This is a huge difference and is still somewhat skewed.

I then apply quantile based method to remove outliers (by column) and a lot of the data is removed (50%) so this method doesn't seem appropriate.

 def remove_outlier_by_Col(df,col,low_q,hi_q):
        low = low_q
        high = hi_q
        quant_df = df.quantile([low, high])
        df = df[(df[col] > quant_df.loc[low, col]) & (df[col] < quant_df.loc[high, col])]
        return df

I am doing this to minimize the effect the 'outliers' have on xgboost but I am having trouble deciding how to treat these outliers when my distribution is heavily skewed.

I have thought about simply Winsorizing, but is this appropriate when data is skewed?

Can somebody please advise what is best thing to do in this situation!

Before Yeo-Johnson transformation on one column:

enter image description here

After Yeo-Johnson on the same column:

enter image description here

$\endgroup$
  • $\begingroup$ What to do depends on knowing more about your variables (you say columns). Minimum information here could be minimum, median and quartiles and maximum for at least some of them, together with information on bounds: Could a variable ever be zero or negative? $\endgroup$ – Nick Cox May 15 at 11:05
  • 1
    $\begingroup$ There are many threads here on outliers as the tag indicates. You could read some of the most upvoted threads, particularly as it's not clear that you have a distinctive question here (without precise information on your data). I think there is a fairly wide consensus among people who call themselves statistical that throwing outliers just because they are awkward is a really bad idea. I can't speak for anyone who marches under a banner of machine learning. But either way, it's what you want to do next that is key here. $\endgroup$ – Nick Cox May 15 at 11:12
  • $\begingroup$ yes i have considered, that a variable can be negative in my case they most cerntainly can (financials). so i do not want to remove my negative variables. $\endgroup$ – Maths12 May 15 at 11:15
  • $\begingroup$ Ok, but my first comment stands. We can show the effect of any transformation given those five summary values. What transformation(s) precisely did you apply? $\endgroup$ – Nick Cox May 15 at 11:17
  • $\begingroup$ it has affected it such that after the yeo johnson transformation the plots do look more 'normal' after it. but the skewness has gone from say 400 to -20 in example above. i thought that after making it more 'normal' i could then use a method to remove outliers, but it results in too much data loss $\endgroup$ – Maths12 May 15 at 11:30
1
$\begingroup$

The criteria here should depend on the goals of the project, including what modelling or other analyses are intended next. Otherwise guidelines might include

  1. Outliers are likely to be genuine, and so in general should be included in any analysis, yet not so that results are highly distorted by a small fraction of extreme outliers.

  2. Any transformations should be easy to report (as exactly what you did) and as far as possible easy to interpret and discuss. (Any report that Box-Cox or Yeo-Johnson method was used is, for example, not informative without knowing the parameter values estimated or chosen.)

  3. Other ideal conditions aside, approximately symmetric distributions are easier to handle than highly asymmetric distributions.

  4. Other ideal conditions aside, approximately normal distributions are easier to handle than others.

On #3 and #4 I note that even ideally it is not marginal distributions of outcomes or predictors that are particularly important, but conditional distributions of outcomes given the predictors.

The minimal information to try out a transformation would be values of (selected) order statistics or quantiles, such as minimum and maximum and at least median and quartiles too, and ideally more such summaries. The elementary but fundamental principle is that quantile of transformed variable $=$ transform of quantile of original variable. (Small print that may bite occasionally is that median and other quantiles may be calculated by some kind of interpolation between original data points, but no more on that from me here.)

If it is not obvious otherwise, it is vital to know the possible support of any variable, including whether zeros or negative values are possible, as that affects which transformations are possible or even convenient.

Note that knowing mean and SD is not especially helpful in choosing a transformation.

The report here makes some comparisons possible. Although not explained in detail, the data have a flavour like profits and losses for a range of firms, such that negative and positive values are both possible and seen and a few values are very large indeed. Transformations that preserve sign -- which arguably helps a great deal with #2 above -- include inverse hyperbolic sinh (often asinh() or some such in software) and neglog (under that or some other name), namely

$$\text{sign}(x) \log(1 + |x|)$$

which behaves like $-\log(-x)$ for $x \ll 0$ and like $\log(x)$ for $x \gg 0$ and is $0$ at $x = 0$ and differentiable throughout.

Trying out these transformations I calculated Bowley-like skewness measures of the form $(U - 2M + L) / (U - L)$ for median $M$, first for $(U, L) = $ (maximum, minimum) and then for $(U, L) = $ (upper quartile, lower quartile). Spelling this out,

$$[(U - M) - (M - L)] / (U - L)$$

approaches $1$ if $M \approx L$ (extreme positive skewness) and it approaches $-1$ if $U \approx M$ (extreme negative skewness). Naturally it is 0 if (and only if) $(U - M) = (M - L)$.

The display here is negligent of how many decimal places are worth thinking about. Nor do units of measurement concern us. This tableau is (minimum, lower quartile, median, upper quartile, maximum) as reported above in the question (YJ = Yeo-Johnson) and also for asinh and neglog.

  +---------------------------------------------+
  | original          YJ       asinh     neglog |
  |---------------------------------------------|
  | -3057.04   -217.7435   -8.718349   -8.02553 |
  | 380.4398    -.185922    6.634477   5.943953 |
  | 871.7755   -.1274842    7.463679   6.771678 |
  | 2478.533   -.0058352     8.50857   7.815825 |
  | 5.15e+08    188.3655    20.75313   20.05998 |
  +---------------------------------------------+

Next we have those skewness measures:

                 (E)       (Q) 
original       1.0000    0.5316
YJ            -0.0717    0.3510
asinh         -0.0981    0.1151
neglog        -0.0537    0.1156


(E) skewness based on extremes 
(Q) skewness based on quartiles 

Morals:

A. The Yeo-Johnson transformation (unstated parameters) evidently involves translation as well as powering as sign is not respected.

B. The asinh and neglog transformations work as well if not better than the Yeo-Johnson, while preserving sign of the original values. The main deal is pulling in far tails but making the middle of the distribution more symmetrical is always welcome.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thank you so much for this nick, i really appreciate this. can you please direct me to some good mathematical books on these concepts? i would love to have this knowledge. $\endgroup$ – Maths12 May 15 at 18:01
  • $\begingroup$ The best book I know on transformations is global.oup.com/academic/product/… but the case of responses that can be positive and negative is not well covered so far as I know in textbooks. For the name neglog see jstor.org/stable/3592674 Much of the discussion of asinh uses the fuller name inverse hyperbolic sine. $\endgroup$ – Nick Cox May 15 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.