1
$\begingroup$

I have over 30 features: several have zero-inflated and highly positive skewed distribution. Those distributions are expected because they are semi-continuous monetary related features.

For example: Revenue earned by age.

If 70% of all the respondents are unemployed and in school, most of them will have 0.

I've read about the different methods: square/cube root, Box-Cox and logistic but I'm not sure which one would apply in my case.

  1. If I choose log and add a 1 to each value , what will be the impact? Could that make sense?

  2. How would the Box-Cox transformation be beneficial in this example and would it perform better than the logistic transformation ?

  3. Cube/square root seems to be an oversimplified technique to achieve this and doesn't seem to properly address my issue. Any thoughts?

Note: My end-goal is to apply pca and then Kmeans clustering.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Several posts on site discuss the problem with transforming values with one or a few large spikes of probability - whatever transformation you use leaves the spike still a spike. It might be worth trying a few searches $\endgroup$
    – Glen_b
    Commented Nov 1, 2021 at 22:34

1 Answer 1

Reset to default
2
$\begingroup$

Whatever you do, if there is a spike at zero, there will be a spike when you transform it, regardless of whether the transform maps zero to zero or not.

A transformation might help to pull in the tail of high values, but none of the methods you mention as a goal require a transformation absolutely.

Square roots, cube roots, log1p meaning $\log (x + 1)$, and asinh can all cope with zeros.

A transformation may help in visualization even if it is a little arbitrary. The same doesn't necessarily apply to modelling.

Details:

I've never seen any transform based on (variable $+ 1$) called logistic. (One reference to "logistic" was fixed as a typo, but another remains.)

Box-Cox(*) in my experience is defined and implemented only for entirely positive variables.

(*) Initial capitals please.

Improve this answer
$\endgroup$
4
  • $\begingroup$ typo on the loga sorry. In regards to the box-cox. My values are all positive. So in this case should i use log+1 or box-cox, which one makes more sense in my context? $\endgroup$
    – Roger Steinberg
    Commented Nov 1, 2021 at 17:44
  • 1
    $\begingroup$ Zero-inflated does not mean "all positive". $\endgroup$
    – Nick Cox
    Commented Nov 1, 2021 at 18:08
  • $\begingroup$ gotcha. so in order for the boxcox to work, it "must" be all positive (i.e no zeros) , should i transform 0 values by adding a one as for the log ? which technique should I choose ? $\endgroup$
    – Roger Steinberg
    Commented Nov 1, 2021 at 18:11
  • 2
    $\begingroup$ There is no "should" here independently of your goals. Adding 1 and then applying Box-Cox is, to my taste., a bad idea when the other possibilities are one step. I can't recommend anything beyond trying your chosen goals both with and without transformations and seeing what makes sense. $\endgroup$
    – Nick Cox
    Commented Nov 1, 2021 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.