6
$\begingroup$

When I am given a variable, I usually decide whether to take its logarithm based on gut feeling. Usually I base it on its distribution - if it has long tail (like: salaries, GDP, ...) I use logarithms.

However, when I need to preprocess a large number of variables, I use ad hoc techniques. With some tweaking I can arrive at "desired" results, but without a good argumentation.

Is there a common or widely accepted way to decide whether to scale a (single) variable with log (or, say, square root)?

Of course, for more refined techniques I need scaling related used method, the meaning of particular parameters or their relations. But e.g. for deciding whether to use log scale in a plot - distribution of a single variable should suffice.

Requirements:

  • It should be relatively method-agnostic (I can do further rescaling, if needed).
  • It should based only on the distribution of values (not e.g. semantics of data).
  • It should be a sensible rule for choosing scales in plots.

I know:

$\endgroup$

2 Answers 2

3
$\begingroup$

As a rule of thumb, try to make the data fit a (standard) normal distribution, a uniform distribution or any other distribution where the values are more or less “evenly” distributed.

As a measurement, one thing that you could aim for is to maximize the distribution’s entropy for a fixed variance.

So, if your data is approximately log-normal distributed, taking its logarithm would probably be a good idea since afterwards it would be approximately normal distributed.

Another way to determine how to preprocess the data would be to transform it to a distribution in which an additive perturbation of a certain size would be equally significant no matter what the value that was being perturbed was. For example, if a 5 % raise in salary can be said to be equally significant no matter how much money you earn, you should probably logarithmize the data since that would make an additive perturbation equally significant for all values.

$\endgroup$
7
  • $\begingroup$ As a rule of thumb, try to make the data fit a (standard) normal distribution, a uniform distribution or any other distribution where the values are more or less “evenly” distributed. Is this rule of thumb applicable for algorithms? $\endgroup$
    – spectre
    Dec 9, 2021 at 16:48
  • $\begingroup$ @spectre I don’t understand your question. What do you mean by “applicable for algorithms”? $\endgroup$ Dec 10, 2021 at 17:15
  • $\begingroup$ You mention in your answer to try to make the data fit a normal distribution. Is this applicable to all algorithms? If my data is right skewed and I am using a RandomForest algorithm, then should I transform all my variables into normal distribution? (RandomForest does not require the data to be normally distributed) $\endgroup$
    – spectre
    Dec 11, 2021 at 5:08
  • $\begingroup$ @spectre While it was a while ago, I may have went with the requirement "it should be a sensible rule for choosing scales in plots" given by OP. No algorithm I know of requires the data to be normally distributed, but some algorithms will work better if it is. I know too little about random forests to be able to say whether that is also true for that algorithm. But if the only thing that matters is which order the numbers come in when sorted (along different axis) and not their actual values, making the data normal distributed will not make a difference and therefore be unnecessary. $\endgroup$ Dec 11, 2021 at 5:37
  • $\begingroup$ @spectre On a related note, it is usually a good thing if the different predictors (input variables) are as little correlated as possible, i.e., that their correlation is very close to 0. I'm quite sure this is desirable even when using random forests, so if your predictors are heavily correlated, it might be a good idea to transform them to new variables that are not correlated. The process of transforming the predictors in this way (along with normalizing them such that the get the variance 1) is known as whitening. $\endgroup$ Jan 4, 2022 at 12:51
1
$\begingroup$

(So to be kosher and not mix the question with an answer.)

Right now I am using scale which minimized the following ratio: $$\frac{\sqrt[4]{\langle (x - \bar{x})^4 \rangle}}{\sqrt{\langle (x - \bar{x})^2 \rangle}}$$ That is, after normalizing a variable (i.e. mean 0 and variance 1) I am looking to have the 4th moment as low as possible (so to penalize too long-tailed, or otherwise disperse, distributions).

For me it works (but I am not sure if it's only my using it; and if there are any easy pitfalls).

$\endgroup$
6
  • 4
    $\begingroup$ This is clever but highly non-robust: the fourth moment is extremely sensitive to outliers. $\endgroup$
    – whuber
    Jul 22, 2015 at 15:54
  • $\begingroup$ @whuber In some sense I want to be sensitive to outliers (even a single outlier can wreck the plotting scale). But you are right that it may result e.g. in choosing logarithmic scale based on a single erroneous datapoint. $\endgroup$ Jul 22, 2015 at 16:52
  • 2
    $\begingroup$ Seems to me a lot quicker and simpler just to look at minimum and maximum, which will deal with outliers too. Rule 1. (sorry if it seems too obvious) If the minimum is not positive, plain logarithms are inapplicable and probably don't make sense any way. (Side comment: some people are happy with adding a constant first to make logarithms defined.) Rule 2. The benefit of using logarithms increases with max/min. I don't seek cutoffs for Rule 2, as .e.g. in a scatter plot, the benefits of using a transformation will depend on the other variable too. $\endgroup$
    – Nick Cox
    Apr 15, 2016 at 10:05
  • $\begingroup$ Why the 4:th moment? Intuitively, it seems smart, but how did you come up with the method? Can you show that it is optimal in some way? $\endgroup$ Sep 17, 2018 at 4:10
  • $\begingroup$ Minimizing the standardized 4:th momentum, a.k.a. the kurtosis, is in fact not optimal, since it results in a PDF with two very sharp peaks. See my answer for a (probably) better measurement to optimize. $\endgroup$ Sep 17, 2018 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.