6
$\begingroup$

When I am given a variable, I usually decide whether to take its logarithm based on gut feeling. Usually I base it on its distribution - if it has long tail (like: salaries, GDP, ...) I use logarithms.

However, when I need to preprocess a large number of variables, I use ad hoc techniques. With some tweaking I can arrive at "desired" results, but without a good argumentation.

Is there a common or widely accepted way to decide whether to scale a (single) variable with log (or, say, square root)?

Of course, for more refined techniques I need scaling related used method, the meaning of particular parameters or their relations. But e.g. for deciding whether to use log scale in a plot - distribution of a single variable should suffice.

Requirements:

  • It should be relatively method-agnostic (I can do further rescaling, if needed).
  • It should based only on the distribution of values (not e.g. semantics of data).
  • It should be a sensible rule for choosing scales in plots.

I know:

$\endgroup$
2
$\begingroup$

As a rule of thumb, try to make the data fit a (standard) normal distribution, a uniform distribution or any other distribution where the values are more or less “evenly” distributed.

As a measurement, one thing that you could aim for is to maximize the distribution’s entropy for a fixed variance.

So, if your data is approximately log-normal distributed, taking its logarithm would probably be a good idea since afterwards it would be approximately normal distributed.

Another way to determine how to preprocess the data would be to transform it to a distribution in which an additive perturbation of a certain size would be equally significant no matter what the value that was being perturbed was. For example, if a 5 % raise in salary can be said to be equally significant no matter how much money you earn, you should probably logarithmize the data since that would make an additive perturbation equally significant for all values.

$\endgroup$
1
$\begingroup$

(So to be kosher and not mix the question with an answer.)

Right now I am using scale which minimized the following ratio: $$\frac{\sqrt[4]{\langle (x - \bar{x})^4 \rangle}}{\sqrt{\langle (x - \bar{x})^2 \rangle}}$$ That is, after normalizing a variable (i.e. mean 0 and variance 1) I am looking to have the 4th moment as low as possible (so to penalize too long-tailed, or otherwise disperse, distributions).

For me it works (but I am not sure if it's only my using it; and if there are any easy pitfalls).

$\endgroup$
  • 4
    $\begingroup$ This is clever but highly non-robust: the fourth moment is extremely sensitive to outliers. $\endgroup$ – whuber Jul 22 '15 at 15:54
  • $\begingroup$ @whuber In some sense I want to be sensitive to outliers (even a single outlier can wreck the plotting scale). But you are right that it may result e.g. in choosing logarithmic scale based on a single erroneous datapoint. $\endgroup$ – Piotr Migdal Jul 22 '15 at 16:52
  • 2
    $\begingroup$ Seems to me a lot quicker and simpler just to look at minimum and maximum, which will deal with outliers too. Rule 1. (sorry if it seems too obvious) If the minimum is not positive, plain logarithms are inapplicable and probably don't make sense any way. (Side comment: some people are happy with adding a constant first to make logarithms defined.) Rule 2. The benefit of using logarithms increases with max/min. I don't seek cutoffs for Rule 2, as .e.g. in a scatter plot, the benefits of using a transformation will depend on the other variable too. $\endgroup$ – Nick Cox Apr 15 '16 at 10:05
  • $\begingroup$ Why the 4:th moment? Intuitively, it seems smart, but how did you come up with the method? Can you show that it is optimal in some way? $\endgroup$ – HelloGoodbye Sep 17 '18 at 4:10
  • $\begingroup$ Minimizing the standardized 4:th momentum, a.k.a. the kurtosis, is in fact not optimal, since it results in a PDF with two very sharp peaks. See my answer for a (probably) better measurement to optimize. $\endgroup$ – HelloGoodbye Sep 17 '18 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.