Unusual transformation of percentage data

Question

I am reviewing some support vector machine (SVM) work by someone who has left my company. The target data is a heavily left skewed set of percentage values. He chose to make an unusual transformation that I don't understand - and can't find anywhere else.

transformed_x = log10(max(x) - x +1)

This appears to be a more successful way to make the data normally distributed than more common transformations such as logit and arcsin - but I don't really understand why? Only power transformations determined from Tukey/Box-Cox (x^19.3 !!) seem to do better.

As a side question - is it only the target data in SVM that should be normally distributed, or should I also address the input variables too? He has only done the target variable.

Answer 1

Thanks to @whuber and @Aksakal, I can answer my own question. It's actually fairly simple and I was over-complicating things because the variable of interest was a percentage and this threw me off into too complicated topics -- I hope this answer will assist others who have come across similar problems, and I learnt some more general things I'll mention too.

First I'll explain a simple case without percentage data.

Let's say we have some measurement that always gives a small number, but never goes below zero. Such as sieve residue when screening minerals at a particular size. Clearly this type of measurement will often give a right skewed distribution of data -- much like a log-normal distribution. Obviously then, it might be sensible to take a log transform to try to convert the data into a normal distribution. In fact, I've done this very thing in the past -- so it's a bit embarrassing I didn't get this solution myself anyway. For example, check slide 9 of this presentation.

In this screening scenario, we are talking about a potentially real log-normal distribution -- because there's no upper bound on the value. But in the original question I was dealing with (left-skewed) percentage data, and tied myself up in knots thinking that only something like a logit transform should be appropriate -- because the data must be bounded at both ends -- but, as noted in my original question, this is not what the original author of the SVM work did (and it doesn't work well).

When I look at the data, from an empirical point of view, and don't worry about the hard and fast principle of the data being bounded, my data won't ever get too close to either 0 % or 100 %, so I don't need to worry about it and can do something else. Which is what the original author obviously realised.

The first thing to do is to flip the data to convert it to right-skewed -- to something that has a chance of being near to a log-normal distribution (and hence can be transformed to a normal distribution by a simple log transform).

This is the part max(x) - x in my original question, and converts

to

The + 1 is then there to avoid the error caused by having a zero value by doing this simple flip (i.e. when max(x) is the x).

Thinking about this, after understanding the flip, I wondered why the original author did max(x) - x + 1, and not 100 - x, given that we know physically the maximum theoretical value is 100 %? What I've realised (I think), is that you get the best possible log transformation -- i.e. the most normal data -- when the offset you apply gives your minimum datum as exactly 1. Looking at a Q-Q plot such as

If the offset is less than 1, then the data at the bottom end of the Q-Q plot tend to stray further from normality (log10 varies quickly as x goes below 1). As the offset becomes > 1, then a similar thing happens, but at the upper end of the Q-Q plot.

So -- maybe this is particular to my case, feel free to comment/edit if that's the case -- an offset of 1 is the optimum value to add to get the most normal data after the log10 transformation. Hence, max(x) - x + 1 is better than 100 - x, unless max(x) = 99.

As long as none of my data gets too close to the upper bound of 100 %, then a simple flip will give something that can be log transformed. I would imagine if the data was right-skewed (and not too near zero) then the flipping would be unnecessary and the log transform would be suitable straight away.

It appears the original author of the SVM work has taken a very empirical look at the data, realised all this way quicker than I did, and gone for the simplest option. One that may have some physical validity

This is another thing I've learned thanks to @whuber's and @Aksakal's comments. Don't obsess about finding the best transformation -- either formally (because you know your data is bounded in some way the transform implies it isn't, as long as you are far enough from the bounds, it's fine) -- or numerically.

The latter point is relevant because a Tukey power ladder (or Box-Cox) on the data implies the best transform is a power transform of over 19! Similarly, if I do a ladder for 100 - x (just for fun) it gives -2 as the best option, but this doesn't seem particularly physical. If I do it for max(x) - x + 1 then, sure enough, it proposes a log transform (lambda = 0).

So, now I understand why the original author did what he did, and, thanks to @whuber and @Aksakal, have also learnt:

1. Don't obsess about what's formally/mathematically correct, if you data can be represented more simply.
2. Keep in mind what is physically sensible. You want to balance what seems physically sensible, with what gives you the best results.
3. Be practical/empirical and if something simple, that is easy to understand/work with, gives you good enough results -- stick with that.

Edit: typos/clarity.