1
$\begingroup$

I am reviewing some support vector machine (SVM) work by someone who has left my company. The target data is a heavily left skewed set of percentage values. He chose to make an unusual transformation that I don't understand - and can't find anywhere else.

transformed_x = log10(max(x) - x +1)

This appears to be a more successful way to make the data normally distributed than more common transformations such as logit and arcsin - but I don't really understand why? Only power transformations determined from Tukey/Box-Cox (x^19.3 !!) seem to do better.

As a side question - is it only the target data in SVM that should be normally distributed, or should I also address the input variables too? He has only done the target variable.

$\endgroup$
  • $\begingroup$ what is min(x) and max(x)? $\endgroup$ – Aksakal Mar 29 '18 at 14:30
  • 2
    $\begingroup$ This transformation, when parsed into its components, is not terribly unusual: the direction of the data was flipped and then a log transformation (with a start value of 1) was applied: this is a simple generalization of a Box-Cox transformation. A power of $19.3$ certainly is ridiculous and is exceptionally unlikely to be of any use--but no reasonable Box-Cox analysis would produce such a power in the first place. The motivation for the original transformation could lie in what kind of quantity x represents. What does it measure? See stats.stackexchange.com/questions/4831. $\endgroup$ – whuber Mar 29 '18 at 14:50
  • 1
    $\begingroup$ @whuber - sorry answered before the edit. For your question: I have a dataset that is a series of (mineral) samples. For each sample is a range of measured parameters that can be values that are always > 0 (e.g. residue at some screen size). They can be percentage values, such as proportion of mineral that is some oxide like SiO2. Or other things. Then the target is the brightness (% reflectance at some wavelength) when the mineral is milled a certain way. He was able to reasonably predict the brightness and I'm just making sure I understand what and why he did. $\endgroup$ – Mooks Mar 29 '18 at 15:03
  • 1
    $\begingroup$ When you have percentages of a whole, it can be reasonable to re-express them in terms of the complementary percentages: that's close to what subtracting from the maximum does. The choice of the maximum (rather than 100%) as well as adding 1 is ad hoc but not unusual: the proof of the pudding lies in whether it is effective and generalizable. For more about the art of finding an appropriate Box-Cox transformation see stats.stackexchange.com/search?q=+box+cox+ladder; also see stats.stackexchange.com/questions/6150; stats.stackexchange.com/questions/30728. $\endgroup$ – whuber Mar 29 '18 at 15:03
  • 1
    $\begingroup$ @whuber. Thanks again for your help - it’s all clear to me now. And depressingly obvious once you and explained it. Should I provide my own answer to my question - which might help others with a similar misunderstanding - or just leave it? $\endgroup$ – Mooks Apr 1 '18 at 11:40
2
$\begingroup$

Thanks to @whuber and @Aksakal, I can answer my own question. It's actually fairly simple and I was over-complicating things because the variable of interest was a percentage and this threw me off into too complicated topics -- I hope this answer will assist others who have come across similar problems, and I learnt some more general things I'll mention too.

First I'll explain a simple case without percentage data.

Let's say we have some measurement that always gives a small number, but never goes below zero. Such as sieve residue when screening minerals at a particular size. Clearly this type of measurement will often give a right skewed distribution of data -- much like a log-normal distribution. Obviously then, it might be sensible to take a log transform to try to convert the data into a normal distribution. In fact, I've done this very thing in the past -- so it's a bit embarrassing I didn't get this solution myself anyway. For example, check slide 9 of this presentation.

In this screening scenario, we are talking about a potentially real log-normal distribution -- because there's no upper bound on the value. But in the original question I was dealing with (left-skewed) percentage data, and tied myself up in knots thinking that only something like a logit transform should be appropriate -- because the data must be bounded at both ends -- but, as noted in my original question, this is not what the original author of the SVM work did (and it doesn't work well).

When I look at the data, from an empirical point of view, and don't worry about the hard and fast principle of the data being bounded, my data won't ever get too close to either 0 % or 100 %, so I don't need to worry about it and can do something else. Which is what the original author obviously realised.

The first thing to do is to flip the data to convert it to right-skewed -- to something that has a chance of being near to a log-normal distribution (and hence can be transformed to a normal distribution by a simple log transform).

This is the part max(x) - x in my original question, and converts

Untransformed data

to

Flipped data

The + 1 is then there to avoid the error caused by having a zero value by doing this simple flip (i.e. when max(x) is the x).

Thinking about this, after understanding the flip, I wondered why the original author did max(x) - x + 1, and not 100 - x, given that we know physically the maximum theoretical value is 100 %? What I've realised (I think), is that you get the best possible log transformation -- i.e. the most normal data -- when the offset you apply gives your minimum datum as exactly 1. Looking at a Q-Q plot such as

Q-Q plot of transformed data, according to transformation in original question

If the offset is less than 1, then the data at the bottom end of the Q-Q plot tend to stray further from normality (log10 varies quickly as x goes below 1). As the offset becomes > 1, then a similar thing happens, but at the upper end of the Q-Q plot.

So -- maybe this is particular to my case, feel free to comment/edit if that's the case -- an offset of 1 is the optimum value to add to get the most normal data after the log10 transformation. Hence, max(x) - x + 1 is better than 100 - x, unless max(x) = 99.

As long as none of my data gets too close to the upper bound of 100 %, then a simple flip will give something that can be log transformed. I would imagine if the data was right-skewed (and not too near zero) then the flipping would be unnecessary and the log transform would be suitable straight away.

It appears the original author of the SVM work has taken a very empirical look at the data, realised all this way quicker than I did, and gone for the simplest option. One that may have some physical validity

This is another thing I've learned thanks to @whuber's and @Aksakal's comments. Don't obsess about finding the best transformation -- either formally (because you know your data is bounded in some way the transform implies it isn't, as long as you are far enough from the bounds, it's fine) -- or numerically.

The latter point is relevant because a Tukey power ladder (or Box-Cox) on the data implies the best transform is a power transform of over 19! Similarly, if I do a ladder for 100 - x (just for fun) it gives -2 as the best option, but this doesn't seem particularly physical. If I do it for max(x) - x + 1 then, sure enough, it proposes a log transform (lambda = 0).

So, now I understand why the original author did what he did, and, thanks to @whuber and @Aksakal, have also learnt:

  1. Don't obsess about what's formally/mathematically correct, if you data can be represented more simply.
  2. Keep in mind what is physically sensible. You want to balance what seems physically sensible, with what gives you the best results.
  3. Be practical/empirical and if something simple, that is easy to understand/work with, gives you good enough results -- stick with that.

Edit: typos/clarity.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.