Support vector regression on skewed/high kurtosis data

Question

I'm using support vector regression to model some fairly skewed data (with high kurtosis). I've tried modeling the data directly but I'm getting erroneous predictions I think mainly due to the distribution of the data, which is right skewed with very fat tails. I'm pretty sure a few outliers (which are legitimate data points) are affecting the SVR training, and perhaps also in the cross validation, where at the moment I'm optimizing the hyperparameters by minimizing mean-squared error.

I've tried to scale my data before applying SVR (e.g. using a sqrt function to reduce the outliers) as well as use a different hyperparameter minimization function (e.g. absolute error), which seems to give better results, but still not very good. I'm curious if anyone has encountered similar problems and how they approached it? Any suggestions and/or alternate methods most welcome.

Answer 1

You can use skewed or heavy-tailed Lambert W distributions to transform your data to something more well-behaved (disclaimer: I am the author of both papers and the LambertW R package). The advantage over the Box-Cox transformation is that they do not have any positivity restriction, the optimal parameters of the transformation can be estimated (MLE) from the data, and you can also forget the transformation and model your data as a Lambert W x F distribution directly.

The LambertW R package provides several estimators, transformations, methods, etc. I especially recommend a look at

   Gaussianize()
   IGMM()
   MLE_LambertW()

The skewed Lambert W x F distribution is a general framework to make a skewed version of any distribution F. Conversely you can then make your skewed data again symmetric; the distribution of this symmetrized data basically determines what kind of Lambert W x F you have; if the data is just a bit asymmetric, then you might have a skewed Lambert W x Gaussian; if your data is additionally heavy-tailed maybe you can try a skewed Lambert W x t.

Heavy-tailed Lambert W x F are a generalization of Tukey's h distribution, and they provide an inverse-transform to make data Gaussian (also from asymmetric). In the paper I demonstrate that even a Cauchy can be Gaussianized to a level that you - and also several Normality tests - can't distinguish it from a Normal sample.

Answer 2

One way to deal with negative values is to shift variables to the positive range (say to greater or equal to 0.1), apply Box-Cox transform (or just log() for a quick test), and then standardize. The standardization can be important for SVR since SVR relies on quadratic penalty applied to all coefficients uniformly (so SVR is not scale invariant and can benefit from variable standardization). Make sure to check the resulting variable distributions - they should not be skewed much (ideally they should look Gaussian)

Another technique one could try is to apply "spatial sign" transformation to the input vectors x <- x / norm(x) as per “Spatial sign preprocessing: a simple way to impart moderate robustness to multivariate estimators”. J. Chem. Inf. Model (2006) vol. 46 (3) pp. 1402–1409 I did not have much luck with this technique though but the mileage may vary.

Answer 3

One way to approach the solution would be building two models: one for the values which are in line with the distribution and other for the outliers. My suggestion in this regard would be to create a binary response variable (0,1) with 0 being the value if the datapoint is within the bounds of your distribution and 1 if it lies outside. So for the cases of the outliers which you want to keep in your data, you will be having 1 in your target variable and the rest as 0. Now run a logistic regression to predict the probabilities of the outliers and you can multiply the average value for the group of outliers with the individual probabilities to get the predictions. For the rest of the data, you can run your SVM to predict the values.

Because the values are outliers, they will have low probabilities associated with it and even if you take mean of the outliers which will be skewed, the expected value of the outliers will be pulled down by their attached low probabilities and there by making it a more reasonable prediction.

Had met with a similar scenario while predicting claims amount for an Insurance service provider. I had used the above-mentioned technique to increase the performance of my model drastically.

Another way could be taking log transformation of your target variable which is possible if you have only positive value in your target variable. But make sure if you are taking a log transformation of your target variable, while predicting the variable you need to include the error component as well.

So, $\log(Y) = a + B'X + \epsilon$ is your model equation for e.g

then, $Y = \exp(a+B'X+\epsilon)$

You can take a look into the following link for log-transformation: http://www.vims.edu/people/newman_mc/pubs/Newman1993.pdf