Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I am trying to find out which SVR is suited for that kind of data.
I know 4 types of SVRs:
I understand linear SVR is more or less like lasso with L1 Reg, But what is the difference between the remaining 3 techniques?
In $\nu$-SVR, the parameter $\nu$ is used to determine the proportion of the number of support vectors you desire to keep in your solution with respect to the total number of samples in the dataset. In $\nu$-SVR the parameter $\epsilon$ is introduced into the optimization problem formulation and it is estimated automatically (optimally) for you.
However, in $\epsilon$-SVR you have no control on how many data vectors from the dataset become support vectors, it could be a few, it could be many. Nonetheless, you will have total control of how much error you will allow your model to have, and anything beyond the specified $\epsilon$ will be penalized in proportion to $C$, which is the regularization parameter.
Depending of what I want, I choose between the two. If I am really desperate for a small solution (fewer support vectors) I choose $\nu$-SVR and hope to obtain a decent model. But if I really want to control the amount of error in my model and go for the best performance, I choose $\epsilon$-SVR and hope that the model is not too complex (lots of support vectors).
The difference between $\epsilon$-SVR and $\nu$-SVR is how the training problem is parametrized. Both use a type of hinge loss in the cost function. The $\nu$ parameter in $\nu$-SVM can be used to control the amount of support vectors in the resulting model. Given appropriate parameters, the exact same problem is solved.1
Least squares SVR differs from the other two by using squared residuals in the cost function instead of hinge loss.
1: C.-C. Chang and C.-J. Lin. Training $\nu$-support vector regression: Theory and algorithms. Neural Computation, 14(8):1959-1977, 2002.
I like both Pablo and Marc answers. One additional point:
In the paper cited by Marc there is written (section 4)
"The motivation of $\nu$-SVR is that it may not be easy to decide the parameter $\epsilon$. Hence, here we are interested in the possible range of $\epsilon$. As expected, results show that $\epsilon$ is related to the target values $y$.
[...]
As the effective range of $\epsilon$ is affected by the target values $y$, a way to solve this difficulty for $\epsilon$-SVM is by scaling the target values before training the data. For example, if all target values are scaled to $[-1,+1]$, then the effective range of $\epsilon$ will be $[0, 1]$, the same as that of $\nu$. Then it may be easier to choose $\epsilon$."
That brings me to think that it should be easier to scale your target variables and use $\epsilon$-SVR, than trying to decide whether to use $\epsilon -$ or $\nu -$ SVR.
What do you think?
