11
$\begingroup$

I am trying to find out which SVR is suited for that kind of data.

I know 4 types of SVRs:

  • epsilon
  • nu
  • least squares and
  • linear.

I understand linear SVR is more or less like lasso with L1 Reg, But what is the difference between the remaining 3 techniques?

$\endgroup$
11
$\begingroup$

In $\nu$-SVR, the parameter $\nu$ is used to determine the proportion of the number of support vectors you desire to keep in your solution with respect to the total number of samples in the dataset. In $\nu$-SVR the parameter $\epsilon$ is introduced into the optimization problem formulation and it is estimated automatically (optimally) for you.

However, in $\epsilon$-SVR you have no control on how many data vectors from the dataset become support vectors, it could be a few, it could be many. Nonetheless, you will have total control of how much error you will allow your model to have, and anything beyond the specified $\epsilon$ will be penalized in proportion to $C$, which is the regularization parameter.

Depending of what I want, I choose between the two. If I am really desperate for a small solution (fewer support vectors) I choose $\nu$-SVR and hope to obtain a decent model. But if I really want to control the amount of error in my model and go for the best performance, I choose $\epsilon$-SVR and hope that the model is not too complex (lots of support vectors).

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

The difference between $\epsilon$-SVR and $\nu$-SVR is how the training problem is parametrized. Both use a type of hinge loss in the cost function. The $\nu$ parameter in $\nu$-SVM can be used to control the amount of support vectors in the resulting model. Given appropriate parameters, the exact same problem is solved.1

Least squares SVR differs from the other two by using squared residuals in the cost function instead of hinge loss.

1: C.-C. Chang and C.-J. Lin. Training $\nu$-support vector regression: Theory and algorithms. Neural Computation, 14(8):1959-1977, 2002.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your answer Marc. So can we distinguish the suitable method based on the dataset we have? If so, can you give me some pointers? I have 40000 samples with 200 distinct outputs. So it can be thought of as 200 sets of 200 unique samples. The input for all 40000 are different though, just the output is unique for 200 samples. $\endgroup$ – Sharath Chandra Apr 18 '14 at 3:39
0
$\begingroup$

I like both Pablo and Marc answers. One additional point:

In the paper cited by Marc there is written (section 4)

"The motivation of $\nu$-SVR is that it may not be easy to decide the parameter $\epsilon$. Hence, here we are interested in the possible range of $\epsilon$. As expected, results show that $\epsilon$ is related to the target values $y$.

[...]

As the effective range of $\epsilon$ is affected by the target values $y$, a way to solve this difficulty for $\epsilon$-SVM is by scaling the target values before training the data. For example, if all target values are scaled to $[-1,+1]$, then the effective range of $\epsilon$ will be $[0, 1]$, the same as that of $\nu$. Then it may be easier to choose $\epsilon$."

That brings me to think that it should be easier to scale your target variables and use $\epsilon$-SVR, than trying to decide whether to use $\epsilon -$ or $\nu -$ SVR.

What do you think?

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.