6
$\begingroup$

I was reading the documentation of sklearn SVM and saw these two statements

  • Still effective in cases where number of dimensions is greater than the number of samples
  • If the number of features is much greater than the number of samples, the method is likely to give poor performances.

Now my understanding regarding SVM is fairly limited (the reason why I am reading the article) but the above 2 statements seem contradictory. But I believe that I am missing something. What is missing in my understanding due to which the above 2 are not contradictory?

$\endgroup$
1
  • 3
    $\begingroup$ "greater" vs "much greater" $\endgroup$
    – D.W.
    Commented Jan 4, 2017 at 18:59

2 Answers 2

7
$\begingroup$

SVMs, like many other linear models, are based on empirical risk minimization, which leads us to an optimization of this sort:

$$\min_w\sum\ell(x_i, y_i, w)+\lambda\cdot r(w)$$

Where $\ell$ is a loss function (the Hinge-loss in SVMs) and $r$ is a regularization function.

The SVM is a squared $\ell_2$-regularized linear model, i.e. $r(w) = \|w\|^2_2$. This guards against huge coefficients, as one would say in regression terms, as the coefficient magnitudes are themselves penalized in the optimization.

Besides that, the regularization allows for a unique solution in the $p> n$ situation, so the 1st statement is true.

The problem when $p \gg n$ is that the bias introduced by regularization can be so large towards the training data the model heavily underperforms. It doesn't mean SVMs can't be used in that scenario (they are usually employed in gene expression data, for example, where $p$ can be thousands times larger than $n$).

So I see no contradiction in statements. The 2nd statement is more likely a warning against overfitting.

$\endgroup$
6
  • 2
    $\begingroup$ I guess it depends on the interpretation "much greater" vs "greater" :) definitely, the more samples the better in any case. $\endgroup$ Commented Jan 4, 2017 at 18:25
  • $\begingroup$ In your answer p refers to number of features and n refers to number of samples. Correct? Would you consider understanding linear models necessary to understand SVM? This question may seem off-topic but is not. Asking this because I am not able to understand what is written and am deciding whether I should proceed in sequence here scikit-learn.org/stable/user_guide.html before coming back here to understand the answer. $\endgroup$ Commented Jan 4, 2017 at 18:37
  • 1
    $\begingroup$ Yes, $p$ is the number of features and $n$ the number of samples. I think a good understanding of linear models is interesting, but in no way necessary to employ SVMs. If you want to understand what the SVM do, then yes, basic concepts of linear models, kernels and optimization are necessary. $\endgroup$
    – Firebug
    Commented Jan 4, 2017 at 18:43
  • 1
    $\begingroup$ @AseemBansal FYI I tried to write some easy introduction to SVM here: How does a Support Vector Machine (SVM) work?. As Firebug points out, it makes use of basic concepts of linear models, kernels and optimization. $\endgroup$ Commented Jan 5, 2017 at 6:36
  • 1
    $\begingroup$ @Firebug Thanks for the explanation. I want to apply them but don't want to be completely unaware when tuning parameters about what changing them actually means. My assumption is that if I understand the various algorithms better then I can make better choices when tuning parameters and choosing between algorithms. $\endgroup$ Commented Jan 5, 2017 at 15:59
4
$\begingroup$
  • Still effective in cases where number of dimensions is greater than the number of samples
  • If the number of features is much greater than the number of samples, the method is likely to give poor performances.

These two points are pretty much contradictory, as one feature is typically one-dimensional (some features may be multidimensional but it is less common).

The first point is the correct one, see SVM, Overfitting, curse of dimensionality and Number of features vs. number of observations for more details.

$\endgroup$
2
  • 1
    $\begingroup$ Just to be sure, according to you the 2nd statement is incorrect. Is my understanding correct? $\endgroup$ Commented Jan 4, 2017 at 18:16
  • 1
    $\begingroup$ @AseemBansal your understanding is correct. $\endgroup$ Commented Jan 4, 2017 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.