4
$\begingroup$

I know that larger values of C in SVM cause the classifier to attempt to classify more points at the expense of a wider margin (and vice versa for smaller values of C). Therefore, is it correct to say that, in terms of a bias-variance tradeoff, larger values of C increase the variance and decrease the bias of the model?

$\endgroup$
1
  • $\begingroup$ Correction: If we have a large C, we prefer small number of misclassified examples because the term in the objective function of SVM, $C\sum_{i=1}^{n} \varepsilon_{i} $, will dominate, so, we will have a smaller margin not wider! $\endgroup$
    – ARAT
    May 21, 2020 at 15:01

1 Answer 1

8
$\begingroup$

Yes.

This can be related to the "regular" regularization tradeoff in the following way. SVMs are usually formulated like $$ \min_{w} \mathrm{regularization}(w) + C \, \mathrm{loss}(w; X, y) ,$$ whereas ridge regression / LASSO / etc are formulated like: $$ \min_{w} \mathrm{loss}(w; X, y) + \lambda \, \mathrm{regularization}(w) .$$ The two are of course equivalent with $C = \tfrac{1}{\lambda}$. I think it's more intuitive to see in the latter case, though, that as $\lambda \to \infty$ your solution is determined entirely by the regularization term, so that your bias is very high and variance very low; as $\lambda \to 0$, you take away all the regularization bias but also lose its variance reduction.

$\endgroup$
5
  • $\begingroup$ Thanks. Just wondering, why isn't $\lambda$ used in the SVM cost function? It seems confusing to have a new parameter, $C$. $\endgroup$ Oct 16, 2016 at 12:58
  • $\begingroup$ I agree. I'm sure the reason is just historical but don't know the details. $\endgroup$
    – Danica
    Oct 16, 2016 at 13:00
  • $\begingroup$ This link says exactly the opposite. The larger the value of $C$, the lower the variance and the higher the bias (because it supposedly generalizes better). Which is correct? Do large values of $C$ correspond to large or small margins? $\endgroup$ Dec 17, 2020 at 5:01
  • $\begingroup$ @VivekSubramanian Looking at the first equation: as $C$ grows, the optimizer cares only about the “loss” term and not the “regularization.” In soft-margin SVMs, the “loss” term controls how much the margin is violated, and the “regularization” term actually controls how big the margin is. Very large $C$ means the model wants a very “hard” margin but doesn’t care how big it is; smaller $C$ allows a “softer” margin (some violations) to make it bigger. Requiring a very “hard” margin means the variance is high, because resampling data points could change your predictor significantly. $\endgroup$
    – Danica
    Dec 17, 2020 at 16:43
  • $\begingroup$ The discussion in your link has the meaning of $C$ reversed. To see that what I said about loss corresponding to margin “hardness” and regularization being margin size, remember that if $y \in \{1, -1\}$, then \begin{gather*} \mathrm{loss} = \sum_{i=1}^n \max(0, 1 - y_i f_w(X_i)) \\ \mathrm{regularization} = \lVert w \rVert^2 .\end{gather*} If $f_w(X_i)$ is correct with abs value at least 1, $\mathrm{loss} = 0$: it measures violations of the margin. A smaller $\lVert w \rVert$, meanwhile, means a larger margin. $\endgroup$
    – Danica
    Dec 17, 2020 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.