I know that larger values of C in SVM cause the classifier to attempt to classify more points at the expense of a wider margin (and vice versa for smaller values of C). Therefore, is it correct to say that, in terms of a bias-variance tradeoff, larger values of C increase the variance and decrease the bias of the model?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Correction: If we have a large C, we prefer small number of misclassified examples because the term in the objective function of SVM, $C\sum_{i=1}^{n} \varepsilon_{i} $, will dominate, so, we will have a smaller margin not wider! $\endgroup$– ARATMay 21, 2020 at 15:01
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
Yes.
This can be related to the "regular" regularization tradeoff in the following way. SVMs are usually formulated like $$ \min_{w} \mathrm{regularization}(w) + C \, \mathrm{loss}(w; X, y) ,$$ whereas ridge regression / LASSO / etc are formulated like: $$ \min_{w} \mathrm{loss}(w; X, y) + \lambda \, \mathrm{regularization}(w) .$$ The two are of course equivalent with $C = \tfrac{1}{\lambda}$. I think it's more intuitive to see in the latter case, though, that as $\lambda \to \infty$ your solution is determined entirely by the regularization term, so that your bias is very high and variance very low; as $\lambda \to 0$, you take away all the regularization bias but also lose its variance reduction.
-
$\begingroup$ Thanks. Just wondering, why isn't $\lambda$ used in the SVM cost function? It seems confusing to have a new parameter, $C$. $\endgroup$ Oct 16, 2016 at 12:58
-
$\begingroup$ I agree. I'm sure the reason is just historical but don't know the details. $\endgroup$– DanicaOct 16, 2016 at 13:00
-
$\begingroup$ This link says exactly the opposite. The larger the value of $C$, the lower the variance and the higher the bias (because it supposedly generalizes better). Which is correct? Do large values of $C$ correspond to large or small margins? $\endgroup$ Dec 17, 2020 at 5:01
-
$\begingroup$ @VivekSubramanian Looking at the first equation: as $C$ grows, the optimizer cares only about the “loss” term and not the “regularization.” In soft-margin SVMs, the “loss” term controls how much the margin is violated, and the “regularization” term actually controls how big the margin is. Very large $C$ means the model wants a very “hard” margin but doesn’t care how big it is; smaller $C$ allows a “softer” margin (some violations) to make it bigger. Requiring a very “hard” margin means the variance is high, because resampling data points could change your predictor significantly. $\endgroup$– DanicaDec 17, 2020 at 16:43
-
$\begingroup$ The discussion in your link has the meaning of $C$ reversed. To see that what I said about loss corresponding to margin “hardness” and regularization being margin size, remember that if $y \in \{1, -1\}$, then \begin{gather*} \mathrm{loss} = \sum_{i=1}^n \max(0, 1 - y_i f_w(X_i)) \\ \mathrm{regularization} = \lVert w \rVert^2 .\end{gather*} If $f_w(X_i)$ is correct with abs value at least 1, $\mathrm{loss} = 0$: it measures violations of the margin. A smaller $\lVert w \rVert$, meanwhile, means a larger margin. $\endgroup$– DanicaDec 17, 2020 at 16:55