2
$\begingroup$

I know that larger values of C in SVM cause the classifier to attempt to classify more points at the expense of a wider margin (and vice versa for smaller values of C). Therefore, is it correct to say that, in terms of a bias-variance tradeoff, larger values of C increase the variance and decrease the bias of the model?

$\endgroup$
5
$\begingroup$

Yes.

This can be related to the "regular" regularization tradeoff in the following way. SVMs are usually formulated like $$ \min_{w} \mathrm{regularization}(w) + C \, \mathrm{loss}(w; X, y) ,$$ whereas ridge regression / LASSO / etc are formulated like: $$ \min_{w} \mathrm{loss}(w; X, y) + \lambda \, \mathrm{regularization}(w) .$$ The two are of course equivalent with $C = \tfrac{1}{\lambda}$. I think it's more intuitive to see in the latter case, though, that as $\lambda \to \infty$ your solution is determined entirely by the regularization term, so that your bias is very high and variance very low; as $\lambda \to 0$, you take away all the regularization bias but also lose its variance reduction.

$\endgroup$
  • $\begingroup$ Thanks. Just wondering, why isn't $\lambda$ used in the SVM cost function? It seems confusing to have a new parameter, $C$. $\endgroup$ – sir_thursday Oct 16 '16 at 12:58
  • $\begingroup$ I agree. I'm sure the reason is just historical but don't know the details. $\endgroup$ – Dougal Oct 16 '16 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.