This can be related to the "regular" regularization tradeoff in the following way. SVMs are usually formulated like $$\min_{w} \mathrm{regularization}(w) + C \, \mathrm{loss}(w; X, y) ,$$ whereas ridge regression / LASSO / etc are formulated like: $$\min_{w} \mathrm{loss}(w; X, y) + \lambda \, \mathrm{regularization}(w) .$$ The two are of course equivalent with $C = \tfrac{1}{\lambda}$. I think it's more intuitive to see in the latter case, though, that as $\lambda \to \infty$ your solution is determined entirely by the regularization term, so that your bias is very high and variance very low; as $\lambda \to 0$, you take away all the regularization bias but also lose its variance reduction.
• Thanks. Just wondering, why isn't $\lambda$ used in the SVM cost function? It seems confusing to have a new parameter, $C$. – sir_thursday Oct 16 '16 at 12:58