3
$\begingroup$

The cost function of the Optimal Margin Classifier(non-kernelized SVM) is given as : $$ J(\mathbf{\vec w}, b) = \frac{1}{2}\|\mathbf{\vec w}\|_{2}^{2} + C \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b)) $$

Why do we multiply the regularization parameter $C$ with the error term instead of the regularization term $\frac{1}{2}\|\mathbf{\vec{w}}\|_2^2$ ?

The general form of a cost function is typically this : $$ J_\lambda(\boldsymbol \theta) = J(\boldsymbol \theta) + \lambda R(\boldsymbol \theta) $$ where $R(\boldsymbol \theta)$ is the regularization term. As you can see, the regularization parameter $\lambda$ is always multiplied by the regularization term. So why is the cost function of SVM so peculiar?

$\endgroup$

1 Answer 1

9
$\begingroup$

They're equivalent; just re-express $\lambda = \frac{1}{C}$ and multiply both sides by $\lambda$. Obviously, we assume $C > 0$.

$$J(\mathbf{\vec w}, b) = \frac{1}{2}\|\mathbf{\vec w}\|_{2}^{2} + C \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b)) \\ \frac{1}{C} J(\mathbf{\vec w}, b) = \frac{1}{2C}\|\mathbf{\vec w}\|_{2}^{2} + \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec w}\cdot \mathbf{\vec x}^{(i)} + b))$$

Multiplying the cost function by a positive constant only changes the value of the cost but not the location of the minimum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.