I've encountered very similiar but different functions in SVM optimization problem, the diffrence is in the location of regularization constant C.

$\sum_{i=1}^n(1-(y_i(w^tx))_+ +\frac{1}{2C} \left\lVert w\right\rVert^2 \rightarrow min_w$

$C\sum_{i=1}^n(1-(y_i(w^tx))_+ +\frac{1}{2} \left\lVert w\right\rVert^2 \rightarrow min_w$

  1. Is it right that technically they are diffrent functions, and if we graph them we will see the difference, but the location of minimum is the same for both of them?

  2. Also I would like to check whether my intuition about regularization is right. In the first function small values of $С$ makes second term of the summation large, that means with large $C$ we want $\left\lVert w\right\rVert$ to be small and thus miximize the separating margin. In the second equation with large values of $C$ we want to reduce the number of misclassifiactions and also penalise more those errors with lesser value of $y_i(w^tx)$, since $y_i(w^tx)$ is negative for misclassified examples. These are two different ways to arive at the same optimal solution. Is it the right logic?


1 Answer 1

  1. Yes, they're different loss functions, where $Cf_1=f_2$, if we call them as $f_1,f_2$ respectively. Since they're related by a constant chosen by you, i.e. we're not optimizing with respect to $C$, their optimum values are the same.
  2. You're correct in your logic for the second function. Large $C$ penalizes the errors more than weights. Similarly, for the first one, large $C$ means small $1/2C$ and penalizing weights less than errors. So, we're again penalizing errors more. The two objective functions behave the same with same $C$ because they're multiples of each other as described in (1).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.