3
$\begingroup$

In SVM, we minimise the Lagrangian (primal form):

$$L_p = \frac{1}{2}\Vert w \Vert^2 - \sum_{i=1}^m \alpha_i[y_i(w \cdot x_i + b)-1]$$

by taking partial derivatives with respect to $w$ and $b$, obtaining the two equations:

$$w = \sum_i \alpha_i y_i x_i$$ and $$\sum \alpha_i y_i = 0,$$ with the coefficients $\alpha_i \geq 0$. The weight vector is expressed as a linear combination of the support vectors, i.e. the vectors that lie on the margin, for which the coefficients $ \alpha_i $ are non-zero.

Using the formulation that includes slack variables, we obtain this new constraint: $$0 \leq \alpha_i \leq C,$$ where $C$ determines how much points on the other side of the margin will be penalised.

My question is the following: What would be the intuitive and geometrical meaning of constraining the sum of the support vector coefficients, say like this:

$$\sum_i \alpha_i = 1?$$

$\endgroup$
1
$\begingroup$

It's basically a harder constraint. Note that $\sum_i\alpha_i=C$ is equivalent to $\sum_i \alpha_i=1$ since we can redefine $\hat{\alpha}_i=\alpha_i/C$. So the original constraint becomes $0\leq \alpha_i\leq 1$, whereas now you have $\sum_i\alpha_i=1$. I'm not sure if there's a geometric meaning but it would in some sense be a form of L1 regularization on your slack variables, causing your classifier to be extra sensitive to certain classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.