1
$\begingroup$

I am working on Java implementation of the SVM.

Suppose that I have the following loss function that needs to be minimized on weight-space:

$$\frac{1}{N} \sum\limits_{i=1}^N \left(\frac{\lambda}{2}\cdot \|w\| + \max(0,1-y_i \cdot f(x_i) )\right)$$

where

  • $N$ - number of the features
  • $x_i$ - the particular feature (basically it is a vector with the length $M$);
  • $w$ - weights that have to be found by gradient descent method;

Based on the following presentation http://www.robots.ox.ac.uk/~az/lectures/ml/lect2.pdf

The update rules are the following:

  • $w_{t+1}\leftarrow w_t-\eta (\lambda w_t-y_i x_i)$, when $y_i (w^Tx_i+b)<0$
  • $w_{t+1}\leftarrow w_t-\eta \lambda w_t$, otherwise

However, I don't understand the following:

$y_i$ might contain only ${0,1}$ - it is scalar, but $x_i$ is vector. How can I simply multiply vector by a scalar and get a vector?

Is it simply the following multiplication

$y_i x_i = (y_i x_i^1,..., y_i x_i^M)$?

$\endgroup$
  • $\begingroup$ I think y_i should be either 1 or -1 under the SVM context. $\endgroup$ – Sean Do Apr 24 '18 at 9:13
2
$\begingroup$

Yes, it is as you conjectured, i.e., each element of the vector is multiplied by the scalar, with the understanding that if $x_i$ is a column vector, as it is in this case, then $y_i x_i$ is also a column vector (which is not clear from your notation).

Given that $y_i$ is either 0 or 1, then $y_i x_i$ will be the M by 1 column vector of zeros if $y_i = 0$, and will be the M by 1 column vector $x_i$, if $y_i = 1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.