# Differentiation in support vector machine

My problem is we want to minimize $1/2 \cdot \text{Norm}(w)^2$ under suitable constraints $y(i){[w^Tx_i+b]-1}$.

We write the Lagrangian and get the above equations. I have a problem in differentiating with respect to $w$ can somebody elaborate on that? How do we get first equation?

$\frac{\partial L}{\partial w} = 0$

is equivalent to:

$\frac{\partial(\frac{1}{2} w^{T}w - \sum_{i=1}^{N}\alpha_{i}(y_{i}(w^{T}\phi(x_{i}) - b) - 1))}{\partial w} = 0$

which comes down to differentiation of four parts:

$\frac{\partial \frac{1}{2} w^{T}w}{\partial w} = \frac{1}{2}*2*w = w$ (quadratic function of $w$)

$\frac{\partial (\sum_{i=1}^{N}a_{i}y_{i}w^{T}\phi(x_{i}))}{\partial w} = \frac{\partial (a_{1}y_{1}w^{T}\phi(x_{1}) + a_{2}y_{2}w^{T}\phi(x_{2}) + \ldots + a_{N}y_{N}w^{T}\phi(x_{N}))}{\partial w} = a_{i}y_{i}\phi(x_{i}) = \frac{\partial a_{1}y_{1}w^{T}\phi(x_{1})}{\partial w} + \frac{\partial a_{2}y_{2}w^{T}\phi(x_{2})}{\partial w} + \ldots + \frac{\partial a_{N}y_{N}w^{T}\phi(x_{N}))}{\partial w} = a_{1}y_{1}\phi(x_{1} + a_{2}y_{2}\phi(x_{2}) + \ldots + a_{N}y_{N}\phi(x_{N})) = \sum_{i=1}^{N}a_{i}y_{i}\phi(x_{i})$ (linear functions of $w$, $\phi$ also doesn't depend on $w$)

$\frac{\partial a_{i}y_{i}b}{\partial w} = [0, \ldots, 0]$ (constants, not depending on $w$)

and

$\frac{\partial (-a_{i})}{\partial w} = [0, \ldots, 0]$ (constant, not depending on $w$)

Which boils down to the result you've presented:

$\frac{\partial(\frac{1}{2} w^{T}w - \sum_{i=1}^{N}\alpha_{i}(y_{i}(w^{T}\phi(x_{i}) - b) - 1))}{\partial w} = [0, \ldots, 0]$

$w - \sum_{i=1}^{N}\alpha_{i}y_{i}\phi(x_{i}) = [0, \ldots, 0]$

$w = \sum_{i=1}^{N}\alpha_{i}y_{i}\phi(x_{i})$

To explain differentiation over a vector take a look at this link. The first derivative can be thus rewritten more precisely as follows:

$\frac{\partial \frac{1}{2}w^{T}w}{\partial w} = [\frac{\partial \frac{1}{2}w^{T}w}{\partial w_{1}}, \frac{\partial \frac{1}{2}w^{T}w}{\partial w_{2}}, \ldots, \frac{\partial \frac{1}{2}w^{T}w}{\partial w_{m}}] = [\frac{\partial \frac{1}{2}[w_{1}, w_{2}, \ldots, w_{m}]^{T}[w_{1}, w_{2}, \ldots, w_{m}]}{\partial w_{1}}, \frac{\partial \frac{1}{2}[w_{1}, w_{2}, \ldots, w_{m}]^{T}[w_{1}, w_{2}, \ldots, w_{m}]}{\partial w_{2}}, \ldots, \frac{\partial \frac{1}{2}[w_{1}, w_{2}, \ldots, w_{m}]^{T}[w_{1}, w_{2}, \ldots, w_{m}]}{\partial w_{m}}] = [\frac{\partial \frac{1}{2}(w_{1}^{2} + w_{2}^{2} + \ldots + w_{m}^{2})}{\partial w_{1}}, \frac{\partial \frac{1}{2}(w_{1}^{2} + w_{2}^{2} + \ldots + w_{m}^{2})}{\partial w_{2}}, \ldots, \frac{\partial \frac{1}{2}(w_{1}^{2} + w_{2}^{2} + \ldots + w_{m}^{2})}{\partial w_{m}}] = [2*\frac{1}{2}w_{1}, 2*\frac{1}{2}w_{2}, \ldots, 2*\frac{1}{2}w_{m}] = 2*\frac{1}{2}*[w_{1}, w_{2}, \ldots, w_{m}] = 2 * \frac{1}{2} * w = w$

Similarly in other cases, we always differentiate a scalar function over a vector $w$ here, which results in a vector of derivatives over each element $w_{j}, 1 \leq j \leq m$.

• Here we are differentiating scalar function with respect to vector that means we are finding gradient which is a vector . now as you have differentiated wTw which is number with respect to vector w how is that that is my doubt – Milan Amrut Joshi Dec 2 '13 at 10:20
• may be we are differentiating with respect to ||W|| if i am not wrong – Milan Amrut Joshi Dec 2 '13 at 10:23
• @MilanAmrutJoshi No, we differentiate with respect to the vector $w$. I've updated the answer - if it's still not clear let me now. – BartoszKP Dec 2 '13 at 12:34
• I am so happy the first part is clear. I am doubtful about differentiating y(i){[wTxi+b}-1}] with respect to w...i will very happy if you clear this. – Milan Amrut Joshi Dec 2 '13 at 12:58
• Because first part of first equation after differentiating is a vector w as you have shown but the second part which is SUM(alpha i yi Phi(xi)) dose not seem to be a vector this is my problem ... – Milan Amrut Joshi Dec 2 '13 at 13:27

There is a neat trick because:

$\dfrac{\partial}{\partial w}\dfrac{1}{2}\|w\|^2$ $=\dfrac{\partial}{\partial w}\dfrac{1}{2}<w,w>$

We can relabel $w$ as $w_1$ and $w_2$,

$=\dfrac{1}{2}( \dfrac{\partial}{\partial w_1}<w_1,w_2>+\dfrac{\partial}{\partial w_2}<w_1,w_2>)$

$=\dfrac{1}{2}(w_2+w_1)$

$=\dfrac{1}{2}(w+w)$

$=w$

This is useful in the case where we want the derivative of a more complicated expression because it allows us to isolate the variable we are taking the derivative with respect to. For example, to minimize

$\|XW-Y\|^2$

One of the terms we need to take the derivative of is

$<XW,XW>$

We can use the fact that $tr(AB)=<A^T,B>$ to say

$<XW,XW>=tr((XW)^TXW)$

$=tr(W^TX^TXW)$

$=<X^TXW,W>$

where the derivative in terms of the lone $W$ is now just $X^TXW$!