enter image description here

My problem is we want to minimize $1/2 \cdot \text{Norm}(w)^2$ under suitable constraints $y(i){[w^Tx_i+b]-1}$.

We write the Lagrangian and get the above equations. I have a problem in differentiating with respect to $w$ can somebody elaborate on that? How do we get first equation?


$\frac{\partial L}{\partial w} = 0$

is equivalent to:

$\frac{\partial(\frac{1}{2} w^{T}w - \sum_{i=1}^{N}\alpha_{i}(y_{i}(w^{T}\phi(x_{i}) - b) - 1))}{\partial w} = 0$

which comes down to differentiation of four parts:

$\frac{\partial \frac{1}{2} w^{T}w}{\partial w} = \frac{1}{2}*2*w = w$ (quadratic function of $w$)

$\frac{\partial (\sum_{i=1}^{N}a_{i}y_{i}w^{T}\phi(x_{i}))}{\partial w} = \frac{\partial (a_{1}y_{1}w^{T}\phi(x_{1}) + a_{2}y_{2}w^{T}\phi(x_{2}) + \ldots + a_{N}y_{N}w^{T}\phi(x_{N}))}{\partial w} = a_{i}y_{i}\phi(x_{i}) = \frac{\partial a_{1}y_{1}w^{T}\phi(x_{1})}{\partial w} + \frac{\partial a_{2}y_{2}w^{T}\phi(x_{2})}{\partial w} + \ldots + \frac{\partial a_{N}y_{N}w^{T}\phi(x_{N}))}{\partial w} = a_{1}y_{1}\phi(x_{1} + a_{2}y_{2}\phi(x_{2}) + \ldots + a_{N}y_{N}\phi(x_{N})) = \sum_{i=1}^{N}a_{i}y_{i}\phi(x_{i})$ (linear functions of $w$, $\phi$ also doesn't depend on $w$)

$\frac{\partial a_{i}y_{i}b}{\partial w} = [0, \ldots, 0] $ (constants, not depending on $w$)


$\frac{\partial (-a_{i})}{\partial w} = [0, \ldots, 0]$ (constant, not depending on $w$)

Which boils down to the result you've presented:

$\frac{\partial(\frac{1}{2} w^{T}w - \sum_{i=1}^{N}\alpha_{i}(y_{i}(w^{T}\phi(x_{i}) - b) - 1))}{\partial w} = [0, \ldots, 0]$

$w - \sum_{i=1}^{N}\alpha_{i}y_{i}\phi(x_{i}) = [0, \ldots, 0]$

$w = \sum_{i=1}^{N}\alpha_{i}y_{i}\phi(x_{i})$

To explain differentiation over a vector take a look at this link. The first derivative can be thus rewritten more precisely as follows:

$\frac{\partial \frac{1}{2}w^{T}w}{\partial w} = [\frac{\partial \frac{1}{2}w^{T}w}{\partial w_{1}}, \frac{\partial \frac{1}{2}w^{T}w}{\partial w_{2}}, \ldots, \frac{\partial \frac{1}{2}w^{T}w}{\partial w_{m}}] = [\frac{\partial \frac{1}{2}[w_{1}, w_{2}, \ldots, w_{m}]^{T}[w_{1}, w_{2}, \ldots, w_{m}]}{\partial w_{1}}, \frac{\partial \frac{1}{2}[w_{1}, w_{2}, \ldots, w_{m}]^{T}[w_{1}, w_{2}, \ldots, w_{m}]}{\partial w_{2}}, \ldots, \frac{\partial \frac{1}{2}[w_{1}, w_{2}, \ldots, w_{m}]^{T}[w_{1}, w_{2}, \ldots, w_{m}]}{\partial w_{m}}] = [\frac{\partial \frac{1}{2}(w_{1}^{2} + w_{2}^{2} + \ldots + w_{m}^{2})}{\partial w_{1}}, \frac{\partial \frac{1}{2}(w_{1}^{2} + w_{2}^{2} + \ldots + w_{m}^{2})}{\partial w_{2}}, \ldots, \frac{\partial \frac{1}{2}(w_{1}^{2} + w_{2}^{2} + \ldots + w_{m}^{2})}{\partial w_{m}}] = [2*\frac{1}{2}w_{1}, 2*\frac{1}{2}w_{2}, \ldots, 2*\frac{1}{2}w_{m}] = 2*\frac{1}{2}*[w_{1}, w_{2}, \ldots, w_{m}] = 2 * \frac{1}{2} * w = w$

Similarly in other cases, we always differentiate a scalar function over a vector $w$ here, which results in a vector of derivatives over each element $w_{j}, 1 \leq j \leq m$.

  • $\begingroup$ Here we are differentiating scalar function with respect to vector that means we are finding gradient which is a vector . now as you have differentiated wTw which is number with respect to vector w how is that that is my doubt $\endgroup$ Dec 2 '13 at 10:20
  • $\begingroup$ may be we are differentiating with respect to ||W|| if i am not wrong $\endgroup$ Dec 2 '13 at 10:23
  • $\begingroup$ @MilanAmrutJoshi No, we differentiate with respect to the vector $w$. I've updated the answer - if it's still not clear let me now. $\endgroup$
    – BartoszKP
    Dec 2 '13 at 12:34
  • $\begingroup$ I am so happy the first part is clear. I am doubtful about differentiating y(i){[wTxi+b}-1}] with respect to w...i will very happy if you clear this. $\endgroup$ Dec 2 '13 at 12:58
  • $\begingroup$ Because first part of first equation after differentiating is a vector w as you have shown but the second part which is SUM(alpha i yi Phi(xi)) dose not seem to be a vector this is my problem ... $\endgroup$ Dec 2 '13 at 13:27

There is a neat trick because:

$\dfrac{\partial}{\partial w}\dfrac{1}{2}\|w\|^2$ $=\dfrac{\partial}{\partial w}\dfrac{1}{2}<w,w>$

We can relabel $w$ as $w_1$ and $w_2$,

$=\dfrac{1}{2}( \dfrac{\partial}{\partial w_1}<w_1,w_2>+\dfrac{\partial}{\partial w_2}<w_1,w_2>)$




This is useful in the case where we want the derivative of a more complicated expression because it allows us to isolate the variable we are taking the derivative with respect to. For example, to minimize


One of the terms we need to take the derivative of is


We can use the fact that $tr(AB)=<A^T,B>$ to say




where the derivative in terms of the lone $W$ is now just $X^TXW$!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.