2
$\begingroup$

In Ch 3.1.5 of Pattern Recognition and Machine Learning how do we take the derivative wrt $W$ of 3.33:

$$ln(p(T|X,W,\beta))=\frac{NK}{2}ln(\frac{\beta}{2\pi}) - \frac{\beta}{2}\sum_{n=1}^N || t_n -W^T \phi (x_n)||^2$$

I attempted to use the chain rule, but I think it might be easier for example to use differentials.

I computed:

$$\frac{\partial}{\partial W}ln(p(T|X,W,\beta))=- \frac{\beta}{2}\sum_{n=1}^N \frac{\partial}{\partial W}(t_n -W^T \phi (x_n))^T(t_n -W^T \phi (x_n)) $$

$$=- \frac{\beta}{2}\sum_{n=1}^N \frac{\partial}{\partial (t_n-W^T \phi (x_n))}(t_n -W^T \phi (x_n))^T(t_n -W^T \phi (x_n))\frac{\partial}{\partial W} (t_n-W^T \phi (x_n)) $$

$$=- \beta\sum_{n=1}^N (t_n -W^T \phi (x_n))\frac{\partial}{\partial W} (t_n-W^T \phi (x_n)) $$

where I used $$\frac{\partial (x^Tx)}{\partial x}=2x$$ but I'm not sure how to compute the derivative:

$$\frac{\partial}{\partial W} (t_n-W^T \phi (x_n))$$

$\endgroup$

2 Answers 2

1
$\begingroup$

From wikipedia, We have

$$\frac{\partial (Xa+b)^TC(Xa+b)}{\partial X}=(C+C)^T(Xa+b)a^T$$

hence \begin{align}\frac{\partial }{\partial W^T}[(t_n-W^T\phi(x_n))^T(t_n-W^T\phi(x_n))] &=\frac{\partial }{\partial W^T}[(W^T\phi(x_n)-t_n)^T(W^T\phi(x_n)-t_n)]\\ &=2(W^T\phi(x_n)-t_n)\phi(x_n)^T\end{align}

We want

$$\sum_{n=1}^N(W^T\phi(x_n)\phi(x_n)^T-t_n\phi(x_n)^T)=0$$

$$W^T\sum_{n=1}^N\phi(x_n)\phi(x_n)^T=\sum_{n=1}^Nt_n\phi(x_n)^T$$

$$\left(\sum_{n=1}^N\phi(x_n)\phi(x_n)^T\right)W=\sum_{n=1}^N\phi(x_n)t_n^T$$

$$W=\left(\sum_{n=1}^N\phi(x_n)\phi(x_n)^T\right)^{-1}\sum_{n=1}^N\phi(x_n)t_n^T$$

$\endgroup$
1
  • $\begingroup$ Thanks, this is really helpful. $\endgroup$ Jun 5, 2021 at 6:02
1
$\begingroup$

$ \def\b{\beta}\def\l{\lambda} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\grad#1#2{\frac{\partial #1}{\partial #2}} $For typing convenience, define the elementwise application of the $\phi$ function as the vector $$y=\phi(x) \quad\implies\quad y_n=\phi(x_n)$$ and use a colon as a product notation for the matrix inner product $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{AB^T} \\ A:A &= \big\|A\big\|^2_F \\ }$$ When applied to vectors this corresponds to the ordinary dot product.

Write the objective function in terms of the above, then calculate its differential and gradient. $$\eqalign{ \l &= \log(p) \\ &= \l_0 - \tfrac\b 2\LR{y^TW-t^T}:\LR{y^TW-t^T} \\ d\l &= -\b\LR{y^TW-t^T}:\LR{y^TdW} \\ &= -\b\LR{yy^TW-yt^T}:dW \\ \grad{\l}{W} &= -\b\LR{yy^TW-yt^T} \\ }$$ Set the gradient to zero and solve for the optimal $W$. $$\eqalign{ yy^TW &= yt^T \\ W &= (yy^T)^+yt^T = (ty^+)^T \\ }$$ where $y^+$ denotes the pseudoinverse of $y$.

We can go a bit further. The pseudoinverse of a vector has a closed-form $$y^+ = \frac {y^T}{y^Ty}$$ which can be substituted to obtain $$W = \frac{yt^T}{y^Ty}$$


Note that the properties of the underlying trace function allow the terms in the inner product to be rearranged in many different ways, e.g. $$\eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ C:\LR{AB} &= \LR{A^TC}:B \;=\; \LR{CB^T}:A \\ }$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.