# Computing $\frac{\partial}{\partial W} || t_n -W^T \phi (x_n) ||^2$

In Ch 3.1.5 of Pattern Recognition and Machine Learning how do we take the derivative wrt $$W$$ of 3.33:

$$ln(p(T|X,W,\beta))=\frac{NK}{2}ln(\frac{\beta}{2\pi}) - \frac{\beta}{2}\sum_{n=1}^N || t_n -W^T \phi (x_n)||^2$$

I attempted to use the chain rule, but I think it might be easier for example to use differentials.

I computed:

$$\frac{\partial}{\partial W}ln(p(T|X,W,\beta))=- \frac{\beta}{2}\sum_{n=1}^N \frac{\partial}{\partial W}(t_n -W^T \phi (x_n))^T(t_n -W^T \phi (x_n))$$

$$=- \frac{\beta}{2}\sum_{n=1}^N \frac{\partial}{\partial (t_n-W^T \phi (x_n))}(t_n -W^T \phi (x_n))^T(t_n -W^T \phi (x_n))\frac{\partial}{\partial W} (t_n-W^T \phi (x_n))$$

$$=- \beta\sum_{n=1}^N (t_n -W^T \phi (x_n))\frac{\partial}{\partial W} (t_n-W^T \phi (x_n))$$

where I used $$\frac{\partial (x^Tx)}{\partial x}=2x$$ but I'm not sure how to compute the derivative:

$$\frac{\partial}{\partial W} (t_n-W^T \phi (x_n))$$

From wikipedia, We have

$$\frac{\partial (Xa+b)^TC(Xa+b)}{\partial X}=(C+C)^T(Xa+b)a^T$$

hence \begin{align}\frac{\partial }{\partial W^T}[(t_n-W^T\phi(x_n))^T(t_n-W^T\phi(x_n))] &=\frac{\partial }{\partial W^T}[(W^T\phi(x_n)-t_n)^T(W^T\phi(x_n)-t_n)]\\ &=2(W^T\phi(x_n)-t_n)\phi(x_n)^T\end{align}

We want

$$\sum_{n=1}^N(W^T\phi(x_n)\phi(x_n)^T-t_n\phi(x_n)^T)=0$$

$$W^T\sum_{n=1}^N\phi(x_n)\phi(x_n)^T=\sum_{n=1}^Nt_n\phi(x_n)^T$$

$$\left(\sum_{n=1}^N\phi(x_n)\phi(x_n)^T\right)W=\sum_{n=1}^N\phi(x_n)t_n^T$$

$$W=\left(\sum_{n=1}^N\phi(x_n)\phi(x_n)^T\right)^{-1}\sum_{n=1}^N\phi(x_n)t_n^T$$

• Thanks, this is really helpful. Jun 5, 2021 at 6:02

$$\def\b{\beta}\def\l{\lambda} \def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)} \def\trace#1{\operatorname{Tr}\LR{#1}} \def\grad#1#2{\frac{\partial #1}{\partial #2}}$$For typing convenience, define the elementwise application of the $$\phi$$ function as the vector $$y=\phi(x) \quad\implies\quad y_n=\phi(x_n)$$ and use a colon as a product notation for the matrix inner product \eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{AB^T} \\ A:A &= \big\|A\big\|^2_F \\ } When applied to vectors this corresponds to the ordinary dot product.

Write the objective function in terms of the above, then calculate its differential and gradient. \eqalign{ \l &= \log(p) \\ &= \l_0 - \tfrac\b 2\LR{y^TW-t^T}:\LR{y^TW-t^T} \\ d\l &= -\b\LR{y^TW-t^T}:\LR{y^TdW} \\ &= -\b\LR{yy^TW-yt^T}:dW \\ \grad{\l}{W} &= -\b\LR{yy^TW-yt^T} \\ } Set the gradient to zero and solve for the optimal $$W$$. \eqalign{ yy^TW &= yt^T \\ W &= (yy^T)^+yt^T = (ty^+)^T \\ } where $$y^+$$ denotes the pseudoinverse of $$y$$.

We can go a bit further. The pseudoinverse of a vector has a closed-form $$y^+ = \frac {y^T}{y^Ty}$$ which can be substituted to obtain $$W = \frac{yt^T}{y^Ty}$$

Note that the properties of the underlying trace function allow the terms in the inner product to be rearranged in many different ways, e.g. \eqalign{ A:B &= B:A \\ A:B &= A^T:B^T \\ C:\LR{AB} &= \LR{A^TC}:B \;=\; \LR{CB^T}:A \\ }