Computing $\frac{\partial}{\partial W} || t_n -W^T \phi (x_n) ||^2$ In Ch 3.1.5 of Pattern Recognition and Machine Learning how do we take the derivative wrt $W$ of 3.33:
$$ln(p(T|X,W,\beta))=\frac{NK}{2}ln(\frac{\beta}{2\pi}) - \frac{\beta}{2}\sum_{n=1}^N || t_n -W^T \phi (x_n)||^2$$
I attempted to use the chain rule, but I think it might be easier for example to use differentials.
I computed:
$$\frac{\partial}{\partial W}ln(p(T|X,W,\beta))=- \frac{\beta}{2}\sum_{n=1}^N \frac{\partial}{\partial W}(t_n -W^T \phi (x_n))^T(t_n -W^T \phi (x_n)) $$
$$=- \frac{\beta}{2}\sum_{n=1}^N \frac{\partial}{\partial (t_n-W^T \phi (x_n))}(t_n -W^T \phi (x_n))^T(t_n -W^T \phi (x_n))\frac{\partial}{\partial W} (t_n-W^T \phi (x_n)) $$
$$=- \beta\sum_{n=1}^N (t_n -W^T \phi (x_n))\frac{\partial}{\partial W} (t_n-W^T \phi (x_n)) $$
where I used $$\frac{\partial (x^Tx)}{\partial x}=2x$$ but I'm not sure how to compute the derivative:
$$\frac{\partial}{\partial W} (t_n-W^T \phi (x_n))$$
 A: From wikipedia, We have
$$\frac{\partial (Xa+b)^TC(Xa+b)}{\partial X}=(C+C)^T(Xa+b)a^T$$
hence
\begin{align}\frac{\partial }{\partial W^T}[(t_n-W^T\phi(x_n))^T(t_n-W^T\phi(x_n))]  &=\frac{\partial }{\partial W^T}[(W^T\phi(x_n)-t_n)^T(W^T\phi(x_n)-t_n)]\\
&=2(W^T\phi(x_n)-t_n)\phi(x_n)^T\end{align}
We want
$$\sum_{n=1}^N(W^T\phi(x_n)\phi(x_n)^T-t_n\phi(x_n)^T)=0$$
$$W^T\sum_{n=1}^N\phi(x_n)\phi(x_n)^T=\sum_{n=1}^Nt_n\phi(x_n)^T$$
$$\left(\sum_{n=1}^N\phi(x_n)\phi(x_n)^T\right)W=\sum_{n=1}^N\phi(x_n)t_n^T$$
$$W=\left(\sum_{n=1}^N\phi(x_n)\phi(x_n)^T\right)^{-1}\sum_{n=1}^N\phi(x_n)t_n^T$$
A: $
\def\b{\beta}\def\l{\lambda}
\def\L{\left}\def\R{\right}\def\LR#1{\L(#1\R)}
\def\trace#1{\operatorname{Tr}\LR{#1}}
\def\grad#1#2{\frac{\partial #1}{\partial #2}}
$For typing convenience, define the elementwise application of the $\phi$ function as the vector
$$y=\phi(x) \quad\implies\quad y_n=\phi(x_n)$$
and use a colon as a product notation for the matrix inner product
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; \trace{AB^T} \\
A:A &= \big\|A\big\|^2_F \\
}$$
When applied to vectors this corresponds to the ordinary dot product.
Write the objective function in terms of the above, then calculate its differential and gradient.
$$\eqalign{
\l &= \log(p) \\
 &= \l_0 - \tfrac\b 2\LR{y^TW-t^T}:\LR{y^TW-t^T} \\
d\l &= -\b\LR{y^TW-t^T}:\LR{y^TdW} \\
  &= -\b\LR{yy^TW-yt^T}:dW \\
\grad{\l}{W} &= -\b\LR{yy^TW-yt^T} \\
}$$
Set the gradient to zero and solve for the optimal $W$.
$$\eqalign{
yy^TW &= yt^T \\
W &= (yy^T)^+yt^T = (ty^+)^T \\
}$$
where $y^+$ denotes the pseudoinverse of $y$.
We can go a bit further. The pseudoinverse of a vector has
a closed-form
$$y^+ = \frac {y^T}{y^Ty}$$
which can be substituted to obtain
$$W = \frac{yt^T}{y^Ty}$$

Note that the properties of the underlying trace function allow the terms in the inner product to be rearranged in many different ways, e.g.
$$\eqalign{
A:B &= B:A \\
A:B &= A^T:B^T \\
C:\LR{AB} &= \LR{A^TC}:B \;=\; \LR{CB^T}:A \\
}$$
