Understanding Equation (3.13) from Bishop's Pattern Recognition and Machine Learning

Question

I'm having trouble deriving the mentioned equation from the text. Specifically, I'm having trouble obtaining the $\mathbf{\phi}(x_n)^T$ term. I checked the errata and while the lack of the $\beta$ parameter is mentioned there is nothing about this term.

Given that $$ \ln\left(p(\mathbf{t}|\mathbf{w}, \beta)\right) = \frac{N}{2} \ln(\beta) - \frac{N}{2} \ln(2 \pi) - \beta \frac{1}{2} \sum\limits_{i=1}^{N} \left(t_i - \mathbf{w}^T \mathbf{\phi}(\mathbf{x}_i) \right)^2. $$ The author states that $$ \nabla \ln(p(\mathbf{t}|\mathbf{w}, \beta) ) = \beta \sum\limits_{i=1}^{N} \left(t_i - \mathbf{w}^T \mathbf{\phi}(\mathbf{x}_i) \right) \mathbf{\phi}(\mathbf{x}_i)^T, $$ where $\nabla=(\partial w_0, \dots, \partial w_M)^T$, $\mathbf{\phi}(\mathbf{x}_i) = (\phi_0(\mathbf{x}_i), \dots, \phi_M(\mathbf{x}_i))^T$ and $\mathbf{w}=(w_0, \dots, w_M)^T$.

However, when I expand the inner product and take the gradient I obtain $$ \begin{aligned} \nabla \ln(p(\mathbf{t}|\mathbf{w}, \beta) &= -\beta \frac{1}{2} \sum\limits_{i=1}^{N} \nabla \left(t_i - \mathbf{w}^T \mathbf{\phi}(\mathbf{x}_i) \right)^2 \\ &= -\beta \frac{1}{2} \sum\limits_{i=1}^{N} \nabla \left(t_i - w_0 \phi_0 - \dots - w_M \phi_M \right)^2 \\ &= -\beta \sum\limits_{i=1}^{N} \left(t_i - w_0 \phi_0 - \dots - w_M \phi_M \right) \left(-\phi_0, -\phi_1, \dots, -\phi_M \right)^T \\ &= \beta \sum\limits_{i=1}^{N} \left(t_i - \mathbf{w}^T \mathbf{\phi}(\mathbf{x}_i) \right) \mathbf{\phi}(\mathbf{x}_i) \end{aligned} $$ Can somebody please illustrate where the error comes from?

Answer 1

If denominator layout is used while taking derivatives, then $\mathbf \phi(x)$ is correct; if numerator layout is used, $\phi(x)^T$ is correct. And, the author doesn't specify which one is used, but as it appears, it implicitly chooses the numerator layout. So, there is no right or wrong answer. Same follow-up equations can be derived by manipulating both expressions.

For example, the book continues with equating the gradient to zero and expanding the multiplication as follows:

$$0 = \sum\limits_{i=1}^{N} \left(t_i - \mathbf{w}^T \mathbf{\phi}(\mathbf{x}_i) \right) \mathbf{\phi}(\mathbf{x}_i)^T \rightarrow0=\sum_i t_i\phi(\mathbf x_i)^T-\mathbf{w}^T\sum_i (\phi(\mathbf x_i)\phi(\mathbf x_i)^T)$$

For the other version, you can have $$0 = \sum\limits_{i=1}^{N} \left(t_i - \mathbf{w}^T \mathbf{\phi}(\mathbf{x}_i) \right) \mathbf{\phi}(\mathbf{x}_i)=\sum\limits_{i=1}^{N} \mathbf{\phi}(\mathbf{x}_i)\left(t_i - \mathbf{\phi}(\mathbf{x}_i)^T\mathbf{w} \right)\\\rightarrow0=\sum_i t_i\phi(\mathbf x_i)-\left(\sum_i\phi(\mathbf x_i)\phi^T(\mathbf x_i)\right)\mathbf{w}$$

which is just the transpose of the first one.

Answer 2

The error is that the left side of (3.13) should actually be the transpose of the gradient vector which is defined to be a column vector, for the purpose of constructing $\bf{\Phi}^T\bf{\Phi}$.