Backpropagation - computing partial derivative with respect to W I am following a chapter on backprop derivation from the online book by Michael Nielsen
In particular, following equation is derived in Chapter 2:
${∂C\over∂w^{l}_{jk}}=a^{l−1}_{k}δ^{l}_{j}$
Now, I am trying to implement this in a vector form. I assume that proper operation is the outer product of vectors:
${{∂C\over∂w^{l}} =a^{l-1} \odot   (δ^{l})^T}$ (Correct me if I am wrong here)
Still, I often see on many sources that this equation is written as:
${{∂C\over∂w^{l}} =a^{l-1}δ^{l}}$
Am I missing something or this is just a different style of notation?
 A: This is a late response but I hope it may help:
Important clarifications
First, we have to realize that the multiplication $a^{l-1}\delta^l$ as it appears in the question can't happen. This is because:

*

*$a^{l-1}$ is a $K\times 1$ vector that contains the activations of the layer $l-1$.

*$\delta^l\,\,\,\,\,\,$ is a $J\times 1$ vector that contains the "error" terms, $\partial C/\partial z^l$ of each neuron of layer $l$.

Where $K$ and $J$ are the total number of neurons in layer $l-1$ and $l$ respectively.
So the correct equation that we have to proof to be true is instead:
$$ \frac{\partial C}{\partial w^l}=\delta^l(a^{l-1})^T$$
With this said, we can also realize that the equation proposed in the question:
$$\frac{\partial C}{\partial w^l}=a^{l-1}\odot (\delta^l)^T$$
can't be true, because the outer product (or Hadamard product as Michael Nielsen refers to it in the book): $\odot$, represents an element-wise multiplication between the terms $a^{l-1}$ and $(\delta^l)^T$. So unless these terms were both scalars ($J=K=1$) we would be unable to apply this element-wise multiplication.

Reasoning using dimensionality of the weight matrix
Now the question that remains is: Why is $\delta^l(a^{l-1})^T$ the correct equation?. This is because all the weights of each layer $l$ are stored in matrix. This matrix is given by:
$$ w^l=W^l=
\begin{pmatrix}
    w_{11}^l & w_{12}^l & \cdots & w_{1K}^l\\
    w_{21}^l & w_{22}^l & \cdots & w_{2K}^l\\
    \vdots   & \vdots   & \ddots & \vdots\\
    w_{J1}^l & w_{J2}^l & \cdots & w_{JK}^l\\
  \end{pmatrix}
$$
Where each entry $w^l_{jk}$ corresponds to the weight that connects the neuron $k$ of layer $l-1$ with the neuron $j$ of layer $l$.
So now we can intuitively see that $\partial C/\partial W^l$ should be of size $J\times K$ (size of $W^l$). Something that can be accomplished using $\partial C/\partial W^l= \delta^l(a^{l-1})^T$.
With this, if we wanted, we would be able to make the correct updates ($\Delta W^l$) on the weights using simple gradient descent (and other types of optimization, of course):
$$ W^l_{n+1} = \underbrace{W^l_n}_{J\times K} - \alpha \underbrace{\Delta W^l}_{J\times K} = W^l_n - \alpha \underbrace{\left( \delta^l(a^{l-1})^T \right)}_{J\times K}$$
Where $\alpha$ is the learning rate.

A more correct reasoning (using the previous one)
To answer this more properly, we can realize that each element at location $(j,k)$ of the matrix $\Delta W^l$ is given by:
$$\Delta W_{(j,k)}^l=\frac{\partial C}{\partial w^l_{jk}}=\delta^l_j a^{l-1}_k$$
So we can write $\Delta W^l$ as:
$$
\Delta W^l=
\begin{pmatrix}
    \delta^l_1 a^{l-1}_1 & \delta^l_1 a^{l-1}_2 & \cdots & \delta^l_1 a^{l-1}_K\\
    \delta^l_2 a^{l-1}_1 & \delta^l_2 a^{l-1}_2 & \cdots & \delta^l_2 a^{l-1}_K\\
    \vdots   & \vdots   & \ddots & \vdots\\
    \delta^l_J a^{l-1}_1 & \delta^l_J a^{l-1}_2 & \cdots & \delta^l_J a^{l-1}_K\\
  \end{pmatrix}
$$

Which is the same as computing: $\delta^l(a^{l-1})^T$
