For back propagation in neural networks , how do we calculate vector by matrix derivative?

Question

I am following the course deep learning ai by Andrew NG. In course1 week4, 04-06-Forward and Backward Propagation, he calculates backward propagation for layer $l$ in neural networks as follows (a single training sample case): $$dz^{[l]}=da^{[l]}*g^{[l]'}(z^{[l]}),(1)$$ $$dw^{[l]}=dz^{[l]}.a^{[l-1]}.(2)$$ The corresponding forward propagation is: $$z^{[l]}=w^{[l]}a^{[l-1]}+b^{[l]},(3)$$ $$a^{[l]}=g^{[l]}(z^{[l]}).(4)$$ I have some problems with Equation $(2)$: I think $$dw^{[l]}=dz^{[l]}.\frac{dw^{[l]}a^{[l-1]}}{dw^{[l]}},(5)$$ where $w^{[l]}$ is of shape $(n^{[l]},n^{[l-1]})$($n^{[l]}$ is the number of neurons in layer $l$) and $a^{[l-1]}$ is of shape $(n^{[l-1]},1)$, so $w^{[l]}a^{[l-1]}$ is $(n^{[l]},1)$, a column vector.

According to vector differentiation by a matrix, vector by matrix derivative is not defined. Then how can Equation $(5)$ be calculated and equal to $a^{[l-1]}$ in Equation (2)?

By the way, I think it should be $(a^{[l-1]})^T$ in Equation $(2)$, otherwise the shapes of $dz^{[l]}$ $(n^{[l]},1)$ $a^{[l-1]}$ $(n^{[l]}-1,1)$do not match at all.

Answer 1

It's true that a vector-matrix derivative isn't really defined. However in backpropagation, you never have to compute the derivative (jacobian) of any node in the computation graph (despite the many tutorials insisting you do). All that is necessary is the ability to compute the "vector jacobian product", or VJP.

The VJP is defined to be the product of the gradient (of the function output) with the jacobian of the function, although it doesn't require you to explicitly write out the jacobian.

It just so happens that the VJP of a function like $f(W,a) = Wa$ with respect to the weights $W$ is $ga^T$, where $g$ is the backpropagated gradient.

I've also explained this in a bit more detail in this other answer here, in case it helps: Higher Order of Vectorization in Backpropagation in Neural Network

And yes, I think you are right about the missing transpose