I am not able to understand how did the elementwise multiplication came into the picture of backpropagation in neural networks I have understood the backpropagation algorithm along with the chain rule well enough that I can derive it on my own, but I don't understand where the elementwise multiplication came from and how does it affect the final result. When I derived it, I didn't consider it as elementwise multiplication, this is where I might be wrong. How do we arrive at elementwise multiplication from matrix multiplication ?

 A: The equation you posted is slightly wrong.
The left hand side of the equation is a scalar, a single element of the matrix $\nabla^{(l)}$. To calculate a single element of that matrix you don't need the whole $\theta^{(l+1)}$, you only need the $i^{th}$ column of $\theta^{(l+1)}$, ie $\theta^{(l+1)}_{:, i}$ and it doesn't use the element-wise multiplication (Hadamard product).
The place where the Hadamard product is used for is in the calculation of $\delta^{(l)}$,
$\nabla^{(l)}=\delta^{(l)}.a^{(l-1)^{T}}$
$\delta^{(l)}=w^{(l+1)^T}.\delta^{(l+1)}\ast\sigma'(z^l)$
The Hadamard product comes into the picture because of the application of chain rule. Backpropagation is derived for each individual element($\frac{\partial C}{\partial w^{(l)}_{ij}}$) in the matrix($\nabla^{(l)}$) and it is then converted to the matrix form for the purpose of vectorization. During this conversion the results of the multiplications resulting from the calculation of partial derivates will be a dot product and the multiplications from the chain rule would be a Hadamard product.
