When calculating gradients for backpropagation, a surprising result is that the gradient of the weights in a ReLU-activated layer ends up being a diagonal matrix. I am seeking to understand how this makes sense intuitively, in terms of gradient descent.
Does this mean that weights in the non-diagonal positions don't have an impact on the loss? How could that be? Do these weights never get updated at all?