I was learning deep learning from Andrew Ng's course on Coursera and in one of the programming assignments to code out Neural Networks from scratch, the formula for the derivative of cost function wrt W is given as:

$$ \frac {dJ} {dW_{i}} = \frac {1} {m} \frac {dJ} {dZ_{i}} A_{i - 1}^T $$

$J$ is the cost function
$m$ is the number of training examples,
$W_{i}$ are the parameters from layer $i - 1$ to layer $i$
$Z_i = W_i^TA_{i - 1} + B_i$.
$A_0$ is $X^T$, i.e., $ dim(A_{0}) = n * m$ and $A_i = ActivationFn(Z_i)$.

However, this seems inconsistent dimensionally, assuming the product of $ \frac {dJ} {dZ_{i}}$ and $A_{i - 1}^T $ is dot product. If we denote the number of nodes in layer $i$ as $n_i$, where layer $0$ is the input $X$ and $n_0 = number of features(n)$,

$$ dim(A_{i - 1}) = (n_{i - 1}, m) $$ $$ dim(Z_i) = (n_{i}, m) $$ $$ dim(\frac{dJ}{dZ_i}) = dim(Z) $$ $$ dim(\frac {dJ} {dZ_{i}} A_{i - 1}^T ) = (n_{i}, n_{i - 1}) $$ $$ dim(\frac {dJ} {dW_{i}}) = (n_{i - 1}, n_i) $$

$$ dim(W_i) = (n_{i - 1}, n_i) $$ I do not have much knowledge of vector calculus but when I tried t derive these equations myself I got stuck on what kind of product (dot or Hadamard product or something else) will be there between the derivatives on applying chain rule. Can someone please help me figure out where have I gone wrong?

Here is an image of the formulae listed there. Note, the author means $\frac{dJ}{dx}$ when he write $dx$ enter image description here


The dimensions in the formula were indeed inconsistent, I could fix it by adding a transpose to the formula. $$ \frac {dJ} {dW_{i}} = \frac {1} {m} (\frac {dJ} {dZ_{i}} A_{i - 1}^T)^T $$

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.