I just don't understand why use matrix transpose, instead of matrix inverse, to calculate delta of weight in gradient descent, like described in http://cs231n.github.io/optimization-2/#mat.

# forward pass
W = np.random.randn(5, 10)
X = np.random.randn(10, 3)
D = W.dot(X)

# now suppose we had the gradient on D from above in the circuit
dD = np.random.randn(*D.shape) # same shape as D
dW = dD.dot(X.T) #.T gives the transpose of the matrix
dX = W.T.dot(dD)

This is my understanding to calculate weight delta: $$ D = WX\\ WXX^-1 = DX^-1\\ W = DX^-1 $$ Could anyone please tell me what wrong with my understanding?


Let $W$ be $i \times k$ and $X$ be $k \times j$. Consider what matrix multiplication is:

$$D_{ij} = \sum_{k=1}^{10}W_{ik} X_{kj}$$

$$\frac{\partial D_{ij}}{\partial W_{ik}} = X_{kj}$$

For a previously described loss function $L$

$$\frac{\partial L_{ij}}{\partial W_{ik}} = \frac{\partial L_{ij}}{\partial D_{ij}} \frac{\partial D_{ij}}{\partial W_{ik}} = \frac{\partial L_{ij}}{\partial D_{ij}} X_{kj} = \frac{\partial L_{ij}}{\partial D_{ij}} X_{jk}^T $$

So we see since $\nabla_W L$ has dimension $i \times k$ (matching $W$) we compute over all $j$ as a matrix multiplication and get $\nabla_W L = (\nabla_D L) X^T$.

It's a good exercise to try matrix multiplication and derivatives out by hand to see why this is true.

  • $\begingroup$ Sorry, but can you explain why $\frac{dD}{dW} = X^T$ ? $\endgroup$ – eric2323223 Mar 19 '18 at 9:01
  • 1
    $\begingroup$ @eric2323223 - following qwr's suggestion about trying out derivatives by hand, try writing out an example with W 2x2 and X 2x1, that might help you see why that's correct. $\endgroup$ – jbowman Mar 19 '18 at 17:28
  • $\begingroup$ @eric2323223 I've added some details. I am still learning notation also but hopefully it's clear what I mean. $\endgroup$ – qwr Mar 19 '18 at 18:52
  • $\begingroup$ I am still editing, please bear with me... $\endgroup$ – qwr Mar 19 '18 at 19:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.