I just don't understand why use matrix transpose, instead of matrix inverse, to calculate delta of weight in gradient descent, like described in http://cs231n.github.io/optimization-2/#mat.

# forward pass
W = np.random.randn(5, 10)
X = np.random.randn(10, 3)
D = W.dot(X)

# now suppose we had the gradient on D from above in the circuit
dD = np.random.randn(*D.shape) # same shape as D
dW = dD.dot(X.T) #.T gives the transpose of the matrix
dX = W.T.dot(dD)

This is my understanding to calculate weight delta: $$ D = WX\\ WXX^-1 = DX^-1\\ W = DX^-1 $$ Could anyone please tell me what wrong with my understanding?

  • 1
    $\begingroup$ You can't take the inverse since most neural network matrices are not guaranteed to have an inverse. To certainly find one the matrix needs to be at least square. Which it usually isn't. Hence you need to do the calculus element-wise. $\endgroup$ – not_a_bot_no_really_82353 Jan 28 at 18:55

Let $W$ be $i \times k$ and $X$ be $k \times j$. Consider what matrix multiplication is:

$$D_{ij} = \sum_{k=1}^{10}W_{ik} X_{kj}$$

$$\frac{\partial D_{ij}}{\partial W_{ik}} = X_{kj}$$

For a previously described loss function $L$

$$\frac{\partial L_{ij}}{\partial W_{ik}} = \frac{\partial L_{ij}}{\partial D_{ij}} \frac{\partial D_{ij}}{\partial W_{ik}} = \frac{\partial L_{ij}}{\partial D_{ij}} X_{kj} = \frac{\partial L_{ij}}{\partial D_{ij}} X_{jk}^T $$

So we see since $\nabla_W L$ has dimension $i \times k$ (matching $W$) we compute over all $j$ as a matrix multiplication and get $\nabla_W L = (\nabla_D L) X^T$.

It's a good exercise to try matrix multiplication and derivatives out by hand to see why this is true.

  • 1
    $\begingroup$ Sorry, but can you explain why $\frac{dD}{dW} = X^T$ ? $\endgroup$ – eric2323223 Mar 19 '18 at 9:01
  • 1
    $\begingroup$ @eric2323223 - following qwr's suggestion about trying out derivatives by hand, try writing out an example with W 2x2 and X 2x1, that might help you see why that's correct. $\endgroup$ – jbowman Mar 19 '18 at 17:28
  • $\begingroup$ @eric2323223 I've added some details. I am still learning notation also but hopefully it's clear what I mean. $\endgroup$ – qwr Mar 19 '18 at 18:52
  • $\begingroup$ I am still editing, please bear with me... $\endgroup$ – qwr Mar 19 '18 at 19:08

This thread is a bit old but I think the question is important for DL practitioners, so let me give a more intuitive answer. If you compare the computational graph for the forward pass with that of the backward pass you'll notice that there are a couple of key differences. In the backward pas, you have to sum across the different contributions from the different inputs to each node. That is, if a node takes several variables as input, you need the sum of the local gradients across those variables. So, what in the forward pass is a set of branches feeding into a given node becomes a sum of local gradients in the backward pass. Branching in forward becomes sum in backward On the oher hand, the opposite is also true. That is if you have a sum of inputs in the forward pass, you want to get all the partial derivatives with respect to each of those variables in the backward pass, that is Sum in forward becomes branching in backward If you just multiply by the inverse you are not perfroming those operations. On the other hand,multiplying by the transpose does the trick! You can think of the transpose as a kind of "inverse" (in the sense that it transforms outputs back to inputs) but which at the same time turns sums into branchings and branchings into sums. This happens because in multiplying you sum across rows of the matrix but get different results (branch) across columns. Transposing swap both operations. This is a wonderful video that goes into a detailed explanation about this point and many others, both formally and in an example graph: https://www.youtube.com/watch?v=kvnBw_D0gfs


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.