Why use matrix transpose in gradient descent? I just don't understand why use matrix transpose, instead of matrix inverse, to calculate delta of weight in gradient descent, like described in http://cs231n.github.io/optimization-2/#mat.
# forward pass
W = np.random.randn(5, 10)
X = np.random.randn(10, 3)
D = W.dot(X)

# now suppose we had the gradient on D from above in the circuit
dD = np.random.randn(*D.shape) # same shape as D
dW = dD.dot(X.T) #.T gives the transpose of the matrix
dX = W.T.dot(dD)

This is my understanding to calculate weight delta:
$$
D = WX\\
WXX^-1 = DX^-1\\
W = DX^-1
$$
Could anyone please tell me what wrong with my understanding?
 A: Consider what matrix multiplication is, and observe the pattern of indices carefully:
$$D_{ij} = \sum_{k}W_{ik} X_{kj}$$
$$\frac{\partial D_{ij}}{\partial W_{ik}} = X_{kj}$$
For a previously described loss function $L$, by the chain rule,
$$\frac{\partial L}{\partial W_{ik}} = 
\sum_j \frac{\partial L}{\partial D_{ij}} \frac{\partial D_{ij}}{\partial W_{ik}} = 
\sum_j \frac{\partial L}{\partial D_{ij}} X_{kj} = 
\sum_j \frac{\partial L}{\partial D_{ij}} X_{jk}^T
$$
Note $\partial D_{i'j}/\partial W_{ik} = 0$ for $i'\ne i$, so our chain rule sum is over the given $i$ ranging over $j$.
Since we used $X^T$, the inner index $j$ matches up for convenient matrix multiplication notation,
$$
\frac{\partial L}{\partial W} = \frac{\partial L}{\partial D} X^T
$$
This matrix of partial derivatives $\partial L / \partial W$ can also be implemented as the outer product of vectors: $(\partial L / \partial D) \otimes X$.
If you really understand the chain rule and are careful with your indexing, then you should be able to reason through every step of the gradient calculation.
We  need to be careful which matrix calculus layout convention we use: here "denominator layout" is used where $\partial L / \partial W$ has the same shape as $W$ and $\partial L / \partial D$ is a column vector.
A: This thread is a bit old but I think the question is important for DL practitioners, so let me give a more intuitive answer.
If you compare the computational graph for the forward pass with that of the backward pass you'll notice that there are a couple of key differences.
In the backward pas, you have to sum across the different contributions from the different inputs to each node. That is, if a node takes several variables as input, you need the sum of the local gradients across those variables.
So, what in the forward pass is a set of branches feeding into a given node becomes a sum of local gradients in the backward pass.
Branching in forward becomes sum in backward
On the oher hand, the opposite is also true. That is if you have a sum of inputs in the forward pass, you want to get all the partial derivatives with respect to each of those variables in the backward pass, that is
Sum in forward becomes branching in backward
If you just multiply by the inverse you are not perfroming those operations.
On the other hand,multiplying by the transpose does the trick! You can think of the transpose as a kind of "inverse" (in the sense that it transforms outputs back to inputs) but which at the same time turns sums into branchings and branchings into sums. This happens because in multiplying you sum across rows of the matrix but get different results (branch) across columns. Transposing swap both operations.
This is a wonderful video that goes into a detailed explanation about this point and many others, both formally and in an example graph:
https://www.youtube.com/watch?v=kvnBw_D0gfs
