# Why use matrix transpose in gradient descent?

Sorry for this silly question, but I just don't understand why use matrix transpose, instead of matrix inverse, to calculate delta of weight in gradient descent, like described in http://cs231n.github.io/optimization-2/#mat.

# forward pass
W = np.random.randn(5, 10)
X = np.random.randn(10, 3)
D = W.dot(X)

# now suppose we had the gradient on D from above in the circuit
dD = np.random.randn(*D.shape) # same shape as D
dW = dD.dot(X.T) #.T gives the transpose of the matrix
dX = W.T.dot(dD)


This is my understanding to calculate weight delta: $$D = WX\\ WXX^-1 = DX^-1\\ W = DX^-1$$ Could anyone please tell me what wrong with my understanding?

## 1 Answer

Let $W$ be $i \times k$ and $X$ be $k \times j$. Consider what matrix multiplication is:

$$D_{ij} = \sum_{k=1}^{10}W_{ik} X_{kj}$$

$$\frac{\partial D_{ij}}{\partial W_{ik}} = X_{kj}$$

For a previously described loss function $L$

$$\frac{\partial L_{ij}}{\partial W_{ik}} = \frac{\partial L_{ij}}{\partial D_{ij}} \frac{\partial D_{ij}}{\partial W_{ik}} = \frac{\partial L_{ij}}{\partial D_{ij}} X_{kj} = \frac{\partial L_{ij}}{\partial D_{ij}} X_{jk}^T$$

So we see since $\nabla_W L$ has dimension $i \times k$ (matching $W$) we compute over all $j$ as a matrix multiplication and get $\nabla_W L = (\nabla_D L) X^T$.

It's a good exercise to try matrix multiplication and derivatives out by hand to see why this is true.

• Sorry, but can you explain why $\frac{dD}{dW} = X^T$ ? – eric2323223 Mar 19 '18 at 9:01
• @eric2323223 - following qwr's suggestion about trying out derivatives by hand, try writing out an example with W 2x2 and X 2x1, that might help you see why that's correct. – jbowman Mar 19 '18 at 17:28
• @eric2323223 I've added some details. I am still learning notation also but hopefully it's clear what I mean. – qwr Mar 19 '18 at 18:52
• I am still editing, please bear with me... – qwr Mar 19 '18 at 19:08