# Minimizing a least square [duplicate]

I'm a bit confused with matrix calculus.

Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b).$$

I don't see how this works. My plan was to substitute the $L_2$ norm and derivate that. Witch give just the substitute (thing in brackets) time it's inner derivative. No clue why the inner derivative would be $A^T$.

• Commented Aug 2, 2018 at 10:58
• See stats.stackexchange.com/search?q=matrix+derivative for many similar questions.
– whuber
Commented Aug 2, 2018 at 12:03
• thank you! haven't first noticed that those are exactly the same. guess it can be closed. Commented Aug 2, 2018 at 14:07

So the first line in this form is $$f(x) = \frac1 2 \sum_{i} \left(\sum_j A_{ij} x_j - b_i \right)^2$$
Let's compute the derivative with the notation $\partial_k := d/d x_k$, which corresponds to the $k$-th entry in the Nabla operator. The derivative is commutative with constants and distributive with sums. $$\partial_k f(x) = \frac 1 2 \sum_{i} \partial_k \left( \sum_j A_{ij} x_j - b_i \right)^2 \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) \partial_k \left(\sum_j A_{ij} x_j - b_i\right) \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) \left(\sum_j A_{ij} \partial_k x_j\right) \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) A_{ik} \\ = \sum_i A_{ki}^T \left(\sum_j A_{ij} x_j - b_i \right) \\$$ In the second line, I used the chain rule. In the forth line, I used that $\partial_k x_j$ is 1 when $k=j$ and 0 otherwise. Therefore, only one term of the sum remains, $A_{ik}$. Then I reordered terms a bit.