I'm a bit confused with matrix calculus.
Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$
I don't see how this works. My plan was to substitute the $L_2$ norm and derivate that. Witch give just the substitute (thing in brackets) time it's inner derivative. No clue why the inner derivative would be $A^T$.
Physicists like to express matrix and tensor calculations as equivalent sums. Algebra with vectors and matrices is just shorthand for the underlying sums. Doing derivatives on sums is easier and you don't need to remember new rules.
So the first line in this form is $$ f(x) = \frac1 2 \sum_{i} \left(\sum_j A_{ij} x_j - b_i \right)^2 $$
Let's compute the derivative with the notation $\partial_k := d/d x_k$, which corresponds to the $k$-th entry in the Nabla operator. The derivative is commutative with constants and distributive with sums. $$ \partial_k f(x) = \frac 1 2 \sum_{i} \partial_k \left( \sum_j A_{ij} x_j - b_i \right)^2 \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) \partial_k \left(\sum_j A_{ij} x_j - b_i\right) \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) \left(\sum_j A_{ij} \partial_k x_j\right) \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) A_{ik} \\ = \sum_i A_{ki}^T \left(\sum_j A_{ij} x_j - b_i \right) \\ $$ In the second line, I used the chain rule. In the forth line, I used that $\partial_k x_j$ is 1 when $k=j$ and 0 otherwise. Therefore, only one term of the sum remains, $A_{ik}$. Then I reordered terms a bit.
The result is equivalent to your expression with matrices and vectors.
