2
$\begingroup$

I'm a bit confused with matrix calculus.

Given is $$f(x) = \frac{1}{2}||Ax-b||^2_2$$ and the derivate of it is in my book $$\nabla_xf(x) = A^T(Ax-b). $$

I don't see how this works. My plan was to substitute the $L_2$ norm and derivate that. Witch give just the substitute (thing in brackets) time it's inner derivative. No clue why the inner derivative would be $A^T$.

$\endgroup$
3

1 Answer 1

3
$\begingroup$

Physicists like to express matrix and tensor calculations as equivalent sums. Algebra with vectors and matrices is just shorthand for the underlying sums. Doing derivatives on sums is easier and you don't need to remember new rules.

So the first line in this form is $$ f(x) = \frac1 2 \sum_{i} \left(\sum_j A_{ij} x_j - b_i \right)^2 $$

Let's compute the derivative with the notation $\partial_k := d/d x_k$, which corresponds to the $k$-th entry in the Nabla operator. The derivative is commutative with constants and distributive with sums. $$ \partial_k f(x) = \frac 1 2 \sum_{i} \partial_k \left( \sum_j A_{ij} x_j - b_i \right)^2 \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) \partial_k \left(\sum_j A_{ij} x_j - b_i\right) \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) \left(\sum_j A_{ij} \partial_k x_j\right) \\ = \sum_i \left(\sum_j A_{ij} x_j - b_i \right) A_{ik} \\ = \sum_i A_{ki}^T \left(\sum_j A_{ij} x_j - b_i \right) \\ $$ In the second line, I used the chain rule. In the forth line, I used that $\partial_k x_j$ is 1 when $k=j$ and 0 otherwise. Therefore, only one term of the sum remains, $A_{ik}$. Then I reordered terms a bit.

The result is equivalent to your expression with matrices and vectors.

$\endgroup$
2
  • $\begingroup$ Thank you for the explanation. You mentioned new rules one could learn. Could you explain me this one, or can you tell me what kind of rules I have to search for? The book is full of such calculations. Your way works great, but it takes some time. Maybe there is a shortcut... $\endgroup$
    – Mr.Sh4nnon
    Aug 2, 2018 at 10:40
  • 1
    $\begingroup$ True, this approach is slow, that's why the shorthand notation is useful. You could start with en.wikipedia.org/wiki/Vector_calculus and en.wikipedia.org/wiki/Gradient, but you should get a formal text book about vector algebra and differential operators. $\endgroup$
    – olq_plo
    Aug 2, 2018 at 11:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.