5
$\begingroup$

I want to compute the derivative of:

$\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$,

(Note that C is a covariance matrix that depends on a set of parameters $\theta$)

for which I used the chain rule: $ \frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}= \frac{\partial y^T C^{-1}(\theta)y}{\partial C(\theta) } \frac{\partial C(\theta) }{\partial \theta_{k}}$.

Using eq. 61 from the Matrix Cookbook (http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3274/pdf/imm3274.pdf) I got:

$ \frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}= \left[-C^{-1}(\theta)y y^{T}C^{-1}(\theta) \right]\frac{\partial C(\theta) }{\partial \theta_{k}}$.

However, this results in a matrix times a matrix ans I must obtain a scalar, I cant figure where my derivation is wrong.

|cite|improve this question
$\endgroup$
3
$\begingroup$

For typing convenience, define $$\eqalign{ Y &= yy^T,\,\,\,\, A=C^{-1},\,\,\,\, J = \frac{\partial C}{\partial\theta} \cr \lambda &= y^TC^{-1}y = {\rm Tr}(Y^TA)= Y:A \cr }$$ Notice that $(A,C,Y)$ are symmetric matrices. Also note that the colon in the final expression is just a convenient (Frobenius product) notation for the trace function.

The cyclic properties of the trace allow the terms of a Frobenius product to be rearranged in a variety of ways. For example, all of the following expressions are equivalent $$\eqalign{ A:BC &= BC:A \cr &= A^T:(BC)^T \cr &= B^TA:C \cr &= AC^T:B \cr }$$ To find $\,\frac{\partial\lambda}{\partial\theta}\,$ start by finding its differential $$\eqalign{ d\lambda &= Y:dA \cr &= -Y:A\,dC\,A \cr &= -AYA:dC \cr &= -AYA:J\,d\theta \cr \frac{\partial\lambda}{\partial\theta} &= -AYA:J \cr &= -{\rm Tr}\Big(C^{-1}yy^TC^{-1}\frac{\partial C}{\partial\theta}\Big) \cr\cr }$$ This is consistent with what you found in the Matrix Cookbook, except you should've used the Frobenius product, instead of the regular matrix product, in the chain rule.

For matrix calculus problems, I find it easier to use differentials rather than the chain rule. For many problems, the intermediate quantities required by the chain rule are 3rd and 4th order tensors, which are difficult to comprehend and even harder to calculate.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ +1. The ideas of simplifying the notation and using differentials are especially useful. But I should note once again that to avoid confusion you ought to state that "$:$" has lower precedence than the other operators (matrix multiplication, inversion, and transposition). $\endgroup$ – whuber♦ Oct 3 '18 at 18:35
1
$\begingroup$

I guess the correct chain rule is $$\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_k} = \sum_{i, j} \frac{\partial y^T C^{-1}(\theta)y}{\partial C_{i,j}(\theta)} \frac{\partial C_{i,j}(\theta)}{\partial \theta_k} = Tr\Big[\Big(\frac{\partial y^T C^{-1}(\theta)y}{\partial C(\theta)}\Big)^T \Big(\frac{\partial C(\theta)}{\partial \theta_k}\Big) \Big]$$

where $Tr(A) = \sum_i a_{i,i}, A \in \Re^{n \times n}$ is a trace function.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Could you explain what this right hand side means? In particular, how is one supposed to compute a derivative with respect to $C(\theta)$? Maybe you could supply a simple worked example? $\endgroup$ – whuber♦ Apr 9 '17 at 19:53
  • $\begingroup$ @whuber The derivative w.r.t $C(\theta)$ is already computed (or rather taken from textbook) in question $\endgroup$ – Łukasz Grad Apr 10 '17 at 6:01
  • $\begingroup$ I see that such an expression does appear in the question--but it doesn't make any sense. Perhaps it's a shortcut for something meaningful, but I believe it requires explanation. $\endgroup$ – whuber♦ Apr 10 '17 at 13:46
  • $\begingroup$ @whuber Why it doesn't make sense, because $C$ is symmetric? The derivative is more complex in that case, I guess $\endgroup$ – Łukasz Grad Apr 10 '17 at 14:52
  • $\begingroup$ Let's look at it abstractly, so we can isolate the issue. "$C$" is a function of $\theta$ and "$C^{-1}$" is a function of $C$. Let's call these two functions "$f$" and "$g$", respectively. Your notation amounts to the expression $$\frac{\partial g(f(\theta))}{\partial f(\theta)}.$$ The notation for differentiation has been extremely overloaded at this point, leaving one wondering what it actually means, but perhaps some sense could be made of it by considering $\frac{\partial g(f)}{\partial f}$, for instance. $\endgroup$ – whuber♦ Apr 10 '17 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.