Derivative of a quadratic form wrt a parameter in the matrix

Question

I want to compute the derivative of:

$\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}$,

(Note that C is a covariance matrix that depends on a set of parameters $\theta$)

for which I used the chain rule: $ \frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}= \frac{\partial y^T C^{-1}(\theta)y}{\partial C(\theta) } \frac{\partial C(\theta) }{\partial \theta_{k}}$.

Using eq. 61 from the Matrix Cookbook (http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/3274/pdf/imm3274.pdf) I got:

$ \frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_{k}}= \left[-C^{-1}(\theta)y y^{T}C^{-1}(\theta) \right]\frac{\partial C(\theta) }{\partial \theta_{k}}$.

However, this results in a matrix times a matrix ans I must obtain a scalar, I cant figure where my derivation is wrong.

Answer 1

For typing convenience, define $$\eqalign{ Y &= yy^T,\,\,\,\, A=C^{-1},\,\,\,\, J = \frac{\partial C}{\partial\theta} \cr \lambda &= y^TC^{-1}y = {\rm Tr}(Y^TA)= Y:A \cr }$$ Notice that $(A,C,Y)$ are symmetric matrices. Also note that the colon in the final expression is just a convenient (Frobenius product) notation for the trace function.

The cyclic properties of the trace allow the terms of a Frobenius product to be rearranged in a variety of ways. For example, all of the following expressions are equivalent $$\eqalign{ A:BC &= BC:A \cr &= A^T:(BC)^T \cr &= B^TA:C \cr &= AC^T:B \cr }$$ To find $\,\frac{\partial\lambda}{\partial\theta}\,$ start by finding its differential $$\eqalign{ d\lambda &= Y:dA \cr &= -Y:A\,dC\,A \cr &= -AYA:dC \cr &= -AYA:J\,d\theta \cr \frac{\partial\lambda}{\partial\theta} &= -AYA:J \cr &= -{\rm Tr}\Big(C^{-1}yy^TC^{-1}\frac{\partial C}{\partial\theta}\Big) \cr\cr }$$ This is consistent with what you found in the Matrix Cookbook, except you should've used the Frobenius product, instead of the regular matrix product, in the chain rule.

For matrix calculus problems, I find it easier to use differentials rather than the chain rule. For many problems, the intermediate quantities required by the chain rule are 3rd and 4th order tensors, which are difficult to comprehend and even harder to calculate.

Answer 2

I guess the correct chain rule is $$\frac{\partial y^T C^{-1}(\theta)y}{\partial \theta_k} = \sum_{i, j} \frac{\partial y^T C^{-1}(\theta)y}{\partial C_{i,j}(\theta)} \frac{\partial C_{i,j}(\theta)}{\partial \theta_k} = Tr\Big[\Big(\frac{\partial y^T C^{-1}(\theta)y}{\partial C(\theta)}\Big)^T \Big(\frac{\partial C(\theta)}{\partial \theta_k}\Big) \Big]$$

where $Tr(A) = \sum_i a_{i,i}, A \in \Re^{n \times n}$ is a trace function.