how to derive eq 21.2.3 in BRML, Factor Analysis (Eigen-approach likelihood optimization)

Question

how to derive eq 21.2.3 in BRML, chapter21 Factor Analysis?

Log Likelihood function (eq. 21.1.13):

$$ \log{p(\mathcal{V} | \mathbf{F}, \mathbf{\Psi})} = -\frac{N}{2}\left( \mathrm{trace}(\mathbf{\Sigma}_{D}^{-1}\mathbf{S}) + \log{\mathrm{det}(2\pi \mathbf{\Sigma}_{D})} \right)\\ \text{where } \mathbf{\Sigma}_D = \mathbf{F}\mathbf{F}^{T} + \mathbf{\Psi}_{diag} , \mathbf{S} = \frac{1}{N} \sum_{i=1}^{N}(\vec{v}_{i} - \bar{\vec{v}})(\vec{v}_{i} - \bar{\vec{v}})^{T} $$

eq. 21.2.1 (derivative the log likelihood func with respect to $\mathbf{F}$ and equate to zero, I solved)：

$$ 0 = \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1}(\partial_{\mathbf{F}}\mathbf{\Sigma}_{D} )\mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \partial_{\mathbf{F}}\mathbf{\Sigma}_{D} \right)\\ $$

eq. 21.2.2 ?:

$$ \partial_{\mathbf{F}} (\mathbf{\Sigma}_{D}) = \partial_{\mathbf{F}} (\mathbf{F}\mathbf{F}^{T}) = \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} $$

(but, this eq can be derivatived a matrix by matrix ?)

eq. 21.2.3 ?:

$$ \mathbf{\Sigma}_D^{-1} \mathbf{F} = \mathbf{\Sigma}_D^{-1} \mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F} $$

please tell me this derivation.

Answer 1

I self-solved.

it substitutes eq.21.2.2 for eq.21.2.1 .

$$ \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1}(\mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} )\mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} (\mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T}) \right) = 0\\ \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T})\mathbf{\Sigma}_D^{-1}\mathbf{S} + \mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F}) \mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + \mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} \right) = 0\\ $$

Using $\mathrm{trace}( \mathbf{A}+\mathbf{B}) = \mathrm{trace} (\mathbf{A})+\mathrm{trace}(\mathbf{B})$,

$$ \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T})\mathbf{\Sigma}_D^{-1}\mathbf{S}\right) + \mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F}) \mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T})\right) - \mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} \right) = 0\\ $$

reparameterize matrices without effect to derivatives:

$$ \mathbf{A} \equiv \mathbf{\Sigma}_D^{-1} \mathbf{F}, \mathbf{B} \equiv \mathbf{\Sigma}_D^{-1}\mathbf{S}, \mathbf{C} \equiv \mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S}, \mathbf{D} \equiv \mathbf{F}^{T} $$

$$ \mathrm{trace}\left( \mathbf{A} (\partial_{\mathbf{F}} \mathbf{F}^{T})\mathbf{B}\right) + \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F}) \mathbf{C} \right) - \mathrm{trace}\left( \mathbf{A} (\partial_{\mathbf{F}} \mathbf{F}^{T})\right) - \mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F})\mathbf{D} \right) = 0\\ $$

Using $\partial \mathrm{trace}(\mathbf{X}) = \mathrm{trace}(\partial \mathbf{X})$, transpose a derivative operator outside trace.

$$ \partial_{\mathbf{F}}\mathrm{trace}\left( \mathbf{A} \mathbf{F}^{T}\mathbf{B}\right) + \partial_{\mathbf{F}}\mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} \mathbf{C} \right) - \partial_{\mathbf{F}}\mathrm{trace}\left( \mathbf{A} \mathbf{F}^{T}\right) - \partial_{\mathbf{F}}\mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} \mathbf{F}\mathbf{D} \right) = \mathbf{0}\\ $$

Using the following derivatives of trace:

$$ \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{A} \mathbf{X}^{T}\mathbf{B}\right)=\mathbf{B}\mathbf{A} , \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{A} \mathbf{X}\mathbf{B}\right)=\mathbf{A}^{T}\mathbf{B}^{T} , \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{X}\mathbf{A} \right)=\mathbf{A}^{T} , \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{A}\mathbf{X}^{T} \right)=\mathbf{A} $$

So,

\begin{eqnarray} \mathbf{B}\mathbf{A} + (\mathbf{\Sigma}_D^{-1})^{T} \mathbf{C}^{T} - \mathbf{A} - (\mathbf{\Sigma}_D^{-1})^{T} \mathbf{D}^{T} &=& \mathbf{0} \\ \mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F} + \mathbf{\Sigma}_D^{-1} (\mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S})^{T} - \mathbf{\Sigma}_D^{-1} \mathbf{F} - \mathbf{\Sigma}_D^{-1}(\mathbf{F}^{T})^{T} &=& \mathbf{0}\\ \mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F} + \mathbf{\Sigma}_D^{-1} \mathbf{S}\mathbf{\Sigma}_D^{-1}\mathbf{F} - \mathbf{\Sigma}_D^{-1} \mathbf{F} - \mathbf{\Sigma}_D^{-1} \mathbf{F} &=& \mathbf{0}\\ 2\mathbf{\Sigma}_D^{-1} \mathbf{F} &=& 2\mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F}\\ \mathbf{\Sigma}_D^{-1} \mathbf{F} &=& \mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F}\\ \end{eqnarray}

it can derive eq.21.2.3 .