# how to derive eq 21.2.3 in BRML, Factor Analysis (Eigen-approach likelihood optimization)

how to derive eq 21.2.3 in BRML, chapter21 Factor Analysis?

Log Likelihood function (eq. 21.1.13):

$$\log{p(\mathcal{V} | \mathbf{F}, \mathbf{\Psi})} = -\frac{N}{2}\left( \mathrm{trace}(\mathbf{\Sigma}_{D}^{-1}\mathbf{S}) + \log{\mathrm{det}(2\pi \mathbf{\Sigma}_{D})} \right)\\ \text{where } \mathbf{\Sigma}_D = \mathbf{F}\mathbf{F}^{T} + \mathbf{\Psi}_{diag} , \mathbf{S} = \frac{1}{N} \sum_{i=1}^{N}(\vec{v}_{i} - \bar{\vec{v}})(\vec{v}_{i} - \bar{\vec{v}})^{T}$$

eq. 21.2.1 (derivative the log likelihood func with respect to $$\mathbf{F}$$ and equate to zero, I solved)：

$$0 = \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1}(\partial_{\mathbf{F}}\mathbf{\Sigma}_{D} )\mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \partial_{\mathbf{F}}\mathbf{\Sigma}_{D} \right)\\$$

eq. 21.2.2 ?:

$$\partial_{\mathbf{F}} (\mathbf{\Sigma}_{D}) = \partial_{\mathbf{F}} (\mathbf{F}\mathbf{F}^{T}) = \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T}$$

(but, this eq can be derivatived a matrix by matrix ?)

eq. 21.2.3 ?:

$$\mathbf{\Sigma}_D^{-1} \mathbf{F} = \mathbf{\Sigma}_D^{-1} \mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F}$$

I self-solved.

it substitutes eq.21.2.2 for eq.21.2.1 .

$$\mathrm{trace}\left( \mathbf{\Sigma}_D^{-1}(\mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} )\mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} (\mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T}) \right) = 0\\ \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T})\mathbf{\Sigma}_D^{-1}\mathbf{S} + \mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F}) \mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T}) + \mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} \right) = 0\\$$

Using $$\mathrm{trace}( \mathbf{A}+\mathbf{B}) = \mathrm{trace} (\mathbf{A})+\mathrm{trace}(\mathbf{B})$$,

$$\mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T})\mathbf{\Sigma}_D^{-1}\mathbf{S}\right) + \mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F}) \mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S} \right) - \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} (\partial_{\mathbf{F}} \mathbf{F}^{T})\right) - \mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F})\mathbf{F}^{T} \right) = 0\\$$

reparameterize matrices without effect to derivatives:

$$\mathbf{A} \equiv \mathbf{\Sigma}_D^{-1} \mathbf{F}, \mathbf{B} \equiv \mathbf{\Sigma}_D^{-1}\mathbf{S}, \mathbf{C} \equiv \mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S}, \mathbf{D} \equiv \mathbf{F}^{T}$$

$$\mathrm{trace}\left( \mathbf{A} (\partial_{\mathbf{F}} \mathbf{F}^{T})\mathbf{B}\right) + \mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F}) \mathbf{C} \right) - \mathrm{trace}\left( \mathbf{A} (\partial_{\mathbf{F}} \mathbf{F}^{T})\right) - \mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} (\partial_{\mathbf{F}} \mathbf{F})\mathbf{D} \right) = 0\\$$

Using $$\partial \mathrm{trace}(\mathbf{X}) = \mathrm{trace}(\partial \mathbf{X})$$, transpose a derivative operator outside trace.

$$\partial_{\mathbf{F}}\mathrm{trace}\left( \mathbf{A} \mathbf{F}^{T}\mathbf{B}\right) + \partial_{\mathbf{F}}\mathrm{trace}\left( \mathbf{\Sigma}_D^{-1} \mathbf{F} \mathbf{C} \right) - \partial_{\mathbf{F}}\mathrm{trace}\left( \mathbf{A} \mathbf{F}^{T}\right) - \partial_{\mathbf{F}}\mathrm{trace}\left(\mathbf{\Sigma}_D^{-1} \mathbf{F}\mathbf{D} \right) = \mathbf{0}\\$$

Using the following derivatives of trace:

$$\partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{A} \mathbf{X}^{T}\mathbf{B}\right)=\mathbf{B}\mathbf{A} , \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{A} \mathbf{X}\mathbf{B}\right)=\mathbf{A}^{T}\mathbf{B}^{T} , \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{X}\mathbf{A} \right)=\mathbf{A}^{T} , \partial_{\mathbf{X}}\mathrm{trace}\left( \mathbf{A}\mathbf{X}^{T} \right)=\mathbf{A}$$

So,

$$\begin{eqnarray} \mathbf{B}\mathbf{A} + (\mathbf{\Sigma}_D^{-1})^{T} \mathbf{C}^{T} - \mathbf{A} - (\mathbf{\Sigma}_D^{-1})^{T} \mathbf{D}^{T} &=& \mathbf{0} \\ \mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F} + \mathbf{\Sigma}_D^{-1} (\mathbf{F}^{T} \mathbf{\Sigma}_D^{-1}\mathbf{S})^{T} - \mathbf{\Sigma}_D^{-1} \mathbf{F} - \mathbf{\Sigma}_D^{-1}(\mathbf{F}^{T})^{T} &=& \mathbf{0}\\ \mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F} + \mathbf{\Sigma}_D^{-1} \mathbf{S}\mathbf{\Sigma}_D^{-1}\mathbf{F} - \mathbf{\Sigma}_D^{-1} \mathbf{F} - \mathbf{\Sigma}_D^{-1} \mathbf{F} &=& \mathbf{0}\\ 2\mathbf{\Sigma}_D^{-1} \mathbf{F} &=& 2\mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F}\\ \mathbf{\Sigma}_D^{-1} \mathbf{F} &=& \mathbf{\Sigma}_D^{-1}\mathbf{S}\mathbf{\Sigma}_D^{-1} \mathbf{F}\\ \end{eqnarray}$$

it can derive eq.21.2.3 .

• Kudos for answering your question and sharing it! (+1) Commented Jul 30, 2019 at 19:03