Spiked Covariance Model and PCA Consider the spiked covariance model $Y_i\sim^{iid}N(\mu,\Sigma)$, where $Y_1,\ldots,Y_n\in \mathbb{R}^p$, $\Sigma=U\Lambda U^\top+\sigma^2 I_p$ be the eigendecomposition: $U\in\mathbb{R}^{p\times r}$ unknown matrix with orthonormal columns, $\Lambda$ is unknown matrix with nonincreasing diagonal entries, and $\sigma^2>0$ is unknown noise level.
I'm curious about how do we estimate $U$ by using PCA? Is it true that $\hat{U}$, the leading $r$ PC's, is the MLE of the matrix $U$? If so, how to show this? Thank you.
 A: I think I figure this out, though it requires several techniques in matrix calculus:
First, we compute the log-likelihood:
$$\begin{aligned}\ell(\mu,\Sigma)&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\sum_{i=1}^n)(Y_i-\mu)^\top\Sigma^{-1}(Y_i-\mu)\\&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\text{tr}\left(\Sigma^{-1}\sum_{i=1}^n(Y_i-\mu)(Y_i-\mu)^\top\right)\\&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\text{tr}(\Sigma^{-1}n\hat{\Sigma})-\frac{1}{2}\text{tr}\left(\Sigma^{-1}(\bar{Y}-\mu)(\bar{Y}-\mu)^\top\right)\end{aligned},$$
For the third term, it is non-positive, and equals to zero if $\mu^{MLE}=\bar{Y}$. Plug in the MLE estimate,
$$=-\frac{n}{2}\log\det(\Sigma)-\frac{n}{2}\text{tr}(\Sigma^{-1}\hat{\Sigma}),$$
We then calculate $\Sigma^{-1}$: first decompose $U=(U\quad U_\perp)$ and write $\Sigma=(U\quad U_\perp)\begin{pmatrix}\Lambda+\sigma^2I & 0\\0 & \sigma^2I\end{pmatrix}(U\quad U_\perp)^\top$, $\Sigma^{-1}=(U\quad U_\perp)\begin{pmatrix}\frac{1}{\Lambda+\sigma^2I} & 0\\0 & \frac{1}{\sigma^2}I\end{pmatrix}(U\quad U_\perp)^\top=\sigma^2 I-U^\top\left(\sigma^{-2}-(\Lambda+\sigma^I)^{-1}\right)U_\perp$. Based on this, we can compute $\det(\Sigma)$: $\det(\Sigma)=\det\left((U\quad U_\perp)\right)^2\det\begin{pmatrix}\Lambda+\sigma^2I & 0\\0 & \sigma^2I\end{pmatrix}=\prod_{i=1}^r (\lambda_i+\sigma^2)(\sigma^2)^{p-r}$. To find the MLE is to solve the following minimization problem:
$$\begin{aligned}\hat{U}^{MLE}&=\text{argmax}_U\left(-\frac{n}{2}\log\det(\Sigma)-\frac{n}{2}\text{tr}(\hat{\Sigma}\Sigma^{-1})\right)\\&=\text{argmin}_U\text{tr}\left\{\left(\Sigma^{-1}\hat{\Sigma}\right)\right\}\\&=\text{argmin}_U\text{tr}\left\{\left(\sigma^{-2}I-U^\top(\sigma^{-2}-(\Lambda+\sigma^2I))^{-1}U\right)\hat{\Sigma}\right\}\\&=\text{argmin}_U\text{tr}\left\{-U^\top\left(\sigma^{-2}I-(\Lambda+\sigma^2I)\right)^{-1}U\hat{\Sigma}\right\}\end{aligned},$$
Take the derivative wrt $U$ to find the minimum of the objection function, with some matrix calculus, we find that $\hat{U}_{MLE}$ is the loading eigenvectors of $\hat{\Sigma}$.
