# Spiked Covariance Model and PCA

Consider the spiked covariance model $$Y_i\sim^{iid}N(\mu,\Sigma)$$, where $$Y_1,\ldots,Y_n\in \mathbb{R}^p$$, $$\Sigma=U\Lambda U^\top+\sigma^2 I_p$$ be the eigendecomposition: $$U\in\mathbb{R}^{p\times r}$$ unknown matrix with orthonormal columns, $$\Lambda$$ is unknown matrix with nonincreasing diagonal entries, and $$\sigma^2>0$$ is unknown noise level.

I'm curious about how do we estimate $$U$$ by using PCA? Is it true that $$\hat{U}$$, the leading $$r$$ PC's, is the MLE of the matrix $$U$$? If so, how to show this? Thank you.

• Do you know $r$ or do you need to estimate it?
– whuber
Commented Oct 9, 2022 at 13:16
• since you have a mathy name, I'm just going to point you in the direction of the Eckart–Young–Mirsky theorem math.stackexchange.com/questions/759032/… which is the only "trick" needed to establish this result. Commented Oct 9, 2022 at 14:34
• @whuber Yes, $r$ is known. Commented Oct 9, 2022 at 15:45
• Then $U$ is a function of $\Sigma,$ whence any MLE of $\Sigma$ yields an MLE of $U.$
– whuber
Commented Oct 9, 2022 at 17:23
• @whuber How to show this? Commented Oct 9, 2022 at 17:44

I think I figure this out, though it requires several techniques in matrix calculus:

First, we compute the log-likelihood:

\begin{aligned}\ell(\mu,\Sigma)&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\sum_{i=1}^n)(Y_i-\mu)^\top\Sigma^{-1}(Y_i-\mu)\\&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\text{tr}\left(\Sigma^{-1}\sum_{i=1}^n(Y_i-\mu)(Y_i-\mu)^\top\right)\\&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\text{tr}(\Sigma^{-1}n\hat{\Sigma})-\frac{1}{2}\text{tr}\left(\Sigma^{-1}(\bar{Y}-\mu)(\bar{Y}-\mu)^\top\right)\end{aligned},

For the third term, it is non-positive, and equals to zero if $$\mu^{MLE}=\bar{Y}$$. Plug in the MLE estimate,

$$=-\frac{n}{2}\log\det(\Sigma)-\frac{n}{2}\text{tr}(\Sigma^{-1}\hat{\Sigma}),$$

We then calculate $$\Sigma^{-1}$$: first decompose $$U=(U\quad U_\perp)$$ and write $$\Sigma=(U\quad U_\perp)\begin{pmatrix}\Lambda+\sigma^2I & 0\\0 & \sigma^2I\end{pmatrix}(U\quad U_\perp)^\top$$, $$\Sigma^{-1}=(U\quad U_\perp)\begin{pmatrix}\frac{1}{\Lambda+\sigma^2I} & 0\\0 & \frac{1}{\sigma^2}I\end{pmatrix}(U\quad U_\perp)^\top=\sigma^2 I-U^\top\left(\sigma^{-2}-(\Lambda+\sigma^I)^{-1}\right)U_\perp$$. Based on this, we can compute $$\det(\Sigma)$$: $$\det(\Sigma)=\det\left((U\quad U_\perp)\right)^2\det\begin{pmatrix}\Lambda+\sigma^2I & 0\\0 & \sigma^2I\end{pmatrix}=\prod_{i=1}^r (\lambda_i+\sigma^2)(\sigma^2)^{p-r}$$. To find the MLE is to solve the following minimization problem: \begin{aligned}\hat{U}^{MLE}&=\text{argmax}_U\left(-\frac{n}{2}\log\det(\Sigma)-\frac{n}{2}\text{tr}(\hat{\Sigma}\Sigma^{-1})\right)\\&=\text{argmin}_U\text{tr}\left\{\left(\Sigma^{-1}\hat{\Sigma}\right)\right\}\\&=\text{argmin}_U\text{tr}\left\{\left(\sigma^{-2}I-U^\top(\sigma^{-2}-(\Lambda+\sigma^2I))^{-1}U\right)\hat{\Sigma}\right\}\\&=\text{argmin}_U\text{tr}\left\{-U^\top\left(\sigma^{-2}I-(\Lambda+\sigma^2I)\right)^{-1}U\hat{\Sigma}\right\}\end{aligned},

Take the derivative wrt $$U$$ to find the minimum of the objection function, with some matrix calculus, we find that $$\hat{U}_{MLE}$$ is the loading eigenvectors of $$\hat{\Sigma}$$.