1
$\begingroup$

Consider the spiked covariance model $Y_i\sim^{iid}N(\mu,\Sigma)$, where $Y_1,\ldots,Y_n\in \mathbb{R}^p$, $\Sigma=U\Lambda U^\top+\sigma^2 I_p$ be the eigendecomposition: $U\in\mathbb{R}^{p\times r}$ unknown matrix with orthonormal columns, $\Lambda$ is unknown matrix with nonincreasing diagonal entries, and $\sigma^2>0$ is unknown noise level.

I'm curious about how do we estimate $U$ by using PCA? Is it true that $\hat{U}$, the leading $r$ PC's, is the MLE of the matrix $U$? If so, how to show this? Thank you.

$\endgroup$
7
  • $\begingroup$ Do you know $r$ or do you need to estimate it? $\endgroup$
    – whuber
    Commented Oct 9, 2022 at 13:16
  • 1
    $\begingroup$ since you have a mathy name, I'm just going to point you in the direction of the Eckart–Young–Mirsky theorem math.stackexchange.com/questions/759032/… which is the only "trick" needed to establish this result. $\endgroup$ Commented Oct 9, 2022 at 14:34
  • $\begingroup$ @whuber Yes, $r$ is known. $\endgroup$ Commented Oct 9, 2022 at 15:45
  • 1
    $\begingroup$ Then $U$ is a function of $\Sigma,$ whence any MLE of $\Sigma$ yields an MLE of $U.$ $\endgroup$
    – whuber
    Commented Oct 9, 2022 at 17:23
  • $\begingroup$ @whuber How to show this? $\endgroup$ Commented Oct 9, 2022 at 17:44

1 Answer 1

1
$\begingroup$

I think I figure this out, though it requires several techniques in matrix calculus:

First, we compute the log-likelihood:

$$\begin{aligned}\ell(\mu,\Sigma)&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\sum_{i=1}^n)(Y_i-\mu)^\top\Sigma^{-1}(Y_i-\mu)\\&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\text{tr}\left(\Sigma^{-1}\sum_{i=1}^n(Y_i-\mu)(Y_i-\mu)^\top\right)\\&=-\frac{n}{2}\log\det(\Sigma)-\frac{1}{2}\text{tr}(\Sigma^{-1}n\hat{\Sigma})-\frac{1}{2}\text{tr}\left(\Sigma^{-1}(\bar{Y}-\mu)(\bar{Y}-\mu)^\top\right)\end{aligned},$$

For the third term, it is non-positive, and equals to zero if $\mu^{MLE}=\bar{Y}$. Plug in the MLE estimate,

$$=-\frac{n}{2}\log\det(\Sigma)-\frac{n}{2}\text{tr}(\Sigma^{-1}\hat{\Sigma}),$$

We then calculate $\Sigma^{-1}$: first decompose $U=(U\quad U_\perp)$ and write $\Sigma=(U\quad U_\perp)\begin{pmatrix}\Lambda+\sigma^2I & 0\\0 & \sigma^2I\end{pmatrix}(U\quad U_\perp)^\top$, $\Sigma^{-1}=(U\quad U_\perp)\begin{pmatrix}\frac{1}{\Lambda+\sigma^2I} & 0\\0 & \frac{1}{\sigma^2}I\end{pmatrix}(U\quad U_\perp)^\top=\sigma^2 I-U^\top\left(\sigma^{-2}-(\Lambda+\sigma^I)^{-1}\right)U_\perp$. Based on this, we can compute $\det(\Sigma)$: $\det(\Sigma)=\det\left((U\quad U_\perp)\right)^2\det\begin{pmatrix}\Lambda+\sigma^2I & 0\\0 & \sigma^2I\end{pmatrix}=\prod_{i=1}^r (\lambda_i+\sigma^2)(\sigma^2)^{p-r}$. To find the MLE is to solve the following minimization problem: $$\begin{aligned}\hat{U}^{MLE}&=\text{argmax}_U\left(-\frac{n}{2}\log\det(\Sigma)-\frac{n}{2}\text{tr}(\hat{\Sigma}\Sigma^{-1})\right)\\&=\text{argmin}_U\text{tr}\left\{\left(\Sigma^{-1}\hat{\Sigma}\right)\right\}\\&=\text{argmin}_U\text{tr}\left\{\left(\sigma^{-2}I-U^\top(\sigma^{-2}-(\Lambda+\sigma^2I))^{-1}U\right)\hat{\Sigma}\right\}\\&=\text{argmin}_U\text{tr}\left\{-U^\top\left(\sigma^{-2}I-(\Lambda+\sigma^2I)\right)^{-1}U\hat{\Sigma}\right\}\end{aligned},$$

Take the derivative wrt $U$ to find the minimum of the objection function, with some matrix calculus, we find that $\hat{U}_{MLE}$ is the loading eigenvectors of $\hat{\Sigma}$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.