Consider the probabilistic PCA model where you have $n$ i.i.d centered obserbations $x_1,...,x_n\in \mathbb{R}^p$ drawn from $$\forall i\leq n, \; \; \; \; x_i = W y_i + \varepsilon_i,$$ where $W$ is an unknown deterministic $p \times d$ matrix, $y_i\sim\mathcal{N}(0,1)$ are i.i.d random latent vectors and $\varepsilon_i$ are i.i.d Gaussian noise terms of variance $\sigma^2$.

The principal feature of this model is that the pricipal axes of the data can be found using the maximum-likelihood estimator of $W$ (hence the name probabilistic PCA). For more details about PPCA, the following classical article by Tipping and Bishop is easily accessible online:

How would you define a signal-to-noise ratio for this model ? A natural way would be to divide the "variance" of $W y_i$ by $\sigma^2$ however I encounter two problems:

  • $W$ is unknown so is it okay to replace it by some estimate ?
  • even if $W$ were known, how would you define the "variance" of the vector $W y_i$ ? Simply taking the trace of its covariance matrix (which is $W W^T$) seems a bit arbitrary but could be a simple way to do it.
  • $\begingroup$ Can you explain the context around or motivation behind your question? Why do you need signal-to-noise ratio for this model? What's the purpose? $\endgroup$ – amoeba Dec 10 '15 at 11:19
  • $\begingroup$ I'm comparing model-selection performances (I select $d$ and $p$ according to different criteria) for different values of $\sigma$. Logically, when $W$ is fixed, as $\sigma$ grows, the problem gets harder and the performances get lower. However, the way $W$ is fixed also has a huge impact, so I figured that the good way to "quantify the hardness" of the problem would be to use some kind of SNR using both sigma and $W$. $\endgroup$ – PAM Dec 10 '15 at 21:08
  • $\begingroup$ Not sure that I understood your comment, but anyway, the most straightforward computation would be to say that total variance of $Wy$ is indeed $\operatorname{tr}(WW^\top)=\|W\|^2$ and the total noise variance is $p \sigma^2$, so the overall signal-to-noise ratio is $\|W\|^2/(p \sigma^2)$. $\endgroup$ – amoeba Dec 10 '15 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.