# How to impose restrictions on a random matrix via its prior distribution?

I am reading the paper Factor analysis and outliers: A Bayesian approach. The author starts with a factor analysis model given by $${\bf y}_i = {\bf \Lambda} {\bf z}_i + {\bf e}_i, \quad i = 1, \ldots, n,$$ where each $${\bf y}_i$$ is a $$p$$-dimensional observation vector, each $${\bf z}_i$$ is a $$K$$-dimensional latent factor vector, and $${\bf \Lambda}$$ is a $$p \times K$$ full-rank matrix of factor loadings. The author assumes that the factors and the error term are Normal: $${\bf z}_i \sim \mathcal{N} ({\bf 0}, {\bf \Phi})$$ $${\bf e}_i \sim \mathcal{N} ({\bf 0}, {\bf \Psi})$$

The author assigns Wishart priors to $${\bf \Phi}^{-1}$$ and $${\bf \Psi}^{-1}$$: $${\bf \Phi}^{-1} \sim \mathcal{W}_K \left( {\bf \Phi}_{*}, \nu_{*} \right)$$ $${\bf \Psi}^{-1} \sim \mathcal{W}_p \left( {\bf \Psi}_{*}, n_{*} \right)$$

In the paper the author writes something I found to be quite interesting:

While classical factor analysis sets $$\bf \Phi = I$$ and uses a diagonal $$\bf \Psi$$ matrix, we impose these restrictions via the prior information matrices $${\bf \Psi}_{*}$$ and $${\bf \Phi}_{*}$$.

Question: What should the values of $${\bf \Psi}_{*}$$ and $${\bf \Phi}_{*}$$ be in order to do what the author is suggesting?

The author does not seem to state exactly how this can be done, but I may have missed it so I will continue reading it. My own research on this matter pointed me to these seemingly similar unanswered questions here and here.

UPDATE: I did some research on the Wishart distribution and if you specify that $$\Psi_*$$ and $$\Phi_*$$ are two diagonal matrices, then $$\mathbb{E} [\Psi]$$ and $$\mathbb{E} [\Phi]$$ will be two diagonal mean matrices. Perhaps, this is what the author is referring to. Still unsure, though.

UPDATE 2: I set $$\Psi_*$$ and $$\Phi_*$$ to diagonal matrices and ran simulations in R, but the results aren't what I expected. The simulated values I obtained are not diagonal, so I think I misinterpreted the author's statement. I thought that if you formulate the factor analysis model with the prior distributions above, that you can consider it the classical factor analysis model by choosing certain hyper-parameter value. But it seems that this formulation does not produce the classical factor analysis model.

UPDATE 3: The classical factor analysis model sets $${\bf \Phi} = {\bf I}$$ (i.e. non-random), sets $$\bf \Psi$$ to be a diagonal matrix (i.e. random diagonal matrix) and assigns prior distributions to only the diagonal elements. What I understand the author's statement to mean, is that I can do the aforementioned things by using Wishart priors on $$\bf \Phi$$ and $$\bf \Psi$$ with special scale matrices $$\bf \Phi_*$$ and $$\bf \Psi_*$$.

Inverse Wishart (which is used in the mentioned article) is used as a prior for the covariance matrix of a multivariate Normal distributed random variable.

This choice is based on the fact that its a conjugate prior for the covariance matrix in this scenario.

If $$\mathbf{X}=(\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n) \sim \mathcal{N}(\mathbf{0}, \mathbf{\Sigma})$$, with a prior $$\mathbf{\Sigma} \sim \mathcal{W}^{-1}(\mathbf{\Psi}, \nu)$$, then the posterior $$p(\mathbf{\Sigma}|\mathbf{X}) \sim \mathcal{W}^{-1}(\mathbf{A}+\mathbf{\Psi},n+\nu)$$ is also an inverse-Wishart distributed random variable ($$\mathbf{A}=\mathbf{X}\mathbf{X}^t$$, $$n$$=number of observations $$\mathbf{X}$$).

Said that, one can impose the structure of the prior for the covariance matrix, by setting the prior scale matrix $$\mathbf{\Psi}$$ opportunely. In the article, the authors set the $$\mathbf{\Psi}=\mathbf{\Psi}^*$$ to be diagonal.

An alternative approach would have been forcing the $$p$$ variables to be independently Normal-distributed. In that case, the conjugate prior for the variance of each dimension would have been the Inverse Gamma.
The limitation of the latter is that forces the posterior $$p$$ variables to be independent, while in the case of an Inverse Wishart, off-diagonal elements of the covariance matrix can have a non-zero-probability to be non-zero.

When setting the scale matrix $$\mathbf{\Psi}^*$$ as diagonal and $$\nu=p+1$$, the correlations in $$\mathbf{\Sigma}$$ have a marginal uniform distribution (par. 2.1 https://arxiv.org/pdf/1408.4050.pdf). This corresponds to a non-informative prior for the correlations, implying that non-zero correlations require strong evidence from the data $$\mathbf{X}$$.

An interesting alternative, suggested by Gelman, is to use Half-Cauchy priors (the linked article focuses on 1-dimensional hierarchical models):

http://www.stat.columbia.edu/~gelman/research/published/taumain.pdf

• This seems to be telling me that imposing the restrictions that the scale matrix ${\bf \Psi}^*$ be diagonal is not the same as imposing the classical factor analysis restrictions? Aug 3 '20 at 21:16
• Perhaps, you, much like the author of the paper, are saying that restricting $\bf \Psi$ to be a diagonal matrix is too harsh of a restriction. The magnitude of the off-diagonal elements could be driven mainly by the data by way of the inverse-Wishart prior. Aug 3 '20 at 21:34
• Yes, that's what the authors say. They don't want to impose zero off-diagonal values. Here projecteuclid.org/download/pdfview_1/euclid.ba/1369407559 you can find the details about this property of the off-diagonal correlations. Also, bear in mind that the Bayesian approach for the covariance assigns a probability distribution for each of the matrix elements, so you cannot directly compare with the classic factor analysis. To do that, you may want to look at the expected value of the inverse Wishart. Aug 3 '20 at 23:39
• Okay, then it seems that I misunderstood the author's statement. I thought that he meant that using diagonal scale matrices in the inverse-Wishart prior distribution would somehow assign values of zero to the off-diagonal elements of $\bf \Psi$ and $\bf \Phi$, but this is incorrect. Aug 4 '20 at 0:12
• The difference in the Bayesian approach is that the prior is set such that the covariance is diagonal on average ($\mathbb{E}[\Sigma|\Psi_*, \eta]=\text{diag}(\ldots)$). This page gives some additional useful info mssc.mu.edu/~daniel/bfa.html Aug 4 '20 at 12:05