What is the difference between a matrix normal distribution and the multivariate gaussian distribution?

Question

$\newcommand{\vec}{\operatorname{vec}}$Consider a set of $N$ matrices $X_1, X_2, \ldots, X_N$. I want to estimate the distribution of these matrices represented by the mean and covariance.

I address this problem by simply vectorizing my matrices $\vec(X_1), \vec(X_2), \ldots, \vec(X_N)$, then I compute the mean and covariance as I would for a multivariate Gaussian distribution. In other words, this implies that $\vec(X) \sim \mathcal{N}(M, \Sigma)$ where $\vec(X)$ is reshaped to get $X$ after sampling

However, I came across matrix normal distributions, which defines the distribution based on the row covariance and column covariance. So, a matrix sampled from a matrix normal distribution is given by $X \sim \mathcal{M}\mathcal{N}(M, U, V)$.

In the wikipedia article, it says that the relation between the matrix normal distribution and the multivariate gaussian distribution for a random matrix $X$ is $X \sim \mathcal{M}\mathcal{N}(M, U, V)$ if and only if $X \sim \mathcal{N}(V \otimes U)$ where $\otimes$ is the operator for the kronecker product.

If I generate a set of random matrices, I can estimate the parameters of the distribution in multiple ways. I can vectorize my matrices as treat as a multivariate Gaussian to get the covariance. Or, I can compute the row covariance and column covariance, then I can compute the covariance from the kronecker product.

However, I am not getting the same values for the covariance from the vectorized approach verses the kronecker product approach. Should I get the same result for each? If not, why are they different? I don't understand what these distributions represent in practice, so I not sure which should be used in which scenario.

Answer 1

Example for diagonal matrices $$U = \begin{bmatrix} u_1 & 0 & 0 \\0& u_2 & 0 \\ 0&0&u_3 \end{bmatrix}$$ and $$V = \begin{bmatrix} v_1 & 0 & 0 \\0& v_2 & 0 \\ 0&0&v_3 \end{bmatrix}$$

lead to $$\Sigma = \begin{bmatrix} u_1v_1 & 0 &0&0 &0&0&0&0&0\\ 0&u_1v_2 & 0 &0&0 &0&0&0&0\\ 0&0&u_1v_3 & 0 &0 &0&0&0&0\\ 0&0&0&u_2v_1 & 0 &0&0&0&0\\ 0&0&0&0&u_2v_2 & 0 &0&0 &0\\ 0&0&0&0&0&u_2v_3 & 0 &0 &0\\ 0&0&0&0&0&0&u_3v_1 & 0 &0\\ 0&0&0&0&0&0&0&u_3v_2 & 0 \\ 0&0&0&0&0&0&0&0&u_3v_3 \\ \end{bmatrix}$$

the $2n$ values in $U$ and $V$ will lead to $n^2$ values in $\Sigma$.

So if you estimate those values in $\Sigma$ directly instead of $U \otimes V$, the you will have a larger degree of possible values and the estimates will be different.

The same is true for non diagonal matrices. The matrix normal distribution restricts the parameterization of the $n^4$ values in $\Sigma$ to the $2n^2$ values in $U$ and $V$.