Result for covariance between elements of the sample covariance matrix

Given data matrix $X_{n,p}$ where $n$ is the sample size, $p$ is the dimension, the sample covariance is

$$\hat{\Sigma}=\frac{1}{n-1}X^\top X=\{\hat{\sigma}_{ij}\}_{1\le i,j\le p}$$

Is there any result on the quantity like $cov(\hat{\sigma}_{i_0,j_0},\hat{\sigma}_{i_0,j_0+1})$? I.e. what is the distribution? Or asymptotic distribution? And for even more general terms like $cov(\hat{\sigma}_{i_1,j_1},\hat{\sigma}_{i_2,j_2})$? We can assume Gaussian distribution first, but not necessarily.

• why would you divide by $n-1$ when you assume the mean of each row is the zero vector? – Taylor Jan 17 '18 at 2:26
• Oh yes, we can assume $\mu=0$ first for simplicity. And I think sample covariance is divided by $n-1$, and MLE is divided by $n$. But this is not a big problem, either is OK. – breezeintopl Jan 17 '18 at 2:30

If you're assuming Normality, you want to have a look at the Wishart distribution.

If $\mathbf{X}_{n,p}$ ($n \ge p$) has rows that are iid multivariate normal with mean $0$ and variance $\Sigma$, then $$\mathbf{X}^T\mathbf{X} \sim \text{Wishart}_p(n,\Sigma).$$ This is for the particular case you have where the mean of each row is the zero vector. Also

1. $\widehat{\Sigma} \sim \text{Wishart}_p(n,\frac{1}{n-1}\Sigma)$ so you probably want to divide by $n$ rather then $n-1$ because $E\hat{\Sigma} = \frac{n}{n-1}\Sigma$. This is biased.
2. The $\text{vec}$ operator the kronecker product and all their properties will come in handy for looking at higher order moments. Closed form expressions exist for all covariances and variances.

If you want to take a closer look at the formula for all the variances and covariances, it helps to write $\mathbf{X} = \mathbf{Z}\mathbf{C}$, where $\mathbf{C}$ is the Cholesky decomposition of $\Sigma = \mathbf{C}^T\mathbf{C}$. The variance is \begin{align*} \text{Var}\left( \text{vec} \left[\mathbf{X}^T\mathbf{X} \right]\right) &= \text{Var}\left( \text{vec} \left[\mathbf{C}^T\mathbf{Z}^T\mathbf{Z} \mathbf{C} \right]\right) \\ &= \text{Var}\left(\left[\mathbf{C}^T\otimes \mathbf{C}^T \right] \text{vec} \left[\mathbf{Z}^T\mathbf{Z} \right]\right)\tag{prop.s of vec/kron} \\ &= \left[\mathbf{C}^T\otimes \mathbf{C}^T \right] \text{Var}(\text{vec}(\mathbf{Z}^T\mathbf{Z})) \left[\mathbf{C}^T\otimes \mathbf{C}^T \right] ^T \tag{properties of var}\\ &= \left[\mathbf{C}^T\otimes \mathbf{C}^T \right] \left[I_p \otimes I_p + M_p \right] \left[\mathbf{C}\otimes \mathbf{C} \right] \end{align*} where $\mathbf{M}_p$ is a $p^2 \times p^2$ matrix that is described further in this document. If you want the variance of $\hat{\Sigma}$, just scale the last expression by $(n-1)^{-2}$.

• Do you mean that (in Gaussian) since $X^\top X$ is known, i.e. Wishart distributed, so we can derive everything based on this distribution (this problem becomes just some calculation of Wishart distribution?)? – breezeintopl Jan 17 '18 at 2:58
• @breezeintopl yes – Taylor Jan 17 '18 at 3:02
• One follow up question. Is $cov(S)$ and $cov(vec(S))$ the same thing? In the document that you mentioned: $cov(S)$ is eq(6), and $cov(vec(S))$ is eq(7). Moreover, in reference 2 of your document proposition 8.3 (ii) says eq(6) can be further derived as $2n\Sigma\otimes\Sigma$ (which is not equal to eq(7)). So, the definition of a $cov(matrix)$ is not the same as $cov(vec(matrix))$? – breezeintopl Jan 21 '18 at 22:27
• On one hand, to my knowledge, the definition of covariance of a (random) matrix only appears in the definition Wishart distribution as sum of (multivariate) squares. So derivation of $cov(S)$ should go back to some transformation of standard Wishart, while $cov(vec(S))$ is just covariance of a random vector (which is well-defined as a matrix). And thus, I think results for the two can be different. However, on the other hand, shouldn't $cov(S)$ and $cov(vec(S))$ contain the same information ($cov(s_{i_1,j_1},s_{i_2,j_2})$) ? Then why they are different? – breezeintopl Jan 21 '18 at 22:37
• @breezeintopl no it's true for any matrix $S$. You always have to $\text{vec}$ it out – Taylor Jan 21 '18 at 23:01