12
$\begingroup$

Say $X \in \mathbb{R}^n$ is a random variable with covariance $\Sigma \in \mathbb{R}^{n\times n}$. By definition, entries of the covariance matrix are covariances: $$ \Sigma_{ij} = Cov( X_i,X_j). $$ Also, it is known that entries of the precision $\Sigma^{-1}$ satisfy: $$ \Sigma^{-1}_{ij} = Cov(X_i,X_j| \{X_k\}_{k=1}^n \backslash X_i,X_j\}), $$ where the right hand side is the covariance of $X_i$ with $X_j$ conditioned on all other variables.

Is there a statistical interpretation to the entries of a square root of $\Sigma$ or $\Sigma^{-1}$? By square root of a square matrix $A$ I mean any matrix $M$ such that $M^tM = A$. An eigenvalue decomposition of said matrices does not give such entry-wise interpretation as far as I can see.

$\endgroup$
3
  • 1
    $\begingroup$ @kjetilbhalvorsen added explanation. Swept a couple of details under the rugh, though. What is the value conditioned on? None given, so it has to be averaged over all values. $\endgroup$
    – Yair Daon
    Jun 28, 2017 at 17:00
  • 1
    $\begingroup$ The account I gave of regression, correlation, and conditional distributions at stats.stackexchange.com/questions/71260/… provides explicit geometric constructions of two different square roots of the inverse covariance matrix. These geometric ideas generalize to higher dimensions, thereby supplying at least two distinct, well-known statistical interpretations (namely, PCA and multiple regression). $\endgroup$
    – whuber
    Dec 21, 2018 at 16:37
  • $\begingroup$ what is the size of the square root of the empirical covariance matrix? is it rectangular? $\endgroup$ Oct 24, 2021 at 14:07

3 Answers 3

3
$\begingroup$

I will write matrix square roots of $\Sigma$ as $\Sigma=A A^T$, to be consistent with the Cholesky decomposition which is written as $\Sigma = L L^T$ where $L$ is lowtri (lower triangular). So let $X$ be a random vector with $\DeclareMathOperator{\E}{\mathbb{E}} \DeclareMathOperator{\Cov}{\mathbb{Cov}}\DeclareMathOperator{\Var}{\mathbb{Var}} \E X=\mu$ and $\Var X =\Sigma$. Let now $Z$ be a random vector with expectation zero and unit covariance matrix.

Note that there are (except for the scalar case) infinitely many matrix square roots. If we let $A$ be one of the, then we can find all the others as $A \mathcal{O}$ where $\mathcal{O}$ is any orthogonal matrix, that is, $\mathcal{O} \mathcal{O}^T = \mathcal{O}^T \mathcal{O} =I$. This is known as unitary freedom of square roots.

Let us look at some particular matrix square roots.

  1. First a symmetric square root. Use the spectral decomposition to write $\Sigma = U \Lambda U^T = U\Lambda^{1/2}(U\Lambda^{1/2})^T$. Then $\Sigma^{1/2}=U\Lambda^{1/2}$ and this can be interpreted as the PCA (principal component analysis) of $\Sigma$.

  2. The Cholesky decomposition $\Sigma=L L^T$ and $L$ is lowtri. We can represent $X$ as $X=\mu + L Z$. Multiplying out to get scalar equations, we get a triangular system in $Z$, which in the time series case can be interpreted as a MA (moving average) representation.

  3. The general case $A= L \mathcal{O}$, using the above we can interpret this as a MA representation after rotating $Z$.

$\endgroup$
7
  • 1
    $\begingroup$ +1 Re square root calculation through spectral decomposition, I've seen $\Sigma^{1/2}=U\Lambda^{1/2}U^{T}$ as well. But this would be different from the one you mentioned, yet both are valid, is that true? $\endgroup$
    – NULL
    Nov 1, 2019 at 14:32
  • $\begingroup$ @NULL I believe that is valid, and is called the principle square root. If I'm not mistaken, $\Sigma^{\frac{1}{2}} = U\Lambda^{\frac{1}{2}}$ would not actually be symmetric, so maybe that is an error. $\endgroup$
    – Brian
    Feb 16, 2021 at 10:46
  • $\begingroup$ what is the size of the square root of the empirical covariance matrix? is it rectangular? $\endgroup$ Oct 24, 2021 at 14:07
  • 1
    $\begingroup$ @Charlie Parker: The square root is of the same size as the covariance matrix. $\endgroup$ Oct 24, 2021 at 18:57
  • $\begingroup$ @kjetilbhalvorsen thanks for the response! If the covariance matrix $\Sigma_{X, Y}$ is of size $[D_1, D_2]$ and of rank $r$, for some reason I'd expect it to be of size $[r, r]$ (similar to the SVD). But that is wrong. Why is that? $\endgroup$ Oct 25, 2021 at 17:32
2
$\begingroup$

An nxn matrix can have many square roots as you mention. However a covariance matrix must be positive semi-definite and a positive semi-definite matrix has only one square root that is also positive semi-definite. Take a look at the wikipedia article titled "Square root of a matrix".

$\endgroup$
4
  • 3
    $\begingroup$ I did not ask for a symmetric square root. For example, a Cholesky decomposition is enough for the purpose of this question. $\endgroup$
    – Yair Daon
    Jan 16, 2017 at 22:03
  • 1
    $\begingroup$ I didn't say that a term in the covariance matrix couldn't be negative. I am talking about a different property that a covariance matrix is at least positive semi-definite. $\endgroup$ Feb 26, 2019 at 23:07
  • $\begingroup$ The wikipedia page about the square root of a matrix and why positive definite and semi-definite matrices have a unique positive (semi-)definite square root are in: en.wikipedia.org/wiki/…. Specifically you can refer to the sections: 2 Positive semidefinite matrices 3 Matrices with distinct eigenvalues $\endgroup$
    – user96265
    Oct 8, 2021 at 20:04
  • $\begingroup$ what is the size of the square root of the empirical covariance matrix? is it rectangular? $\endgroup$ Oct 24, 2021 at 14:07
0
$\begingroup$

Sometimes people are interested in estimating the locations of zeros in the precision matrix for the same reason you describe above. If $M$ is your square root matrix, i.e. $M'M = \Sigma^{-1}$, then for nodes $i \neq j$ $$ \Sigma^{-1}_{i,j} = 0 \iff M_i'M_j = 0 $$ so I imagine that looking at the inner product between columns of your estimated square root matrix will give you some number akin to how close to conditionally independent two nodes are. Just an idea.

The square root of the covariance matrix, that's the scale. I imagine simulating a standard normal random vector, and then pre-multiplying by the square root matrix. If this matrix is lower-triangular, then I always imagine doing all the little multiplications and additions out.

There are also individual cases you can think of where certain elements of the square root matrix are just square roots of individual elements of the covariance matrix. This isn't that interesting, though, so I figure you have already thought of this.

$\endgroup$
4
  • $\begingroup$ what is the size of the square root of the empirical covariance matrix? is it rectangular? $\endgroup$ Oct 19, 2021 at 21:07
  • $\begingroup$ @CharlieParker en.wikipedia.org/wiki/Square_root_of_a_matrix in this case, since $\Sigma^{-1}$ is full rank positive definite symmetric, $M$ is rectangular $\endgroup$
    – Taylor
    Oct 19, 2021 at 21:20
  • $\begingroup$ so is the shape of the square root of a covariance matrix $\Sigma_{X, Y}$ (of size [D1, D2]) of size $[r, r]$ for $\Sigma^{1/2}$ where $r = rank(\Sigma_{X, Y})$? $\endgroup$ Oct 24, 2021 at 14:10
  • $\begingroup$ actually for $\Sigma_{X} = X^T X$ covariance of (centered) X with itself. $\endgroup$ Oct 24, 2021 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.