Skip to main content
50 votes

How to create an arbitrary covariance matrix

I like to have control over the objects I create, even when they might be arbitrary. Consider, then, that all possible $n\times n$ covariance matrices $\Sigma$ can be expressed in the form $$\Sigma= ...
whuber's user avatar
  • 328k
41 votes
Accepted

How to create an arbitrary covariance matrix

Create an $n\times n$ matrix $A$ with arbitrary values and then use $\Sigma = A^T A$ as your covariance matrix. For example ...
Henry's user avatar
  • 40.4k
40 votes
Accepted

Different covariance types for Gaussian Mixture Models

A Gaussian distribution is completely determined by its covariance matrix and its mean (a location in space). The covariance matrix of a Gaussian distribution determines the directions and lengths of ...
whuber's user avatar
  • 328k
27 votes
Accepted

Is the average of positive-definite matrices also positive-definite?

Yes, it is. jth asnwer is correct (+1) but I think you can get a much simple explanation with just basic Linear Algebra. Assume $A$ and $B$ are positive definite matrices for size $n$. By definition ...
usεr11852's user avatar
  • 44.8k
27 votes

Determine the off - diagonal elements of covariance matrix, given the diagonal elements

You might find it instructive to start with a basic idea: the variance of any random variable cannot be negative. (This is clear, since the variance is the expectation of the square of something and ...
whuber's user avatar
  • 328k
26 votes

A measure of "variance" from the covariance matrix?

The variance of a scalar variable is defined as the expected square deviation of the variable from its mean: $$\operatorname{Var}(X) = \operatorname E\left[\left(X - \operatorname E\left[X\right]\...
HelloGoodbye's user avatar
25 votes

Why does inversion of a covariance matrix yield partial correlations between random variables?

Here is a proof with just matrix calculations. I appreciate the answer by whuber. It is very insightful on the math behind the scene. However, it is still not so trivial how to use his answer to ...
Po C.'s user avatar
  • 350
25 votes
Accepted

A measure of "variance" from the covariance matrix?

(The answer below merely introduces and states the theorem proven in Eq. (0) The beauty in that paper is that most of the arguments are made in terms of basic linear algebra. To answer this question ...
user603's user avatar
  • 22.8k
24 votes
Accepted

Why are symmetric positive definite (SPD) matrices so important?

A (real) symmetric matrix has a complete set of orthogonal eigenvectors for which the corresponding eigenvalues are are all real numbers. For non-symmetric matrices this can fail. For example, a ...
Matthew Drury's user avatar
19 votes
Accepted

What to do with random effects correlation that equals 1 or -1?

Singular random-effect covariance matrices Obtaining a random effect correlation estimate of +1 or -1 means that the optimization algorithm hit "a boundary": correlations cannot be higher than +1 or ...
amoeba's user avatar
  • 106k
18 votes
Accepted

Are a sum and a product of two covariance matrices also a covariance matrix?

Background A covariance matrix $\mathbb{A}$ for a vector of random variables $X=(X_1, X_2, \ldots, X_n)^\prime$ embodies a procedure to compute the variance of any linear combination of those random ...
whuber's user avatar
  • 328k
17 votes
Accepted

Do the Determinants of Covariance and Correlation Matrices and/or Their Inverses Have Useful Interpretations?

I was able to cobble together some general principles, use cases and properties of these matrices from a desultory set of sources; few of them address these topics directly, with most merely mentioned ...
SQLServerSteve's user avatar
17 votes

Determine the off - diagonal elements of covariance matrix, given the diagonal elements

An intuitive method to determine this answer quickly is to just remember that covariance matrices may be interpreted in the form \begin{equation} A = \begin{pmatrix} \sigma_1^2 & \rho_{12}\sigma_1\...
jodag's user avatar
  • 343
16 votes
Accepted

Covariance matrix as linear transformation

This is not true for all (non-zero) vectors, but let's explore. The covariance matrix $A$ has an orthonormal basis $v_1, \ldots, v_n$ of eigenvectors with eigenvalues $\lambda_1 \geq \lambda_2 \geq \...
WimC's user avatar
  • 306
16 votes
Accepted

Is the sum of the diagonal elements of a covariance matrix always equal or larger than the sum of its off-diagonal elements?

Consider the general equi-correlation covariance matrix: \begin{align} \Sigma = \begin{bmatrix} 1 & \rho & \cdots & \rho \\ \rho & 1 & \cdots & \rho \\ \vdots & \vdots &...
Zhanxiong's user avatar
  • 20.5k
15 votes

Is the average of positive-definite matrices also positive-definite?

Of course. The set of positive definite matrices forms a cone, meaning it is closed under positive linear combinations and scaling.
nth's user avatar
  • 806
15 votes
Accepted

Robust covariance and OGK outlier detection

To justify the OGK, you need to see the following chain: Outlier detection and robust estimation are essentially equivalent problems, At its core, outlier detection is related to the problem of (...
user603's user avatar
  • 22.8k
14 votes
Accepted

Covariance matrix of complex random variables

Here is a geometric interpretation. First, take two vectors in $\mathbb{R}^2$ $$\vec{\mathbb{z}}=[x,y] \,, \vec{\mathbb{w}}=[u,v]$$ For these vectors, there are two standard types of "products", ...
GeoMatt22's user avatar
  • 13k
14 votes

Are a sum and a product of two covariance matrices also a covariance matrix?

A real matrix is a covariance matrix if and only if it is symmetric positive semi-definite. Hints: 1) If $X$ and $Y$ are symmetric, is $X+Y$ symmetric? If $z^TX z \ge 0$ for any $z$ and $z^TY z \ge ...
Mark L. Stone's user avatar
14 votes

A measure of "variance" from the covariance matrix?

Although the trace of the covariance matrix, tr(C), gives you a measure of the total variance, it does not take into account the correlation between variables. If you need a measure of overall ...
Sahar's user avatar
  • 141
14 votes

Determine the off - diagonal elements of covariance matrix, given the diagonal elements

$A$ is posdef so, by Sylvesters criterion, $det(A) = 121 \cdot 81 - c^2 \geq 0$. Thus, any $c \in [-99, 99]$ will produce a valid covariance matrix.
Hunaphu's user avatar
  • 2,232
14 votes

Is it possible to have a multivariate random distribution with all its random variables (pair-wise) reverse correlated?

If $X \mathrel{:=} \left(X_1, \ldots, X_k\right)^\top \sim \mathop{\mathrm{Multinomial}}\left(n, \left(p_1, \ldots, p_k\right)^\top\right)$ with $n \in \mathbb N_{\geq 1}, k \in \mathbb N_{\geq 2},$ ...
statmerkur's user avatar
  • 6,235
13 votes
Accepted

What is the geometric relationship between the covariance matrix and the inverse of the covariance matrix?

Before I answer your questions, allow me to share how I think about covariance and precision matrices. Covariance matrices have a special structure: they are positive semi-definite (PSD), which means ...
PAF's user avatar
  • 605
13 votes

Is the sum of two singular covariance matrices also singular?

The Spectral Theorem informs us that any $n\times n$ covariance matrix $\Sigma$ can be diagonalized. That is, there exists an orthogonal matrix $Q$ for which $$Q\Sigma Q^\prime = \pmatrix{\sigma^2_1 &...
whuber's user avatar
  • 328k
13 votes
Accepted

Is it possible to have a multivariate random distribution with all its random variables (pair-wise) reverse correlated?

Mathematically, as I commented, you can use the equi-correlation matrix \begin{align*} \Sigma = \begin{bmatrix} 1 & \rho & \cdots & \rho \\ \rho & 1 & \cdots & \rho \\ \vdots ...
Zhanxiong's user avatar
  • 20.5k
13 votes
Accepted

Beyond AR(1) as a covariance structure for mixed models with repeated measures

When we encounter models with repeated measures in time, we naturally want to account for the correlations within subjects and also the possibility of heterogeneous variances across time points. When ...
Robert Long's user avatar
12 votes

Why are symmetric positive definite (SPD) matrices so important?

You'll find some intuition in the many elementary ways of showing the eigenvalues of a real symmetric matrix are all real: https://mathoverflow.net/questions/118626/real-symmetric-matrix-has-real-...
Alex R.'s user avatar
  • 14k
12 votes
Accepted

Relation between Covariance matrix and Jacobian in Nonlinear Least Squares

This is based on the standard approximation to the Hessian of a nonlinear least squares problem used by Gauss-Newton and Levenberg-Marquardt algorithms. Consider the nonlinear least squares problem: ...
Mark L. Stone's user avatar
12 votes
Accepted

Using shrinkage when estimating covariance matrix before doing PCA

The paper you cited (Donoho et al. 2013 Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model) is an impressive piece of work which I confess I did not really study. Nevertheless, I believe ...
amoeba's user avatar
  • 106k
12 votes
Accepted

Is CoStandard Deviation a thing?

One useful property of the standard deviation is that it has the same units as the mean, so the magnitudes of $\sigma_X$ and $\bar X$ are directly comparable. I've never seen anyone compute the co-...
Ben Bolker's user avatar
  • 44.2k

Only top scored, non community-wiki answers of a minimum length are eligible