# Why is covariance matrix not positive-definite when number of observations is less than number of dimensions?

I have a data matrix $X$ of size $n\times p$ with $n < p$, where $n$ is the number of observations and $p$ is the number of dimensions.

My question is: why $n < p$ results in not a positive-definite covariance matrix?

(By the way I want to use this data in a Factor Analysis model. Do you have any idea about Matlab code implementing a standard Factor Analysis for this kind of data when $n < p$?)

• You cannot do factor analysis (most algorithms and implementations won't allow) on a singular correlation matrix (and when n<p, it is but singular) as well as negative-definite matrix (which could appear sometimes with pairwise deletion of missng values). Feb 25 '16 at 19:19
• @ttnphns; Is there any solution to the problem or simply I have to forget factor analysis? Feb 26 '16 at 11:49
• This is a theoretical problem (see Pt 6). Due to relatively low n correlations cannot enough differentiate from one another and do not allow the factor model to play in full accordingly. So forget FA. It is good to have n>p at least 3-5 times, practically. Feb 26 '16 at 11:56
• Then in my case which method of dimension reduction would you suggest? And can you also propose a standard matlab code for that method? Mar 2 '16 at 10:34
• +1 but your second question (about the Matlab code) is off-topic here. Mar 3 '16 at 0:06

This result is a direct, simple consequence of the fact that the rank of the $$p\times p$$ matrix $$X^\prime X$$ cannot be any greater than the smaller of $$n$$ and $$p$$, which is strictly less than $$p$$ in this case. That makes the $$p\times p$$ matrix $$X^\prime X$$ singular, which is equivalent to the existence of a nonzero $$x$$ for which $$X^\prime X x = 0$$. Consequently $$x^\prime X^\prime X x = x 0 = 0$$ demonstrates that $$X^\prime X$$ is indefinite.

Although I referenced $$X$$ in this argument, the column-centered version of $$X$$ that is used in computing the covariance matrix also has dimensions $$n\times p$$, so the same conclusions apply to it.

### Definitions

The rank of a matrix $$X$$ is the dimension of its image, defined to be the set of all $$Xx$$ as $$x$$ ranges among all possible vectors.

The column-centered version of a matrix is obtained by subtracting the arithmetic mean of each column from the entries in that column.

The covariance matrix of $$X$$ is proportional to $$Y^\prime Y$$ where $$Y$$ is the column-centered version of $$X$$. (Depending on convention, the factor of proportionality is $$1/n$$ or $$1/(n-1)$$.)

A square matrix $$A$$ is singular when it has no multiplicative inverse. Equivalently, there is a nonzero vector $$x$$ for which $$Ax=0$$. ($$A$$ has a nontrivial kernel.) Equivalently, the rank of $$A$$ is strictly less than the dimension of its image space (equal to the number of rows of $$A$$).

A square matrix $$A$$ is semi-definite when all numbers of the form $$x^\prime A x$$ have the same sign (or are zero), regardless of what the vector $$x$$ might be. According to the sign, $$A$$ would be called negative semi-definite or positive semi-definite.

A semi-definite square matrix $$A$$ is definite when the only vector $$x$$ for which $$x^\prime A x = 0$$ is the zero vector itself.