5
$\begingroup$

I have a data matrix $X$ of size $n\times p$ with $n < p$, where $n$ is the number of observations and $p$ is the number of dimensions.

My question is: why $n < p$ results in not a positive-definite covariance matrix?

(By the way I want to use this data in a Factor Analysis model. Do you have any idea about Matlab code implementing a standard Factor Analysis for this kind of data when $n < p$?)

$\endgroup$
6
  • 2
    $\begingroup$ You cannot do factor analysis (most algorithms and implementations won't allow) on a singular correlation matrix (and when n<p, it is but singular) as well as negative-definite matrix (which could appear sometimes with pairwise deletion of missng values). $\endgroup$
    – ttnphns
    Feb 25 '16 at 19:19
  • $\begingroup$ @ttnphns; Is there any solution to the problem or simply I have to forget factor analysis? $\endgroup$
    – pierre
    Feb 26 '16 at 11:49
  • 1
    $\begingroup$ This is a theoretical problem (see Pt 6). Due to relatively low n correlations cannot enough differentiate from one another and do not allow the factor model to play in full accordingly. So forget FA. It is good to have n>p at least 3-5 times, practically. $\endgroup$
    – ttnphns
    Feb 26 '16 at 11:56
  • $\begingroup$ Then in my case which method of dimension reduction would you suggest? And can you also propose a standard matlab code for that method? $\endgroup$
    – pierre
    Mar 2 '16 at 10:34
  • $\begingroup$ +1 but your second question (about the Matlab code) is off-topic here. $\endgroup$
    – amoeba
    Mar 3 '16 at 0:06
4
$\begingroup$

This result is a direct, simple consequence of the fact that the rank of the $p\times p$ matrix $X^\prime X$ cannot be any greater than the smaller of $n$ and $p$, which is strictly less than $p$ in this case. That makes the $p\times p$ matrix $X^\prime X$ singular, which is equivalent to the existence of a nonzero $x$ for which $X^\prime X x = 0$. Consequently $$x^\prime X^\prime X x = x 0 = 0$$ demonstrates that $X^\prime X$ is indefinite.

Although I referenced $X$ in this argument, the column-centered version of $X$ that is used in computing the covariance matrix also has dimensions $n\times p$, so the same conclusions apply to it.


Definitions

The rank of a matrix $X$ is the dimension of its image, defined to be the set of all $Xx$ as $x$ ranges among all possible vectors.

The column-centered version of a matrix is obtained by subtracting the arithmetic mean of each column from the entries in that column.

The covariance matrix of $X$ is proportional to $Y^\prime Y$ where $Y$ is the column-centered version of $X$. (Depending on convention, the factor of proportionality is $1/n$ or $1/(n-1)$.)

A square matrix $A$ is singular when it has no multiplicative inverse. Equivalently, there is a nonzero vector $x$ for which $Ax=0$. ($A$ has a nontrivial kernel.) Equivalently, the rank of $A$ is strictly less than the dimension of its image space (equal to the number of rows of $A$).

A square matrix $A$ is semi-definite when all numbers of the form $x^\prime A x$ have the same sign (or are zero), regardless of what the vector $x$ might be. According to the sign, $A$ would be called negative semi-definite or positive semi-definite.

A semi-definite square matrix $A$ is definite when the only vector $x$ for which $x^\prime A x = 0$ is the zero vector itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.