# When the sample covariance matrix becomes singular

Assume a data set $X$ which contains $k$ iid random vectors of size $p$. Denote by $S$ the sample covariance matrix.

Really I have some questions and I need your very appreciated opinions:

1) Ledoit and Wolf (2003) considered that the sample covariance $S$ is also can be written as $S=k^{-1} X(I - k^{-1} 11')X'$ where $1$ is a comfortable vector of ones. They demonstrate the reason of why $S$ becomes not invertible when $p$ is $>=k$. By the way, the reason was that under the case $k>p+1$, the rank of $S$ is at most equal to the rank of the matrix $I - k^{-1} 11'$ which is $k-1$. So when $p$ exceeds $k$, the rank of $S$ becomes deficient, and so $S$ becomes singular.

In fact, I didn't understand how $k^{-1} X(I - k^{-1} 11')X'$ is equal to $k^{-1} XX'$ ? And why when $p>=k$, the matrix becomes deficient?

2) To handle the problem of $p>=k$, Ledoit and Wolf(2004) highlight on the problem of creating a weighted average between the sample covariance matrix which has little or no structure and other (called shrinkage target) which has a lot of structure (also called structured matrix). They also mentioned that those estimated coefficients in $S$ thar are extremely high tend to contain a lot of positive error, and the extreme low cofficients tend to contain a lot of negative error.

I didn't understand what was meant by "has little or no structure" and by "a lot of structure" ?? Furthermore, how can we explain in another way the fact of extreme coefficients of $S$?

3) When $p>=k$, the sample covariance matrix is not positive definite (and in fact it will be semi-positive definite). In this case the eigenvalues of $S$ will spread out a lot.

I know that the some eigenvalues will erroneously be equal to zero. But what was meant by "Spread out"? can someone gives me more details and explanations about that.

• Note that your opening sentence refers to a zero mean. Does Ledoit and Wolfe specify zero mean or are they dealing with a general mean? Jun 8, 2015 at 10:15
• Ah no in fact; Ledoit and Wolf do not consider a specific distribution to S, their shrinkage estimator was distribution free Jun 8, 2015 at 10:26
• I wasn't really talking about the distribution - if they deal with the general case rather than the zero-mean case you shouldn't expect the formulas to be the same. Jun 8, 2015 at 10:37
• Yes you're right. I deleted the assumption from my question:) Jun 8, 2015 at 11:10
• But the statement about the formula for the sample covariance matrix relies on that deleted assumption. The first formula is now wrong. Jun 8, 2015 at 12:32