# Prove that sample covariance matrix is positive definite [duplicate]

Consider the $$p \times p$$ sample covariance matrix:

$$\mathbf{S} = \frac{1}{n-1} \cdot \mathbf{Y}_\mathbf{c}^\text{T} \mathbf{Y}_\mathbf{c} \quad \quad \quad \mathbf{Y}_\mathbf{c} = \mathbf{C} \mathbf{Y},$$

where $$\mathbf{C} = \mathbf{I}-\frac{1}{n} \mathbf{1} \mathbf{1}^\text{T}$$ is the $$n \times n$$ centering matrix and $$\mathbf{Y}$$ is an $$n \times p$$ matrix. How can it be proved that if the variables are continuos, not linearly related and $$n-1> p$$ then the sample covariance matrix is ​​positive definite?

The following clue is found in Ranchera's book:

• No, this isn't my question. – user. Sep 15 '20 at 1:58
• You've asked how to prove that a sample covariance matrix is positive definite. The answers provide proofs that the sample covariance matrix is positive semi-definite. How is this not an answer to your question? – Sycorax Sep 15 '20 at 2:00
• I want to know how the condition $𝑛 - 1> 𝑝$ makes the matrix positive definite. I know covariance matrix is positive semi-definite. – user. Sep 15 '20 at 2:12
• It's possible that the misunderstanding arises because the claim is false. $S$ is positive semi-definite whenever the columns of $Y$ are linearly dependent -- for example, if each row of $Y$ sums to the same number. This circumstance is addressed in the duplicate. – Sycorax Sep 15 '20 at 2:16
• Your question about $n-1\gt p$ is asked and answered at stats.stackexchange.com/questions/60622. – whuber Sep 15 '20 at 12:51

First, let's simplify the equation for your sample covariance matrix. Using the fact that the centering matrix is symmetric and idempotent you get the $$p \times p$$ form:

\begin{align} \mathbf{S} &= \frac{1}{n-1} \cdot \mathbf{Y}_\mathbf{c}^\text{T} \mathbf{Y}_\mathbf{c} \\[6pt] &= \frac{1}{n-1} \cdot (\mathbf{C} \mathbf{Y})^\text{T} (\mathbf{C} \mathbf{Y}) \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{Y}^\text{T} \mathbf{C}^\text{T} \mathbf{C} \mathbf{Y} \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{Y}^\text{T} \mathbf{C} \mathbf{Y}. \\[6pt] \end{align}

This is a simple quadratic form in $$\mathbf{Y}$$. I will show that this matrix is non-negative definite (or "positive semi-definite" if you prefer) but it is not always positive definite. To do this, consider an arbitrary non-zero column vector $$\mathbf{z} \in \mathbb{R}^p - \{ \mathbf{0} \}$$ and let $$\mathbf{a} = \mathbf{Y} \mathbf{z} \in \mathbb{R}^n$$ be the resulting column vector. Since the centering matrix is non-negative definite (it has one eigenvalue equal to zero and the rest are equal to one) you have:

\begin{align} \mathbf{z}^\text{T} \mathbf{S} \mathbf{z} &= \frac{1}{n-1} \cdot \mathbf{z}^\text{T} \mathbf{Y}^\text{T} \mathbf{C} \mathbf{Y} \mathbf{z} \\[6pt] &= \frac{1}{n-1} \cdot (\mathbf{Y} \mathbf{z})^\text{T} \mathbf{C} \mathbf{Y} \mathbf{z} \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{a}^\text{T} \mathbf{C} \mathbf{a} \geqslant 0. \\[6pt] \end{align}

This shows that $$\mathbf{S}$$ is non-negative definite. However, it is not always positive definite. To see this, take any $$\mathbf{z} \neq \mathbf{0}$$ giving $$\mathbf{a} = \mathbf{Y} \mathbf{z} \propto \mathbf{1}$$ and substitute into the quadratic form to get $$\mathbf{z}^\text{T} \mathbf{S} \mathbf{z} = 0$$.

Update: This update is based on the additional information you have added in your edit to the question and your comments. In order to get a positive definite sample variance matrix you need $$\mathbf{a}^\text{T} \mathbf{C} \mathbf{a} > 0$$. If $$n-1>p$$ and all $$n$$ rows of $$\mathbf{Y}$$ are linearly independent then $$\mathbf{Y} \mathbf{z} \propto \mathbf{1}$$ implies $$\mathbf{z} = \mathbf{0}$$. The contrapositive implication is that $$\mathbf{a}^\text{T} \mathbf{C} \mathbf{a} > 0$$ for all $$\mathbf{z} \neq 0$$, which establishes that the sample covariance matrix is positive definite. Presumably this is what you are looking for.

• ok, I understand this, but How does the condition $𝑛 - 1> 𝑝$ make the matrix positive definite? – user. Sep 15 '20 at 2:06
• As far as I can see, it doesn't necessarily do this. However, under some additional conditions on $\mathbf{Y}$ you might be able to ensure that $\mathbf{a} \neq \mathbf{0}$. – Ben Sep 15 '20 at 2:45
• There were two other conditions that I forgot, the variables are not linearly related and they are continuous. I placed a clue to solve the proof in the question. – user. Sep 15 '20 at 2:50
• @JavierMariño Ben's answer provides that $S$ is positive semidefinite in general. You've edited the question to specify that $Y$ has linearly independent columns (equiv. $Y$ has rank $p$). By the definition of linear independence, we know that there is no $z\in \mathbb{R}^p \setminus 0$ such that $Yz=0$. This completes Ben's proof. – Sycorax Sep 15 '20 at 2:59
• It's not necessary. It's implied by the linear independence, since you can't have $p$ linearly independent columns if $p\geq n$. Finding a linear dependence involves solving $p$ equations in $n$ unknowns, which you can do if $p\geq n$ – Thomas Lumley Sep 15 '20 at 3:36