4
$\begingroup$

Consider the $p \times p$ sample covariance matrix:

$$\mathbf{S} = \frac{1}{n-1} \cdot \mathbf{Y}_\mathbf{c}^\text{T} \mathbf{Y}_\mathbf{c} \quad \quad \quad \mathbf{Y}_\mathbf{c} = \mathbf{C} \mathbf{Y},$$

where $\mathbf{C} = \mathbf{I}-\frac{1}{n} \mathbf{1} \mathbf{1}^\text{T}$ is the $n \times n$ centering matrix and $\mathbf{Y}$ is an $n \times p$ matrix. How can it be proved that if the variables are continuos, not linearly related and $n-1> p$ then the sample covariance matrix is ​​positive definite?

The following clue is found in Ranchera's book:

enter image description here

Improve this question
$\endgroup$
7
  • $\begingroup$ No, this isn't my question. $\endgroup$
    – user.
    Commented Sep 15, 2020 at 1:58
  • 2
    $\begingroup$ You've asked how to prove that a sample covariance matrix is positive definite. The answers provide proofs that the sample covariance matrix is positive semi-definite. How is this not an answer to your question? $\endgroup$
    – Sycorax ♦
    Commented Sep 15, 2020 at 2:00
  • 1
    $\begingroup$ I want to know how the condition $𝑛 - 1> 𝑝$ makes the matrix positive definite. I know covariance matrix is positive semi-definite. $\endgroup$
    – user.
    Commented Sep 15, 2020 at 2:12
  • 2
    $\begingroup$ It's possible that the misunderstanding arises because the claim is false. $S$ is positive semi-definite whenever the columns of $Y$ are linearly dependent -- for example, if each row of $Y$ sums to the same number. This circumstance is addressed in the duplicate. $\endgroup$
    – Sycorax ♦
    Commented Sep 15, 2020 at 2:16
  • $\begingroup$ Your question about $n-1\gt p$ is asked and answered at stats.stackexchange.com/questions/60622. $\endgroup$
    – whuber ♦
    Commented Sep 15, 2020 at 12:51

1 Answer 1

Reset to default
5
$\begingroup$

First, let's simplify the equation for your sample covariance matrix. Using the fact that the centering matrix is symmetric and idempotent you get the $p \times p$ form:

$$\begin{align} \mathbf{S} &= \frac{1}{n-1} \cdot \mathbf{Y}_\mathbf{c}^\text{T} \mathbf{Y}_\mathbf{c} \\[6pt] &= \frac{1}{n-1} \cdot (\mathbf{C} \mathbf{Y})^\text{T} (\mathbf{C} \mathbf{Y}) \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{Y}^\text{T} \mathbf{C}^\text{T} \mathbf{C} \mathbf{Y} \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{Y}^\text{T} \mathbf{C} \mathbf{Y}. \\[6pt] \end{align}$$

This is a simple quadratic form in $\mathbf{Y}$. I will show that this matrix is non-negative definite (or "positive semi-definite" if you prefer) but it is not always positive definite. To do this, consider an arbitrary non-zero column vector $\mathbf{z} \in \mathbb{R}^p - \{ \mathbf{0} \}$ and let $\mathbf{a} = \mathbf{Y} \mathbf{z} \in \mathbb{R}^n$ be the resulting column vector. Since the centering matrix is non-negative definite (it has one eigenvalue equal to zero and the rest are equal to one) you have:

$$\begin{align} \mathbf{z}^\text{T} \mathbf{S} \mathbf{z} &= \frac{1}{n-1} \cdot \mathbf{z}^\text{T} \mathbf{Y}^\text{T} \mathbf{C} \mathbf{Y} \mathbf{z} \\[6pt] &= \frac{1}{n-1} \cdot (\mathbf{Y} \mathbf{z})^\text{T} \mathbf{C} \mathbf{Y} \mathbf{z} \\[6pt] &= \frac{1}{n-1} \cdot \mathbf{a}^\text{T} \mathbf{C} \mathbf{a} \geqslant 0. \\[6pt] \end{align}$$

This shows that $\mathbf{S}$ is non-negative definite. However, it is not always positive definite. To see this, take any $\mathbf{z} \neq \mathbf{0}$ giving $\mathbf{a} = \mathbf{Y} \mathbf{z} \propto \mathbf{1}$ and substitute into the quadratic form to get $\mathbf{z}^\text{T} \mathbf{S} \mathbf{z} = 0$.

Update: This update is based on the additional information you have added in your edit to the question and your comments. In order to get a positive definite sample variance matrix you need $\mathbf{a}^\text{T} \mathbf{C} \mathbf{a} > 0$. If $n-1>p$ and all $n$ rows of $\mathbf{Y}$ are linearly independent then $\mathbf{Y} \mathbf{z} \propto \mathbf{1}$ implies $\mathbf{z} = \mathbf{0}$. The contrapositive implication is that $\mathbf{a}^\text{T} \mathbf{C} \mathbf{a} > 0$ for all $\mathbf{z} \neq 0$, which establishes that the sample covariance matrix is positive definite. Presumably this is what you are looking for.

Improve this answer
$\endgroup$
8
  • $\begingroup$ ok, I understand this, but How does the condition $𝑛 - 1> 𝑝$ make the matrix positive definite? $\endgroup$
    – user.
    Commented Sep 15, 2020 at 2:06
  • $\begingroup$ As far as I can see, it doesn't necessarily do this. However, under some additional conditions on $\mathbf{Y}$ you might be able to ensure that $\mathbf{a} \neq \mathbf{0}$. $\endgroup$
    – Ben
    Commented Sep 15, 2020 at 2:45
  • $\begingroup$ There were two other conditions that I forgot, the variables are not linearly related and they are continuous. I placed a clue to solve the proof in the question. $\endgroup$
    – user.
    Commented Sep 15, 2020 at 2:50
  • $\begingroup$ @JavierMariño Ben's answer provides that $S$ is positive semidefinite in general. You've edited the question to specify that $Y$ has linearly independent columns (equiv. $Y$ has rank $p$). By the definition of linear independence, we know that there is no $z\in \mathbb{R}^p \setminus 0$ such that $Yz=0$. This completes Ben's proof. $\endgroup$
    – Sycorax ♦
    Commented Sep 15, 2020 at 2:59
  • 1
    $\begingroup$ It's not necessary. It's implied by the linear independence, since you can't have $p$ linearly independent columns if $p\geq n$. Finding a linear dependence involves solving $p$ equations in $n$ unknowns, which you can do if $p\geq n$ $\endgroup$
    – Thomas Lumley
    Commented Sep 15, 2020 at 3:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.