Return to Answer

The unbiased estimator of the sample covariance matrix given $n$ data points $\newcommand{\x}{\mathbf x}\x_i \in \mathbb R^d$ is $$\mathbf C = \frac{1}{n-1}\sum_{i=1}^n (\x_i - \bar \x)(\x_i - \bar \x)^\top,$$ where $\bar \x = \sum \x_i /n$ is the average over all points. Let's denote $(\x_i-\bar \x)$ as $\newcommand{\z}{\mathbf z}\z_i$. The $\frac{1}{n-1}$ factor does not change the rank, and each term in the sum has (by definition) rank $1$, so the core of the question is as follows:

Why does $\sum \z_i\z_i^\top$ have rank $n-1$ and not rank $n$, as it would seem because we are summing $n$ rank-$1$ matrices?

The answer is that $\z_i$ are not independent. By construction, $\sum\z_i = 0$. So if you know $n-1$ of $\z_i$, then the last remaining $\z_n$ is completely determined; we are not summing $n$ independent rank-$1$ matrices, we are summing only $n-1$ independent rank-$1$ matrices and then adding one more rank-$1$ matrix that is fully linearly determined by the rest. This last addition does not change the overall rank.

We can see this directly if we rewrite $\sum\z_i = 0$ as $$\z_n = -\sum_{i=1}^{n-1}\z_i,$$ and now plug it into the above expression: $$\sum_{i=1}^n \z_i\z_i^\top = \sum_{i=1}^{n-1} \z_i\z_i^\top + \Big(-\sum_{i=1}^{n-1}\z_i\Big)\z_n^\top=\sum_{i=1}^{n-1} \z_i(\z_i-\z_n)^\top.$$ Now there is only $n-1$ terms left in the sum and it's clear that the whole sum can have at most rank $n-1$.

This result, by the way, hints to why the factor in the unbiased estimator of covariance is $\frac{1}{n-1}$ and not $\frac{1}{n}$.

The geometric intuition that I alluded to in the comments above is that one can always fit a 1D line to any two points in 2D and one can always fit a 2D plane to any three points in 3D, i.e. the dimensionality of the subspace is always $n-1$; this only works because we assume that this line (and plane) can be "moved around" in order to fit our points. "Postitioning" this line (or plane) such that it passes through $\bar x$ is equivalent of centering in the algebraic argument above.

The unbiased estimator of the sample covariance matrix given $n$ data points $\newcommand{\x}{\mathbf x}\x_i \in \mathbb R^d$ is $$\mathbf C = \frac{1}{n-1}\sum_{i=1}^n (\x_i - \bar \x)(\x_i - \bar \x)^\top,$$ where $\bar \x = \sum \x_i /n$ is the average over all points. Let us denote $(\x_i-\bar \x)$ as $\newcommand{\z}{\mathbf z}\z_i$. The $\frac{1}{n-1}$ factor does not change the rank, and each term in the sum has (by definition) rank $1$, so the core of the question is as follows:

Why does $\sum \z_i\z_i^\top$ have rank $n-1$ and not rank $n$, as it would seem because we are summing $n$ rank-$1$ matrices?

The answer is that it happens because $\z_i$ are not independent. By construction, $\sum\z_i = 0$. So if you know $n-1$ of $\z_i$, then the last remaining $\z_n$ is completely determined; we are not summing $n$ independent rank-$1$ matrices, we are summing only $n-1$ independent rank-$1$ matrices and then adding one more rank-$1$ matrix that is fully linearly determined by the rest. This last addition does not change the overall rank.

We can see this directly if we rewrite $\sum\z_i = 0$ as $$\z_n = -\sum_{i=1}^{n-1}\z_i,$$ and now plug it into the above expression: $$\sum_{i=1}^n \z_i\z_i^\top = \sum_{i=1}^{n-1} \z_i\z_i^\top + \Big(-\sum_{i=1}^{n-1}\z_i\Big)\z_n^\top=\sum_{i=1}^{n-1} \z_i(\z_i-\z_n)^\top.$$ Now there is only $n-1$ terms left in the sum and it becomes clear that the whole sum can have at most rank $n-1$.

This result, by the way, hints to why the factor in the unbiased estimator of covariance is $\frac{1}{n-1}$ and not $\frac{1}{n}$.

The geometric intuition that I alluded to in the comments above is that one can always fit a 1D line to any two points in 2D and one can always fit a 2D plane to any three points in 3D, i.e. the dimensionality of the subspace is always $n-1$; this only works because we assume that this line (and plane) can be "moved around" in order to fit our points. "Positioning" this line (or plane) such that it passes through $\bar \x$ is equivalent of centering in the algebraic argument above.

The unbiased estimator of the sample covariance matrix given $n$ datapoints $\newcommand{\x}{\mathbf x}\x_i \in \mathbb R^d$ is $$\mathbf C = \frac{1}{n-1}\sum_{i=1}^n (\x_i - \bar \x)^\top(\x_i - \bar \x),$$ where $\bar \x = \sum \x_i /n$ is the average over all points. Let's denote $(\x_i-\bar \x)$ as $\newcommand{\z}{\mathbf z}\z_i$. The $\frac{1}{n-1}$ factor does not change the rank, and each term in the sum has (by definition) rank $1$, so the core of the question is as follows:

Why does $\sum \z_i^\top\z_i$ have rank $n-1$ and not rank $n$, as it would seem because we are summing $n$ rank-$1$ matrices?

The answer is that $\z_i$ are not independent. By construction, $\sum\z_i = 0$. So if you know $n-1$ of $\z_i$, then the last remaining $\z_n$ is completely determined; we are not summing $n$ independent rank-$1$ matrices, we are summing only $n-1$ independent rank-$1$ matrices and then adding one more rank-$1$ matrix that is fully linearly determined by the rest. This last addition does not change the overall rank.

We can see this directly if we rewrite $\sum\z_i = 0$ as $$\z_n = -\sum_{i=1}^{n-1}\z_i,$$ and now plug it into the above expression: $$\sum_{i=1}^n \z_i^\top\z_i = \sum_{i=1}^{n-1} \z_i^\top\z_i + \Big(-\sum_{i=1}^{n-1}\z_i\Big)^\top\z_n=\sum_{i=1}^{n-1} \z_i^\top(\z_i-\z_n).$$ Now there is only $n-1$ terms left in the sum and it's clear that the whole sum can have at most rank $n-1$.

This result, by the way, hints to why the factor in the unbiased estimator of covariance is $\frac{1}{n-1}$ and not $\frac{1}{n}$.

The geometric intuition that I alluded to in the comments above is that one can always fit a 1D line to any two points in 2D and one can always fit a 2D plane to any three points in 3D, i.e. the dimensionality of the subspace is always $n-1$; this only works because we assume that this line (and plane) can be "moved around" in order to fit our points. "Postitioning" this line (or plane) such that it passes through $\bar x$ is equivalent of centering in the algebraic argument above.

The unbiased estimator of the sample covariance matrix given $n$ data points $\newcommand{\x}{\mathbf x}\x_i \in \mathbb R^d$ is $$\mathbf C = \frac{1}{n-1}\sum_{i=1}^n (\x_i - \bar \x)(\x_i - \bar \x)^\top,$$ where $\bar \x = \sum \x_i /n$ is the average over all points. Let's denote $(\x_i-\bar \x)$ as $\newcommand{\z}{\mathbf z}\z_i$. The $\frac{1}{n-1}$ factor does not change the rank, and each term in the sum has (by definition) rank $1$, so the core of the question is as follows:

Why does $\sum \z_i\z_i^\top$ have rank $n-1$ and not rank $n$, as it would seem because we are summing $n$ rank-$1$ matrices?

The answer is that $\z_i$ are not independent. By construction, $\sum\z_i = 0$. So if you know $n-1$ of $\z_i$, then the last remaining $\z_n$ is completely determined; we are not summing $n$ independent rank-$1$ matrices, we are summing only $n-1$ independent rank-$1$ matrices and then adding one more rank-$1$ matrix that is fully linearly determined by the rest. This last addition does not change the overall rank.

We can see this directly if we rewrite $\sum\z_i = 0$ as $$\z_n = -\sum_{i=1}^{n-1}\z_i,$$ and now plug it into the above expression: $$\sum_{i=1}^n \z_i\z_i^\top = \sum_{i=1}^{n-1} \z_i\z_i^\top + \Big(-\sum_{i=1}^{n-1}\z_i\Big)\z_n^\top=\sum_{i=1}^{n-1} \z_i(\z_i-\z_n)^\top.$$ Now there is only $n-1$ terms left in the sum and it's clear that the whole sum can have at most rank $n-1$.

This result, by the way, hints to why the factor in the unbiased estimator of covariance is $\frac{1}{n-1}$ and not $\frac{1}{n}$.

The geometric intuition that I alluded to in the comments above is that one can always fit a 1D line to any two points in 2D and one can always fit a 2D plane to any three points in 3D, i.e. the dimensionality of the subspace is always $n-1$; this only works because we assume that this line (and plane) can be "moved around" in order to fit our points. "Postitioning" this line (or plane) such that it passes through $\bar x$ is equivalent of centering in the algebraic argument above.

The unbiased estimator of the sample covariance matrix given $n$ datapoints $\newcommand{\x}{\mathbf x}\x_i \in \mathbb R^d$ is $$\mathbf C = \frac{1}{n-1}\sum_{i=1}^n (\x_i - \bar \x)^\top(\x_i - \bar \x),$$ where $\bar \x = \sum \x_i /n$ is the average over all points. Let's denote $(\x_i-\bar \x)$ as $\newcommand{\z}{\mathbf z}\z_i$. The $\frac{1}{n-1}$ factor does not change the rank, and each term in the sum has (by definition) rank $1$, so the core of the question is as follows:

Why does $\sum \z_i^\top\z_i$ have rank $n-1$ and not rank $n$, as it would seem because we are summing $n$ rank-$1$ matrices?

The answer is that $\z_i$ are not independent. By construction, $\sum\z_i = 0$. So if you know $n-1$ of $\z_i$, then the last remaining $\z_n$ is completely determined; we are not summing $n$ independent rank-$1$ matrices, we are summing only $n-1$ independent rank-$1$ matrices and then adding one more rank-$1$ matrix that is fully linearly determined by the rest. This last addition does not change the overall rank.

We can see this directly if we rewrite $\sum\z_i = 0$ as $$\z_n = -\sum_{i=1}^{n-1}\z_i,$$ and now plug it into the above expression: $$\sum_{i=1}^n \z_i^\top\z_i = \sum_{i=1}^{n-1} \z_i^\top\z_i + \Big(-\sum_{i=1}^{n-1}\z_i\Big)^\top\z_n=\sum_{i=1}^{n-1} \z_i^\top(\z_i-\z_n).$$ Now there is only $n-1$ terms in the sum and it's clear that it can have at most $n-1$ rank.

The geometric intuition that I alluded to in the comments above is that one can always fit a 1D line to any two points in 2D and one can always fit a 2D plane to any three points in 3D, i.e. the dimensionality of the subspace is always $n-1$; this only works because we assume that this line (and plane) can be "moved around" in order to fit our points. "Postitioning" this line (or plane) such that it passes through $\bar x$ is equivalent of centering in the algebraic argument above.

The unbiased estimator of the sample covariance matrix given $n$ datapoints $\newcommand{\x}{\mathbf x}\x_i \in \mathbb R^d$ is $$\mathbf C = \frac{1}{n-1}\sum_{i=1}^n (\x_i - \bar \x)^\top(\x_i - \bar \x),$$ where $\bar \x = \sum \x_i /n$ is the average over all points. Let's denote $(\x_i-\bar \x)$ as $\newcommand{\z}{\mathbf z}\z_i$. The $\frac{1}{n-1}$ factor does not change the rank, and each term in the sum has (by definition) rank $1$, so the core of the question is as follows:

Why does $\sum \z_i^\top\z_i$ have rank $n-1$ and not rank $n$, as it would seem because we are summing $n$ rank-$1$ matrices?

The answer is that $\z_i$ are not independent. By construction, $\sum\z_i = 0$. So if you know $n-1$ of $\z_i$, then the last remaining $\z_n$ is completely determined; we are not summing $n$ independent rank-$1$ matrices, we are summing only $n-1$ independent rank-$1$ matrices and then adding one more rank-$1$ matrix that is fully linearly determined by the rest. This last addition does not change the overall rank.

We can see this directly if we rewrite $\sum\z_i = 0$ as $$\z_n = -\sum_{i=1}^{n-1}\z_i,$$ and now plug it into the above expression: $$\sum_{i=1}^n \z_i^\top\z_i = \sum_{i=1}^{n-1} \z_i^\top\z_i + \Big(-\sum_{i=1}^{n-1}\z_i\Big)^\top\z_n=\sum_{i=1}^{n-1} \z_i^\top(\z_i-\z_n).$$ Now there is only $n-1$ terms left in the sum and it's clear that the whole sum can have at most rank $n-1$.

This result, by the way, hints to why the factor in the unbiased estimator of covariance is $\frac{1}{n-1}$ and not $\frac{1}{n}$.

The geometric intuition that I alluded to in the comments above is that one can always fit a 1D line to any two points in 2D and one can always fit a 2D plane to any three points in 3D, i.e. the dimensionality of the subspace is always $n-1$; this only works because we assume that this line (and plane) can be "moved around" in order to fit our points. "Postitioning" this line (or plane) such that it passes through $\bar x$ is equivalent of centering in the algebraic argument above.