Under-constrained models and invertibility of covariance matrix

Question

In Goodfellow et al.'s Deep Learning, the authors write on page 232:

[$\mathbf{X^\top X}$] can be singular whenever the data-generating distribution truly has no variance in some direction, or when no variance is observed in some direction because there are fewer examples (rows of $\mathbf{X}$) than input features (columns of $\mathbf{X}$).

I understand the first part of this statement: if two predictors are collinear, we only see variance along one dimension (e.g. the points will lie on a line rather than being scattered on a plane).

I'm confused about the second part: mathematically, it makes sense that if there are fewer data points than dimensions, the covariance matrix will be rank deficient and therefore non-invertible. How can one visualize a singular covariance matrix graphically? Does it just mean that we don't know how the data is spread in the all the dimensions, and if so, how do you depict that?

Moreover, I'm trying to understand what "invertible" means graphically. What would inverting a covariance matrix do to the matrix geometrically, and why does this not work for singular covariance matrices?

Answer 1

As an example imagine your dimension is 3 but you only have two points. So $X$ looks like:

$$X=\begin{pmatrix}a & b & c\\d & e & f\end{pmatrix}.$$

Then $X^TX$ is not invertible. This is because $X$ has rank 2: notice that you have 3 columns of dimension 2, which means at least two of the columns are linearly dependent. So there exists a $y$ such that $Xy=0$, which necessarily imples that $X^TX$ isn't invertible.

Non-invertibility of covariance matrices is very rare in general. Most empirical observations are continuous, so the chance of collinearity is almost 0, except possibly when certain variables are perfectly correlated, or when your $X_{ij}$ are coming from a discrete random variable. As well, you rarely have thin $X$, where the number of data points is less than the number of features.