In Goodfellow et al.'s Deep Learning, the authors write on page 232:
[$\mathbf{X^\top X}$] can be singular whenever the data-generating distribution truly has no variance in some direction, or when no variance is observed in some direction because there are fewer examples (rows of $\mathbf{X}$) than input features (columns of $\mathbf{X}$).
I understand the first part of this statement: if two predictors are collinear, we only see variance along one dimension (e.g. the points will lie on a line rather than being scattered on a plane).
I'm confused about the second part: mathematically, it makes sense that if there are fewer data points than dimensions, the covariance matrix will be rank deficient and therefore non-invertible. How can one visualize a singular covariance matrix graphically? Does it just mean that we don't know how the data is spread in the all the dimensions, and if so, how do you depict that?
Moreover, I'm trying to understand what "invertible" means graphically. What would inverting a covariance matrix do to the matrix geometrically, and why does this not work for singular covariance matrices?