What does it mean when a Data Matrix has full rank? Suppose I have a matrix of observations $X \in \mathbb{R}^{n \times p}$. So I have $n$ observations that each have dimension $p$. Lets suppose also that we have more data points than dimensions to them, so $n > p$. Is there an intuitive interpretation for the matrix to have full rank?
 A: I want to connect the concept of identifiability with the rank of the design matrix in linear regression, as well as take a more linear algebraic look at the problem, since you mention you have a math background.
Some parameter in our regression model is called identifiable if it's possible for us to even guess what it is. To contrast this with the typical case, we'll never know the exact value of any parameter if there's any randomness in the process, but the least squares estimates can give a good guess if there's enough good data. But when a parameter is not identified, it's absolutely impossible to have any idea what it is: no matter what value the parameter is set to, the data that's coming out will look the exact same. And since we use the data to guess what the parameters are, if there's no influence on the data from a certain parameter, the data are not providing us any information about it.
A good example is the situation where we have categorical factors, for instance:
we want to compare medical treatment Treatment A to Treatment B to Treatment C:

We've also compared the patient's weights. The treatment outcomes ("y's") are not shown; they don't play into identifiability at all.
Here's the problem: Treatment A and C were always given to the same people. We can tell because their columns have exactly the same data in them. Inuitively, we already know we're in trouble: if those patients recover more than usual, it could be the case that Treatment A is doing all the work, and Treatment C is inert. Or maybe Treatment A is actually harmfull, but Treatment C is super beneficial and makes up for it. There are infinite possibilities, and we can't be sure what's going on with Treatment A and C individually.
A more subtle version of this is the general unidentifiable case. To see the general case, let's now bring rank into the picture. A key consequence of Rank from linear algebra is that a matrix with less than full rank turns some set of vectors that aren't zero into the zero vector: $\mathbf{X}\mathbf{b} =  \mathbf{0}$ (these vectors are said to belong to the kernel or nullspace of $\mathbf{x}$).
In the case of the matrix above, one such vector is $c(1,0,-1,0)$:

This is the mathematical interpretation of our above discussion: we can't know how much better Treatment A is than Treatment C, since a 1 in the first entry and a -1 in the third entry is going to compute the difference between the first and third rows upon matrix multiplication.
But we haven't talked about any of our regression coefficients: we don't have a coefficient on what the different is between Treatments A and C. We have one individually on Treatments A and C. So why do we care if the difference is not identified? The problem is that we haven't actually cleared anything from unidentifiability yet.
This is because of the projection theorem from linear algebra, which states that any vector $\mathbf{v}$ in a linear space $\mathcal{L}$ can be decomposed along some subspace $\mathcal{S}$ as $\mathbf{v} = \mathbf{v}_{\mathcal{S}} + \mathbf{v}_{\mathcal{S}^\perp}$, where $\mathbf{v}_{\mathcal{S}} \in \mathcal{S}$ and $\mathbf{v}_{\mathcal{S}^\perp}$ is orthogonal to that space. So thinking of input parameters as unit vectors, they can be decomposed as $\mathbf{e}_i = \mathbf{v}_{\mathcal{N}} + \mathbf{v}_{\mathcal{N}^\perp} $, where $\mathbf{v}_{\mathcal{N}}$ is in the nullspace of $\mathbf{X}$ and $\mathbf{v}_{\mathcal{N}^\perp}$ is orthogonal to that.
With this decomposition, we see that it's not just $c(1,0,-1,0)$ that we can't estimate: whenever the component $\mathbf{v}_{\mathcal{N}}$ is not the zero vector, it can't be estimated. On the contrary, any vector which is orthogonal to the vector $c(1,0,-1,0)$, or in general, orthogonal to the nullspace, will be identifiable. Recall from Gil Strang's fundamental theorem of linear algebra that the subspace orthogonal to matrix's kernel is that same matrix's rowspace. Thus the rowspace of our design matrix can still be estimated even if our matrix is less than full rank. Intuitively, we should have been able to estimate the effect of "Treatment A and Treatment B", and we can verify that their sum is indeed in the rowspace and hence identifiable.
TLDR: If your matrix is less than full rank, it will send some vector to zero. This means that changing regression coefficients along that direction results in no change to the data, and we can't learn anything about what that linear combination of our parameters are, and thus what the parameters that contributed are themselves.
A: Suppose you have a $10×10$ matrix $X$, and $rank(X) = 1$, which is stored somewhere on your memory disk. For some reason, your memory disk was damaged, and so was the matrix. Some rows remained intact, whereas in other rows several digits were lost. The question is, how many full rows do you need to know in order to restore the entire matrix?
The minimum number of rows you need is equal to the rank of the matrix. This is due to the fact that rank represents the number of linearly independent rows. All other rows can be obtained by multiplying that one row by an arbitrary number.
E.g., if our damaged matrix looks like this:
$\begin{bmatrix}
1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\
 & 6 & & 12 & & & & & & \\
 & & & & 20 & 24 & & & & \\
7 & & & & & & & & & 70 \\
& & & & \dots\ & & & & \\ 
\end{bmatrix}$
Then, we already know that our matrix is:
$\begin{bmatrix}
1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\
3 & 6 & 9 & 12 & 15 & 18 & 21 & 24 & 27 & 30 \\
4 & 8 & 12 & 16 & 20 & 24 & 28 & 32 & 36 & 40 \\
7 & 14 & 21 & 28 & 35 & 42 & 49 & 56 & 63 & 70 \\
& & & & \dots\ & & & & \\ 
\end{bmatrix}$
Were $rank(X) = 2$, that woudn't work because we would need at least two full rows. Finally, if matrix $X$ had full rank (i.e. $rank(X) = 10$), we would be in a world of trouble. Since all rows are linearly independent, not a single lost digit could be restored.
This is one way to look at the concept of matrix rank.
