4
$\begingroup$

Suppose I have a matrix of observations $X \in \mathbb{R}^{n \times p}$. So I have $n$ observations that each have dimension $p$. Lets suppose also that we have more data points than dimensions to them, so $n > p$. Is there an intuitive interpretation for the matrix to have full rank?

$\endgroup$
2
  • $\begingroup$ Given the model is $Y = X\beta + \varepsilon$, $\mathrm{rank}(X) = p$ is a necessary and sufficient condition for the parameter $\beta$ is estimable, or the model is identifiable. If $\mathrm{rank}(X) < p$, then there would be infinitely many $\hat{\beta}$ can be served as the least-square estimate, which is not desirable from the (classical) inference perspective. $\endgroup$ – Zhanxiong Mar 28 at 14:31
  • 2
    $\begingroup$ Another interpretation: if X has full rank, then the least squares loss function is strictly convex and looks like a bowl. If X has rank that is one lower, the loss function looks like a taco. Each rank deficiency is another independent direction where the loss function is flat. $\endgroup$ – bjb568 Mar 28 at 21:48
10
$\begingroup$

If the matrix has full rank, i.e. $rank(M) = p$ and $n > p$, the p variables are linearly independent and therefore there is no redundancy in the data. If instead the $rank(M) < p$ some columns can be recreated by linearly combining the others. In this latter case, you couldn't use all the columns of M as explanatory variables in a linear model (you could of course but it wouldn't make sense).

For example, in R:

library(Matrix)
set.seed(1234)
options(digits= 3)

M <- matrix(data= rnorm(n= 15), ncol= 3)
M
       [,1]   [,2]    [,3]
[1,] -1.207  0.506 -0.4772
[2,]  0.277 -0.575 -0.9984
[3,]  1.084 -0.547 -0.7763
[4,] -2.346 -0.564  0.0645
[5,]  0.429 -0.890  0.9595

rankMatrix(M) # -> rank 3

# 4th column is a linear combination of column 1 and 2 - there is redundancy
M2 <- cbind(M, M[,1] + M[,2])
M2
       [,1]   [,2]    [,3]   [,4]
[1,] -1.207  0.506 -0.4772 -0.701
[2,]  0.277 -0.575 -0.9984 -0.297
[3,]  1.084 -0.547 -0.7763  0.538
[4,] -2.346 -0.564  0.0645 -2.910
[5,]  0.429 -0.890  0.9595 -0.461

rankMatrix(M2) # still rank 3 even if you have four columns

Try to fit a linear model on y using M or M2 as explanatory variables.


# A dummy response variable
y <- M[,1] + rnorm(n= nrow(M))

# Ok, the coefficients of all the  variables in M can be estimated 
lm(y ~ M)

# Coefficients:
# (Intercept)           M1           M2           M3  
#     -0.6786       1.0682      -0.0178      -0.7026  


# Here the coefficient of 4th variable cannot be estimated no
# matter how many observations (rows) you have
lm(y ~ M2)

# Coefficients:
# (Intercept)          M21          M22          M23          M24  
#     -0.6786       1.0682      -0.0178      -0.7026           NA  

I guess that a geometric interpretation of the linear model example is that the response matrix M defines the space where the vector of coefficient can be fitted. More variables in M means more dimensions available. However, if a variable in M is a linear combination of other variables, like in M2, there is no real increase in additional dimensions (or space available)


As noted by @SingleMalt, a matrix can be full rank and still be redundant for practical purposes. For example, if you add some minimal jitter to column 4 of M2 you obtain full rank (4). You could use M2 as predictor in a linear model and estimate all 4 coefficients but the results would be very unstable and unreliable. As shown here the estimated coefficient have "gone crazy":

# Add some minimal jitter to column 4
M2[,4] <- M2[,4] + rnorm(n= nrow(M2), sd= 0.0001)

rankMatrix(M2) # <- 4 Full rank

lm(y ~ M2)

# Coefficients:
# (Intercept)          M21          M22          M23          M24  
#      -0.267    36603.840    36604.443        1.318   -36603.504

Returning to the geometric interpretation, this means that column 4 does add a fourth dimension available to the vector of coefficients. But the increase in space is so small that some dimensions almost perfectly overlap and cannot be estimated with any reliable accuracy.


That's my interpretation - hope it helps...

$\endgroup$
2
  • 1
    $\begingroup$ This is a clear answer. The Matrix package documentation says of function rankMatrix: “Compute ‘the’ matrix rank, a well-defined functional in theory, somewhat ambiguous in practice”. Thus a small amount of jitter when creating M2 would generally yield a rank of four (although I have not verified this in terms of this particular R function). $\endgroup$ – Single Malt Mar 27 at 21:30
  • $\begingroup$ @SingleMalt I think you are right - I edited my answer to include this practical consideration. $\endgroup$ – dariober Mar 28 at 8:30
3
$\begingroup$

Suppose you have a $10×10$ matrix $X$, and $rank(X) = 1$, which is stored somewhere on your memory disk. For some reason, your memory disk was damaged, and so was the matrix. Some rows remained intact, whereas in other rows several digits were lost. The question is, how many full rows do you need to know in order to restore the entire matrix?

The minimum number of rows you need is equal to the rank of the matrix. This is due to the fact that rank represents the number of linearly independent rows. All other rows can be obtained by multiplying that one row by an arbitrary number.

E.g., if our damaged matrix looks like this: $\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\ & 6 & & 12 & & & & & & \\ & & & & 20 & 24 & & & & \\ 7 & & & & & & & & & 70 \\ & & & & \dots\ & & & & \\ \end{bmatrix}$

Then, we already know that our matrix is: $\begin{bmatrix} 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8 & 9 & 10 \\ 3 & 6 & 9 & 12 & 15 & 18 & 21 & 24 & 27 & 30 \\ 4 & 8 & 12 & 16 & 20 & 24 & 28 & 32 & 36 & 40 \\ 7 & 14 & 21 & 28 & 35 & 42 & 49 & 56 & 63 & 70 \\ & & & & \dots\ & & & & \\ \end{bmatrix}$

Were $rank(X) = 2$, that woudn't work because we would need at least two full rows. Finally, if matrix $X$ had full rank (i.e. $rank(X) = 10$), we would be in a world of trouble. Since all rows are linearly independent, not a single lost digit could be restored.

This is one way to look at the concept of matrix rank.

$\endgroup$
1
  • $\begingroup$ Thank you, that is quite a good answer for what it generally means for a matrix to not have full rank. I have a mathematics background so this interpretation I was aware of. My question was more in regards to what it tells me about the observations when I have a data matrix for modelling specifically $\endgroup$ – lpnorm Mar 28 at 8:55
2
$\begingroup$

I want to connect the concept of identifiability with the rank of the design matrix in linear regression, as well as take a more linear algebraic look at the problem, since you mention you have a math background.

Some parameter in our regression model is called identifiable if it's possible for us to even guess what it is. To contrast this with the typical case, we'll never know the exact value of any parameter if there's any randomness in the process, but the least squares estimates can give a good guess if there's enough good data. But when a parameter is not identified, it's absolutely impossible to have any idea what it is: no matter what value the parameter is set to, the data that's coming out will look the exact same. And since we use the data to guess what the parameters are, if there's no influence on the data from a certain parameter, the data are not providing us any information about it.

A good example is the situation where we have categorical factors, for instance: we want to compare medical treatment Treatment A to Treatment B to Treatment C:

enter image description here

We've also compared the patient's weights. The treatment outcomes ("y's") are not shown; they don't play into identifiability at all.

Here's the problem: Treatment A and C were always given to the same people. We can tell because their columns have exactly the same data in them. Inuitively, we already know we're in trouble: if those patients recover more than usual, it could be the case that Treatment A is doing all the work, and Treatment C is inert. Or maybe Treatment A is actually harmfull, but Treatment C is super beneficial and makes up for it. There are infinite possibilities, and we can't be sure what's going on with Treatment A and C individually.

A more subtle version of this is the general unidentifiable case. To see the general case, let's now bring rank into the picture. A key consequence of Rank from linear algebra is that a matrix with less than full rank turns some set of vectors that aren't zero into the zero vector: $\mathbf{X}\mathbf{b} = \mathbf{0}$ (these vectors are said to belong to the kernel or nullspace of $\mathbf{x}$).

In the case of the matrix above, one such vector is $c(1,0,-1,0)$:

enter image description here

This is the mathematical interpretation of our above discussion: we can't know how much better Treatment A is than Treatment C, since a 1 in the first entry and a -1 in the third entry is going to compute the difference between the first and third rows upon matrix multiplication.

But we haven't talked about any of our regression coefficients: we don't have a coefficient on what the different is between Treatments A and C. We have one individually on Treatments A and C. So we do we care if the difference is not identified? The problem is that we haven't actually cleared anything from unidentifiability yet.

This is because of the projection theorem from linear algebra, which states that any vector $\mathbf{v}$ in a linear space $\mathcal{L}$ can be decomposed along some subspace $\mathcal{S}$ as $\mathbf{v} = \mathbf{v}_{\mathcal{S}} + \mathbf{v}_{\mathcal{S}^\perp}$, where $\mathbf{v}_{\mathcal{S}} \in \mathcal{S}$ and $\mathbf{v}_{\mathcal{S}^\perp}$ is orthogonal to that space. So thinking of input parameters as unit vectors, they can be decomposed as $\mathbf{e}_i = \mathbf{v}_{\mathcal{N}} + \mathbf{v}_{\mathcal{N}^\perp} $, where $\mathbf{v}_{\mathcal{N}}$ is in the nullspace of $\mathbf{X}$ and $\mathbf{v}_{\mathcal{N}^\perp}$ is orthogonal to that.

With this decomposition, we see that it's not just $c(1,0,-1,0)$ that we can't estimate: whenever the component $\mathbf{v}_{\mathcal{N}}$ is not the zero vector, it can't be estimated. On the contrary, any vector which is orthogonal to the vector $c(1,0,-1,0)$, or in general, orthogonal to the nullspace, will be identifiable. Recall from Gil Strang's fundamental theorem of linear algebra that the subspace orthogonal to matrix's kernel is that same matrix's rowspace. Thus the rowspace of our design matrix can still be estimated even if our matrix is less than full rank. Intuitively, we should have been able to estimate the effect of "Treatment A and Treatment B", and we can verify that their sum is indeed in the rowspace and hence identifiable.

TLDR: If your matrix is less than full rank, it will send some vector to zero. This means that changing regression coefficients along that direction results in no change to the data, and we can't learn anything about what that linear combination of our parameters are, and thus what the parameters that contributed are themselves.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.