Full rank assumption in the linear regression model explanation:

Question

I need some help understanding the full-rank assumption. My book, Econometric Analysis by Greene, presents the following example:

Suppose that a cross-section model specifies that consumption, $C$, relates to income as follows: $$C = \beta_1 + \beta_2X_{non-labor\ income} + \beta_3X_{salary} + \beta_4X_{total\ income} + \epsilon$$ where $X_{total\ income}$is exactly equal to $X_{salary}$ plus $X_{non-labor\ income}$. Clearly, there is an exact linear dependency in the model. Now let $\beta_2' = \beta_2 + a, \beta_3' = \beta_3 + a$, and $\beta_4' = \beta_4 - a$, where a is any number. Then the exact same value appears on the right-hand side of $C$ if we substitute $\beta_2', \beta_3', \beta_4'$ for $\beta_2, \beta_3$, and $\beta_4$. Obviously, there is no way to estimate the parameters of this model.

What I do not understand of the above explanation are the second-to-last and last sentences. What do they mean? What are they trying to convey??

Answer 1

There is an error. It should be $\beta_4'=\beta_4-a$. If we substitute these new $\beta$s into the regression equation we get:

\begin{align} C &= \beta_1 + \beta_2'X_{non-labor-income}+\beta_3'X_{non-labor-income}+\beta_4'X_{total-income}+\varepsilon \\ & = \beta_1 + (\beta_2 +a)X_{non-labor-income}+(\beta_3+a)X_{salary}+(\beta_4-a)X_{total-income}+\varepsilon\\ & = \beta_1 + \beta_2X_{non-labor-income}+\beta_3X_{salary}+\beta_4X_{total- income}+\varepsilon \\ & + a(X_{non-labor-income}+X_{salary}-X_{total-income}) \end{align}

and the last term is zero. So we substituted new coefficients, but the regression did not change, which means that there are multitude of coefficient values which give the same results, whereas the main regression assumption is that the coefficients are uniquely defined.

Answer 2

Old post but since I've been wondering about the same question here's my take on it. I came up a with the following numerical example to help me understand why the design matrix $\textbf{X}$ must be full rank.

First, with a linear model we want to solve (leaving out the error term $\boldsymbol{\epsilon}$):

$$\textbf{y} = \textbf{X} \boldsymbol{\beta}$$

where $\textbf{y}$ is vector of $n$ observation and $\textbf{X}$ is matrix of known covariates with $n$ rows and $p$ columns. We want solve to obtain the vector of $p$ coefficients $\boldsymbol{\beta}$.

Matrix $\textbf{X}$ is a transformation to pass from the vector space $\mathbb{R}^n$ of $\textbf{y}$ to the $\mathbb{R}^p$ space of $\boldsymbol{\beta}$. If $\textbf{X}$ is not full rank we cannot do this mapping uniquely, i.e. there is more than one solution for $\boldsymbol{\beta}$.

For example, with:

$$ \textbf{y} = \begin{bmatrix} 8 \\ 19 \\ 27 \\ 35 \\ \end{bmatrix} \textbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 5 \\ 3 & 7 \\ 4 & 9 \\ \end{bmatrix} $$

$\boldsymbol{X}$ is full rank and the model has a unique solution for $\boldsymbol{\beta} = \boldsymbol{(X^TX)^{-1}X^Ty} = [2 \ \ 3]$.

R code to reproduce this:

y <- c(8, 19, 27, 35)
X <- cbind(c(1, 2, 3, 4), c(2, 5, 7, 9))

qr(X)$rank == ncol(X) # Check X is full rank: TRUE

solve((t(X) %*% X)) %*% t(X) %*% y # Returns [2, 3]

# ---
# Alternatively, use the linear regression machinery:
lm(y ~ 0 + X[,1] + X[,2])

#    Coefficients:
#    X[, 1]  X[, 2]  
#        2       3

Now, consider:

$$ \textbf{y} = \begin{bmatrix} 8 \\ 16 \\ 24 \\ 32 \\ \end{bmatrix} \textbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \\ 4 & 8 \\ \end{bmatrix} $$

$\boldsymbol{X}$ is not full rank (column two is twice column 1, i.e. it's linear combination of column 1) so there are multiple solutions for $\boldsymbol{\beta}$. For example with $\boldsymbol{\beta} = [2 \ \ 3]$, or $[4 \ \ 2]$, or $[6 \ \ 1]$. R code:

y <- c(8, 16, 24, 32)
X <- cbind(c(1, 2, 3, 4), c(2, 4, 6, 8))

qr(X)$rank == ncol(X) # FALSE: not full rank

X %*% c(2, 3) == y # All these solutions return TRUE 
X %*% c(4, 2) == y
X %*% c(6, 1) == y

# Try to solve:
solve((t(X) %*% X)) %*% t(X) %*% y

Error in solve.default((t(X) %*% X)) : 
  Lapack routine dgesv: system is exactly singular: U[2,2] = 0