Full rank assumption in the linear regression model explanation:

I need some help understanding the full-rank assumption. My book, Econometric Analysis by Greene, presents the following example:

Suppose that a cross-section model specifies that consumption, $C$, relates to income as follows: $$C = \beta_1 + \beta_2X_{non-labor\ income} + \beta_3X_{salary} + \beta_4X_{total\ income} + \epsilon$$ where $X_{total\ income}$is exactly equal to $X_{salary}$ plus $X_{non-labor\ income}$. Clearly, there is an exact linear dependency in the model. Now let $\beta_2' = \beta_2 + a, \beta_3' = \beta_3 + a$, and $\beta_4' = \beta_4 - a$, where a is any number. Then the exact same value appears on the right-hand side of $C$ if we substitute $\beta_2', \beta_3', \beta_4'$ for $\beta_2, \beta_3$, and $\beta_4$. Obviously, there is no way to estimate the parameters of this model.

What I do not understand of the above explanation are the second-to-last and last sentences. What do they mean? What are they trying to convey??

• I only really got linear regression when I learned the linear algebra behind it. Basically, when the columns of your matrix X aren't independent, then the matrix $$X^{T}* X$$can't be inverted. In other words, you put that matrix formula in your calculator and you get an error--not even a bad answer, you get no answer. Sep 30, 2015 at 6:00
• your new $\beta' s$ are only old $\beta s$ plus a constant $a$ what are that for? It is not even a random coefficient regression. Sep 30, 2015 at 6:06
• Hey, I just tried to clean up the LaTeX a little--hopefully it will be a little clearer with the names written as subscripts... Sep 30, 2015 at 6:38
• Important: The problem has $\beta_4 - a$! (and not $\beta_4 + a$)--does that clear things up?? Sep 30, 2015 at 6:56
• To see what the author is getting at, try substituting in $(\beta' - a)$ for each $\beta$-value (and $\beta' + a$ for $\beta_{total\ income}$). Then, replace the term $a*X_{total\ income}$ with $a*(X_{non-labor\ income} + X_{salary})$ and notice how when you simplify the equation all the "a" terms cancel out. Sep 30, 2015 at 6:56

There is an error. It should be $\beta_4'=\beta_4-a$. If we substitute these new $\beta$s into the regression equation we get:

\begin{align} C &= \beta_1 + \beta_2'X_{non-labor-income}+\beta_3'X_{non-labor-income}+\beta_4'X_{total-income}+\varepsilon \\ & = \beta_1 + (\beta_2 +a)X_{non-labor-income}+(\beta_3+a)X_{salary}+(\beta_4-a)X_{total-income}+\varepsilon\\ & = \beta_1 + \beta_2X_{non-labor-income}+\beta_3X_{salary}+\beta_4X_{total- income}+\varepsilon \\ & + a(X_{non-labor-income}+X_{salary}-X_{total-income}) \end{align}

and the last term is zero. So we substituted new coefficients, but the regression did not change, which means that there are multitude of coefficient values which give the same results, whereas the main regression assumption is that the coefficients are uniquely defined.

Old post but since I've been wondering about the same question here's my take on it. I came up a with the following numerical example to help me understand why the design matrix $$\textbf{X}$$ must be full rank.

First, with a linear model we want to solve (leaving out the error term $$\boldsymbol{\epsilon}$$):

$$\textbf{y} = \textbf{X} \boldsymbol{\beta}$$

where $$\textbf{y}$$ is vector of $$n$$ observation and $$\textbf{X}$$ is matrix of known covariates with $$n$$ rows and $$p$$ columns. We want solve to obtain the vector of $$p$$ coefficients $$\boldsymbol{\beta}$$.

Matrix $$\textbf{X}$$ is a transformation to pass from the vector space $$\mathbb{R}^n$$ of $$\textbf{y}$$ to the $$\mathbb{R}^p$$ space of $$\boldsymbol{\beta}$$. If $$\textbf{X}$$ is not full rank we cannot do this mapping uniquely, i.e. there is more than one solution for $$\boldsymbol{\beta}$$.

For example, with:

$$\textbf{y} = \begin{bmatrix} 8 \\ 19 \\ 27 \\ 35 \\ \end{bmatrix} \textbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 5 \\ 3 & 7 \\ 4 & 9 \\ \end{bmatrix}$$

$$\boldsymbol{X}$$ is full rank and the model has a unique solution for $$\boldsymbol{\beta} = \boldsymbol{(X^TX)^{-1}X^Ty} = [2 \ \ 3]$$.

R code to reproduce this:

y <- c(8, 19, 27, 35)
X <- cbind(c(1, 2, 3, 4), c(2, 5, 7, 9))

qr(X)$rank == ncol(X) # Check X is full rank: TRUE solve((t(X) %*% X)) %*% t(X) %*% y # Returns [2, 3] # --- # Alternatively, use the linear regression machinery: lm(y ~ 0 + X[,1] + X[,2]) # Coefficients: # X[, 1] X[, 2] # 2 3  Now, consider: $$\textbf{y} = \begin{bmatrix} 8 \\ 16 \\ 24 \\ 32 \\ \end{bmatrix} \textbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 4 \\ 3 & 6 \\ 4 & 8 \\ \end{bmatrix}$$ $$\boldsymbol{X}$$ is not full rank (column two is twice column 1, i.e. it's linear combination of column 1) so there are multiple solutions for $$\boldsymbol{\beta}$$. For example with $$\boldsymbol{\beta} = [2 \ \ 3]$$, or $$[4 \ \ 2]$$, or $$[6 \ \ 1]$$. R code: y <- c(8, 16, 24, 32) X <- cbind(c(1, 2, 3, 4), c(2, 4, 6, 8)) qr(X)$rank == ncol(X) # FALSE: not full rank

X %*% c(2, 3) == y # All these solutions return TRUE
X %*% c(4, 2) == y
X %*% c(6, 1) == y

# Try to solve:
solve((t(X) %*% X)) %*% t(X) %*% y

Error in solve.default((t(X) %*% X)) :
Lapack routine dgesv: system is exactly singular: U[2,2] = 0