I'm doing a linear regression, in R. The values are like this -

u  <-  c(1,2,3,4,5,6,7,8,9,10)
v <- c(21,22,23,24,25,26,27,28,29,30)
w <- c(41,42,43,44,45,46,47,48,49,50)
y <- c(128.2305,132.4040,140.1732,147.3236, 154.5410, 158.7206, 165.1761, 169.7121,178.9751,181.0309)

If I call linear regression function, it's returning a model, which is disregarding v and w.

model <- lm(y~u+v+w)

(Intercept)            u            v            w   
    122.074        6.101           NA           NA  



lm(formula = y ~ u + v + w)

       Min       1Q   Median       3Q      Max 
-2.05143 -0.92734  0.04845  0.73362  1.99357 

Coefficients: (2 not defined because of singularities)
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 122.0743     1.0197  119.72 2.65e-14 ***
u             6.1008     0.1643   37.12 3.04e-10 ***
v                 NA         NA      NA       NA    
w                 NA         NA      NA       NA    
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.493 on 8 degrees of freedom
Multiple R-squared:  0.9942,    Adjusted R-squared:  0.9935 
F-statistic:  1378 on 1 and 8 DF,  p-value: 3.04e-10

I tried to fit a linear model before with different values of y,u,v (with two predictor variables, w was absent), and there also, v was being assigned NA, and only u was getting co-efficients. What's happening?

  • $\begingroup$ Try summary(model) command. It may be useful to post its output here. For reproducible code in R, use proper commands like u = c(1,2,...) etc. $\endgroup$
    – rnso
    Apr 12 '15 at 7:34

Two of your three regressors ($v$ and $w$, say) are linear combinations of the intercept column (the "1"-vector) and the third regressor ($u$, say), thus causing "perfect multicollinearity". For instance, the vector $v$ equals $u$ plus 20.

Technically speaking, the design matrix $X$ with rows $$ X_{i,.} = (1, u_i, v_i, w_i) $$ is not of full column rank, so the square matrix $(X'X)$ appearing in the least-squares solution $\hat \beta$ $$ \hat \beta =(X'X)^{-1}X'y $$ cannot be inverted.

Fortunately, R is so smart that it still provides the correct solution by dropping two of the three linearly dependent regressors. So it uses the design matrix $\tilde X$ with rows

$$ \tilde X_i = (1, u_i) $$ with the corresponding solution in your output.

  • $\begingroup$ okay. So you mean to say, the variables are like v = ku+e, or w = ku+e, so it'll be a case for multi-collinearity e is some error or intercept? $\endgroup$ Apr 12 '15 at 8:07
  • $\begingroup$ no, thats not what I meant. There is no "e" in my text. $v = u + 20\cdot 1$ e.g. $\endgroup$
    – Michael M
    Apr 12 '15 at 8:09
  • $\begingroup$ Thanks so much... :) Okay. This one.. v(1) = u(1)+20; v(2) = u(2)+20; $\endgroup$ Apr 12 '15 at 8:13
  • $\begingroup$ Exactly, that is right. $\endgroup$
    – Michael M
    Apr 12 '15 at 8:43
  • $\begingroup$ I liked the clarity with which you answered. Could you please suggest a good discussion on regression, with math concepts, preferably using R.. $\endgroup$ Apr 12 '15 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.