I am running some linear regression models in R using the lm function and calling the summary() function on my linear model gives a nice summary of the degrees of freedom, F-statistic, sum of squares, etc.

lm(formula = y ~ a + b + c + d, data = my_data)

       Min         1Q     Median         3Q        Max 
-0.0313491 -0.0034180 -0.0000105  0.0030041  0.0306064 

                 Estimate Std. Error t value Pr(>|t|)    
(Intercept)     1.004e-04  3.396e-04   0.362    0.717    
a              -2.739e-02  0.893e-02  -1.447    0.149    
b               5.019      5.127e-01  19.249   <2e-16 ***
c               7.090      2.954e-01  31.171   <2e-16 ***
d               4.241      2.760e-01  22.978   <2e-16 ***
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.006236 on 354 degrees of freedom
  (1 observation deleted due to missingness)
Multiple R-squared:  0.881, Adjusted R-squared:  0.8797

Now, this gives p-values / significance tests based on the null hypothesis that: $$H_0: \mu_1 = \mu_2 =\mu_3 =\mu_4 = 0$$

Therefore, since each p-value is statistically significant at $\alpha=0.05$, we can reject the null hypothesis and conclude there is a linear relationship between each variable and the response variable.

Now I am interested in determining if one variable is statistically different from another variable, i.e. that it my model can be simply explained by just $b$ or just $b$ and $c$, etc. (I guess this is the same as determining if a two variables are co-linear but I am uncertain.)

What tools can I use clarify this and find the most significant model?

  • $\begingroup$ I think that you want to do an analysis of covariance. In R try, aov(formula = y ~ a + b + c + d, data = my_data) $\endgroup$ – Brad S. Dec 4 '17 at 21:20
  • $\begingroup$ @BradS. I did this but I don't know how to interpret the results when I feed it 2 models for comparison...I searched around for a detailed explanation but I found nothing $\endgroup$ – guy Dec 4 '17 at 21:23
  • $\begingroup$ Note that comparing models with Anova requires them to be nested within one another. $\endgroup$ – Yuval Spiegler Dec 4 '17 at 21:28
  • 1
    $\begingroup$ The p-values reported in this output do not correspond to the (single) hypothesis you quote. They correspond to four distinct hypotheses: $\mu_1=0$, $\mu_2=0$, etc. These four p-values do not tell you whether you can reject the null hypothesis of no linear relationship: that is the purpose of the F test (which you eliminated from the bottom of the output). $\endgroup$ – whuber Dec 4 '17 at 21:43
  • $\begingroup$ @whuber So the individual p-values from the t-tests are whether each individual variable is significant and the final F-statistic's p-value tests the null that all coefficients are zero? $\endgroup$ – guy Dec 4 '17 at 22:02

You are looking at comparing models, not comparing covariates really. If you have the models in mind, that is, you want to compare model $a$ to model $b$, you can use AIC, which estimates the fit of a model while taking its complexity into account: aic(model_a,model_b) and choose the lower outcome AIC model.

Another method, is using the adjusted $R^2$ of all the models you want and compare them. Generally, the model with the smallest amount of covariates and the highest adjusted $R^2$ is your winner. If you want to check all covariate combinations instead of manually choosing specific models to compare, you can use an all subsets regression from library(leaps) in R. This will allow you to choose the number of best combinations to show, which you can also plot() for a visual aid. If this is what you need, see this great tutorial from R-studio pubs.

It might be worth noting that choosing the fittest model should be done with care. I come from social sciences, and the way I see it, sometimes it is worth adding non-significant covariates to the model - it they have a meaning. As a simple example, if I check for predictors of salary, and gender is non-significant, than perhaps some other covariate made it non-significant. If so - that is of interest. If not, than perhaps there is no gender pay-gap - also interesting. My point is Choose a model with care, not just by looking at the numbers - they do not always convey the entirety of the picture.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.