2
$\begingroup$

I would like to know whether R produces single degrees of freedom tests for a formula. Assume we have a model in R:

model = lm(x ~ a + b + c, data=mydata)  # augmented model

Is R only doing an omnibus test and comparing my augmented model to a null model with only an intercept:

null=lm(x ~ 1, data=mydata)  # compact model

or is it doing multiple single-degree-of-freedom tests, such as comparing:

model = lm(x ~ a + b + c, data=mydata) # augmented model

to all of the following compact models:

null1=lm(x ~ b + c, data=mydata) # compact model1
null2=lm(x ~ a + c, data=mydata) # compact model2
null3=lm(x ~ a + b, data=mydata) # compact model3

I am worried because I was taught to compare models with only a single degree of freedom between them, ie an augmented model that includes the parameter of interest against a compact model that excludes the parameter of interest. So, if I was interested in the effect of a, then I was taught to compare the augmented model with the compact model as follows:

model = lm(x ~ a + b + c, data=mydata) #augmented model
null = lm(x ~ b + c, data=mydata)      #compact model
anova(null, model)                    # single degree-of-freedom comparison between augmented model and compact model.

But this latter approach isn't very often taught, particularly when one is using R, though Bodo Winter seems to be an exception [EDIT: I've realised that the Winter tutorial is for random effects using lmer(), which does not automatically produce a p value, so this would explain why he teaches the model comparison approach in that context]. Is there any point in doing the comparison?

In other words, if R is doing the first thing (comparing to a null with just an intercept), then I think this is contrary to what I've been taught, but if the latter (comparing to multiple nulls, each test being a single degree of freedom comparison), then I don't think there is a problem.

$\endgroup$
0
$\begingroup$

From your elaboration, it appears you want to know what the F statistic means when you summary(lm(x~a+b+c))?

Basically, it tells you the difference between the model x~a+b+c compared to the intercept only model x~1. That is not the same as comparing full vs reduced models. To compare full vs reduced models you are removing only one variable and seeing if that affects the sums of squares explained by the new model. On this case, anyone of a, b, or c could be having an effect. But the point is just look at the t values and you can see which one is significant--within the model anyway.

Or maybe that's exactly what you want, to just compare your model against the intercept only model, which is what it's doing; but a single significant t test would tell you that the F test would therefore have to be significant anyway.

Just to show you that the F statistic is the same: run the following code (I have generated a bunch of random uniform variables to make a regression, but you can replace it with your own variables and it should be the same)

    > a<-runif(100,1,3); b<-runif(100,1,2); c<-runif(100,1,2);summary(lm(a~b+c))

    Call:
    lm(formula = a ~ b + c)

    Residuals:
     Min      1Q  Median      3Q     Max 
     -0.8875 -0.5368 -0.1291  0.5712  1.0798 

    Coefficients:
                Estimate Std. Error t value Pr(>|t|)    
    (Intercept)  2.09554    0.48939   4.282 4.36e-05 ***
    b           -0.13188    0.21375  -0.617    0.539    
    c           -0.01779    0.22475  -0.079    0.937    
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Residual standard error: 0.5995 on 97 degrees of freedom
    Multiple R-squared:  0.003936,  Adjusted R-squared:  -0.0166 
    F-statistic: 0.1916 on 2 and 97 DF,  p-value: 0.8259

Notice the F statistic and how many degrees of freedom it has

    > anova(lm(a~1),lm(a~b+c))
    Analysis of Variance Table

    Model 1: a ~ 1
    Model 2: a ~ b + c
     Res.Df    RSS Df Sum of Sq      F Pr(>F)
1        99 35.003                           
2        97 34.865  2   0.13776 0.1916 0.8259

Same F, same degrees of freedom.

$\endgroup$
  • $\begingroup$ Dear @Huy Pham, thank you very much, that's fantastic! So what I was taught was right - I should be comparing different models to get the F statistic, so that I know that it is the particular parameter that is reducing the error, rather than doing an omnibus test - that would not tell you which of the parameters was having the effect. $\endgroup$ – Poul Apr 8 '19 at 7:05
  • $\begingroup$ I'm still a little confused as to how R computes the individual t statistics though. Does R effectively run a series of single degree of freedom tests to work these out? $\endgroup$ – Poul Apr 8 '19 at 7:08
  • $\begingroup$ EDIT: this reply is only in response to your second question comment, good to hear the first comment! : No, it calculates them all at once using maximising likelihood. The way you are talking about is called comparing FULL VS REDUCED models--you cana look that phrase up and there'll be more examples because i think your terminology is less used. It's pretty useless, except in very rare specific circumstances, like when you want compare whole models against each other, or compare models with parameters other than betas and see if one fits better. $\endgroup$ – Huy Pham Apr 8 '19 at 9:57
  • $\begingroup$ The individual T statistics are the beta estimates (in the 1st summary) divided by the standard error. Beta estimates are found by maximising likelihoods (which in the case of regressions turns out to be minimising the sums of squares for residuals). Beta estimates are the rate of change from moving from one level of the IV to the next. The beta of the intercept+beta of IV is the mean at that IV. (mean-mean)/SE is a T test if you recall. IF you are more used to the ANOVA way of looking at things, the F stat is simply the T stat squared for the individual pairwise contrasts. $\endgroup$ – Huy Pham Apr 8 '19 at 9:57
4
$\begingroup$

The two approaches (the t and the partial F-with-1-df) are equivalent as long as the t is two-tailed.

There's no additional benefit to the F (it never tells you anything you couldn't get from the t-test), except for the fact that it generalizes - e.g. to testing more than one variable at a time, and to the general linear hypothesis.

$\endgroup$
  • $\begingroup$ Thanks @Glen_b, however, the first approach seems to be an omnibus test, so you couldn't tell whether a, b, or c has the effect. Unless R somehow also does a series of tests with 1 degree of freedom, individually testing a, b, and c? $\endgroup$ – Poul Apr 4 '19 at 13:47
  • 1
    $\begingroup$ Okay, now I'm confused. What "single degree of freedom tests" can you possibly mean? (noting that you now say they're omnibus tests). I had assumed you meant t-tests for individual coefficients -- but those aren't omnibus tests. Please be completely explicit in your question so we can tell what you're asking about $\endgroup$ – Glen_b -Reinstate Monica Apr 4 '19 at 14:38
  • $\begingroup$ Sorry Glen_b, essentially I would like to know whether the first approach only compares the model to a null with only an intercept (an omnibus test), or does it compare multiple models, such as by leaving out one of the terms each time, so it's essentially conducting a series of tests with one degree of freedom? $\endgroup$ – Poul Apr 4 '19 at 16:54
  • $\begingroup$ As I already indicated, I don't know what you're really talking about when you say "the first" -- you present only a formula, not what p-values you're looking at. You need to be explicit. $\endgroup$ – Glen_b -Reinstate Monica Apr 5 '19 at 1:48
  • $\begingroup$ Sorry @Glen_b, I'm trying to be as clear as I can. I know the t test and the f test are equivalent. I want to know what comparisons R is doing when you ask it t o test a model with several parameters. Is it doing an omnibus test (so you can't see which parameter is doing the work) or is it doing several tests, each with a single degree of freedom (so you CAN see which parameter is doing the work)? $\endgroup$ – Poul Apr 5 '19 at 4:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.