4
$\begingroup$

I have two models:

frm.mE <- glm(frm ~ age + education + socialrole + countedmembers +
            offset(log(words)), family=quasipoisson, data=daten.alle.kom)
frm.oE <- glm(frm ~ age + socialrole + countedmembers +
                 offset(log(words)), family=quasipoisson, data=daten.alle.kom)

now I want to know which model is the better one, but because of quasipoisson, AIC don't work

summary(frm.mE)
Call:
glm(formula = frm ~ age + education + socialrole + countedmembers + 
offset(log(words)), family = quasipoisson, data = daten.alle.kom)

Deviance Residuals: 
Min       1Q   Median       3Q      Max  
-6.7040  -1.6727  -0.2329   1.0003   7.4897  

Coefficients:
           Estimate Std. Error t value Pr(>|t|)    
(Intercept)    -3.95362    0.21432 -18.448  < 2e-16 ***
age             0.01293    0.07041   0.184  0.85454    
education1      0.11532    0.11647   0.990  0.32367    
socialrole1    -0.28367    0.23685  -1.198  0.23287    
socialrole2    -0.80474    0.29054  -2.770  0.00629 ** 
countedmembers -0.03716    0.06120  -0.607  0.54461    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for quasipoisson family taken to be 5.792638)

Null deviance: 909.51  on 160  degrees of freedom
Residual deviance: 841.35  on 155  degrees of freedom
AIC: NA

Number of Fisher Scoring iterations: 5

and the second model:

Call:
glm(formula = frm ~ age + socialrole + countedmembers + offset(log(words)), 
family = quasipoisson, data = daten.alle.kom)

Deviance Residuals: 
Min       1Q   Median       3Q      Max  
-6.4844  -1.6613  -0.3583   1.1036   7.1557  

Coefficients:
           Estimate Std. Error t value Pr(>|t|)    
(Intercept)    -3.89079    0.20350 -19.119  < 2e-16 ***
age             0.00540    0.06966   0.078  0.93832    
socialrole1    -0.33991    0.22947  -1.481  0.14054    
socialrole2    -0.75470    0.28553  -2.643  0.00905 ** 
countedmembers -0.02634    0.05996  -0.439  0.66104    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for quasipoisson family taken to be 5.761264)

Null deviance: 909.51  on 160  degrees of freedom
Residual deviance: 847.08  on 156  degrees of freedom
AIC: NA

Number of Fisher Scoring iterations: 5

is there another way to compare them? or to know if I should keep the variable "education"? thanks for any help!

I tried a F test, but not sure if it makes sense:

 anova(frm.mE, frm.oE, test="F")
Analysis of Deviance Table

Model 1: frm ~ age + education + socialrole + countedmembers + offset(log(words))
Model 2: frm ~ age + socialrole + countedmembers + offset(log(words))
Resid. Df Resid. Dev Df Deviance      F Pr(>F)
1       155     841.35                          
2       156     847.08 -1  -5.7368 0.9904 0.3212

but I'm not sure how to understand it, does it mean that I should keep "education" because model 2 has a too big p-value?

$\endgroup$
  • 1
    $\begingroup$ Aren't your models perfectly nested? Why not use a standard test? $\endgroup$ – gung - Reinstate Monica Oct 26 '14 at 22:45
5
$\begingroup$

No, it means that the model including "education" fails to reduce the error in your model by a significant degree. The F statistic is simply looking at the reduction in error between Model A and Model B. If your F statistic were larger (and p-value were smaller) that would mean "education" was worth having in your model.

$F = \dfrac{\frac{SSE(Model_C) - SSE(Model_A)}{PA - PC}}{\frac{SSE(Model_A)}{n - PA}}$

where $Model_C$ is the model with fewer predictors (frm.oE), $Model_A$ is the augmented model (frm.mE), $PA$ is the number of parameters in $Model_A$, $PC$ is the number of parameters in $Model_C$, and $n$ is the size of your sample. $SSE()$ refers to the sum of squared errors of a model.

Essentially, $F$ is the ratio of the reduction in error between models relative to the number of additional parameters to the error of the new model relative to the remaining degrees of freedom. In other words, does adding 'education' to your model decrease the error enough to justify adding it to your model. If your $F$ value is larger than the critical $F$ value for the number of new parameters and remaining degrees of freedom (the critical value can be obtained with qf() if you're curious), we say that we are justified in adding the new predictor (so $p$ would be below .05). This is not the case in the example you provided.

$\endgroup$
  • $\begingroup$ thanks! could you explain a bit what you mean with "F statistiik larger" and "p value smaller"? do you mean smaller than 0.05? and is there another more suitable test for this? $\endgroup$ – user2373707 Oct 27 '14 at 18:05
  • $\begingroup$ @user2373707 yes, although in general it's not a great idea to rely on p<0.05. See, eg, the second paragraph of stat.columbia.edu/~gelman/research/unpublished/murtaugh2.pdf $\endgroup$ – shadowtalker Oct 29 '14 at 2:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.