1
$\begingroup$

I have this linear model:

fit = glm(Survived ~ Pclass + Sex + Age + SibSp + Parch + Fare, 
          data=passengers, family=binomial)
summary(fit) 

Coefficients:
         Estimate Std. Error z value Pr(>|z|)    
(Intercept)  5.318162   0.571693   9.302  < 2e-16 ***
Pclass      -1.175648   0.145979  -8.054 8.04e-16 ***
Sexmale     -2.760823   0.199952 -13.807  < 2e-16 ***
Age         -0.043866   0.008220  -5.336 9.49e-08 ***
SibSp       -0.428252   0.106963  -4.004 6.24e-05 ***
Parch       -0.099051   0.118328  -0.837    0.403    
Fare         0.002587   0.002362   1.095    0.274

So according to the Z score I can say that the decision variables Pclass, Sexmale, Age, SibSp deal a huge part in deciding whether the depending variable will be 0 or 1.

Now I change the fit to the following (added Title):

fit = glm(Survived ~ Pclass + Sex + Age + SibSp + Parch + Fare + Title, 
          data=passengers, family=binomial)
summary(fit)

Coefficients:
          Estimate Std. Error z value Pr(>|z|)    
(Intercept)  3.458e+01  1.906e+03   0.018  0.98552    
Pclass      -1.108e+00  1.556e-01  -7.123 1.06e-12 ***
Sexmale     -3.180e+01  1.906e+03  -0.017  0.98669    
Age         -3.110e-02  9.574e-03  -3.249  0.00116 ** 
SibSp       -6.079e-01  1.250e-01  -4.862 1.16e-06 ***
Parch       -3.614e-01  1.346e-01  -2.685  0.00726 ** 
Fare         4.168e-03  2.573e-03   1.620  0.10531    
TitleDr     -6.257e-01  1.691e+00  -0.370  0.71142    
TitleLady   -1.605e+01  1.452e+03  -0.011  0.99118    
TitleMaster  2.541e+00  1.552e+00   1.637  0.10163    
TitleMiss   -2.982e+01  1.906e+03  -0.016  0.98752    
TitleMlle   -1.642e+01  2.356e+03  -0.007  0.99444    
TitleMr     -9.856e-01  1.439e+00  -0.685  0.49339    
TitleMrs    -2.904e+01  1.906e+03  -0.015  0.98784    
TitleMs     -1.498e+01  3.064e+03  -0.005  0.99610    
TitleRev    -1.576e+01  9.616e+02  -0.016  0.98692    
TitleSir    -4.003e-01  1.705e+00  -0.235  0.81438

So now my model is much "weaker" in a sense that I have more weak decision variables and the Sexmale is also "overshadowed" with the categorical Title variable.

Could someone tell me the reason? Also, please help me deciding if the new model is really weaker than the previous one or am I missing some fundamental thing here?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Why don't you run cross validation and see what model gives better prediction? $\endgroup$
    – ffriend
    Commented Aug 28, 2014 at 22:42
  • $\begingroup$ I will but testing the model comes after fitting it and I want to understand why are the scores so different? Or maybe they don't really matter after all? $\endgroup$
    – SLOBY
    Commented Aug 28, 2014 at 22:45
  • 2
    $\begingroup$ As you stated yourself, variables are not really independent and some of them "shadow" others. Sex itself is a strong predictor, but in presence of more fine-grained Title* variables it loses some of its importance. However, many "weaker" variables may give better results than few "stronger". That's why I proposed to evaluate models first. In general, if variables are highly correlated, coefficients may not really reflect importance of features. $\endgroup$
    – ffriend
    Commented Aug 28, 2014 at 23:03
  • $\begingroup$ above is a good point. but the use of "weaker" is also incorrect, or rather I don't understand the logic behind its use here. You've provided no measures of model performance, how are you comparing models? $\endgroup$
    – charles
    Commented Aug 28, 2014 at 23:06
  • $\begingroup$ Ok, so based on the above 2 models without measuring, what exactly can we infer? $\endgroup$
    – SLOBY
    Commented Aug 28, 2014 at 23:08

1 Answer 1

Reset to default
4
$\begingroup$

This is a version of a frequently asked question. The answer is that Title is correlated with the other variables that are already in the model. As a result, the estimates for the variables in the model can change when Title is added. The basic story is discussed in many places on CV, but if you want a generic introduction, it may help you to read my answer here: Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you, the answer you linked was very helpful. The indicator for correlation is the fact that adding Title in the model greatly affected other variable's z score rigth? Still I'm confused about the z score then. Should I care about it? What should indicate in my case that I might get a better fit by using the second fit? $\endgroup$
    – SLOBY
    Commented Aug 29, 2014 at 8:35
  • $\begingroup$ If you want to know if the fit is better, take a look at measures of fit for both models. A common measure is the area under the ROC curve (AUC). If you are wondering about the models' out of sample performance, you can cross-validate to get estimates of out of sample AUCs. I wouldn't worry too much about the z-scores in either case. $\endgroup$
    – gung - Reinstate Monica
    Commented Aug 29, 2014 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.