1
$\begingroup$

I'm newbie to GLM. I'm modeling frequence of accident from driver age, vehicle age, vehicle power... using GLM Poisson I create a training set and a test set. However, I notice for each time I run the model (so the training set changes), the coefficients change (and also the significance level). My question is that: should we change our training set several times or not? If yes, how to deal with the problem above.

I show there 2 examples (2 different training sets):

Call:
glm(formula = TPL ~ DrivAge + DrivGender + MaritalStatus + BonusMalus + 
    LicenceNb + PayFreq + JobCode + VehAge + VehClass + VehPower + 
    VehGas + VehUsage + Garage + Area + Region + Channel + Marketing, 
    family = quasipoisson, data = train)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.7683  -0.4040  -0.3583  -0.3164   4.8324  

Coefficients: (3 not defined because of singularities)
                                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)                      -3.433879   0.804993  -4.266 2.01e-05 ***
DrivAge                           0.008009   0.003139   2.552  0.01073 *  
DrivGenderM                      -0.025324   0.073180  -0.346  0.72931    
MaritalStatusDivorced             0.397871   0.269480   1.476  0.13985    
MaritalStatusMarried             -0.017458   0.084927  -0.206  0.83713    
MaritalStatusSingle               0.280122   0.099957   2.802  0.00508 ** 
MaritalStatusWidowed             -0.384041   0.235483  -1.631  0.10294    
BonusMalus                        0.011595   0.002348   4.939 7.94e-07 ***
LicenceNb                        -0.011739   0.050265  -0.234  0.81535    
PayFreqHalf-yearly                0.010197   0.078224   0.130  0.89629    
PayFreqMonthly                    0.116260   0.162018   0.718  0.47303    
PayFreqQuarterly                  0.025511   0.140871   0.181  0.85629    
JobCodeFarmer                     0.282743   0.294779   0.959  0.33749    
JobCodeOther                     -0.090008   0.233286  -0.386  0.69963    
JobCodePrivate employee           0.050860   0.179362   0.284  0.77675    
JobCodePublic employee            0.094397   0.183370   0.515  0.60671    
JobCodeRetailer                 -12.532778 283.850222  -0.044  0.96478    
JobCodeRetiree                    0.177935   0.217747   0.817  0.41385    
VehAge                           -0.021195   0.007865  -2.695  0.00705 ** 
VehClassCheaper                  -0.033183   0.110429  -0.300  0.76381    
VehClassCheapest                 -0.144738   0.148141  -0.977  0.32857    
VehClassExpensive                 0.224670   0.375376   0.599  0.54950    
VehClassMedium                    0.113419   0.197041   0.576  0.56489    
VehClassMedium high              -0.029006   0.288382  -0.101  0.91988    
VehClassMedium low                0.029851   0.135363   0.221  0.82546    
VehClassMore expensive           -0.115352   0.544292  -0.212  0.83217    
VehClassMost expensive            0.397490   0.632560   0.628  0.52976    
VehPowerP11                       0.138692   0.129379   1.072  0.28375    
VehPowerP12                       0.096941   0.139545   0.695  0.48726    
VehPowerP13                       0.134677   0.166905   0.807  0.41974    
VehPowerP14                       0.249407   0.199613   1.249  0.21152    
VehPowerP15                      -0.030085   0.308138  -0.098  0.92222    
VehPowerP16                       0.393038   0.398141   0.987  0.32357    
VehPowerP17                       0.320942   0.809079   0.397  0.69161    
VehPowerP2                      -11.743427 907.484450  -0.013  0.98968    
VehPowerP4                      -12.058187 573.089763  -0.021  0.98321    
VehPowerP5                       -0.995431   0.594664  -1.674  0.09417 .  
VehPowerP7                        0.344036   0.268571   1.281  0.20022    
VehPowerP8                        0.379639   0.142235   2.669  0.00761 ** 
VehPowerP9                        0.173538   0.133922   1.296  0.19506    
VehGasRegular                    -0.127739   0.078080  -1.636  0.10186    
VehUsageProfessional              0.299550   0.225092   1.331  0.18328    
VehUsageProfessional run          0.576924   0.295204   1.954  0.05068 .  
GarageClosed zbox                 0.213356   0.096891   2.202  0.02768 *  
GarageOpened collective parking   0.047413   0.116148   0.408  0.68312    
GarageStreet                      0.231024   0.122183   1.891  0.05867 .  
AreaA12                          -0.161991   1.237855  -0.131  0.89588    
AreaA2                           -0.500636   0.723129  -0.692  0.48875    
AreaA3                           -0.546334   0.722099  -0.757  0.44931    
AreaA4                           -0.223314   0.725155  -0.308  0.75812    
AreaA5                           -0.352101   0.719116  -0.490  0.62440    
AreaA6                           -0.568578   0.774222  -0.734  0.46273    
AreaA7                           -0.414094   0.727111  -0.570  0.56902    
AreaA8                           -0.375806   0.736787  -0.510  0.61002    
AreaA9                           -0.230268   0.725615  -0.317  0.75099    
RegionHeadquarters                0.155680   0.113223   1.375  0.16916    
RegionParis area                  0.182765   0.122177   1.496  0.13470    
RegionSouth West                 -0.472009   0.586351  -0.805  0.42084    
ChannelB                          0.142026   0.248928   0.571  0.56831    
ChannelL                         -0.139609   0.082359  -1.695  0.09007 .  
MarketingM2                             NA         NA      NA       NA    
MarketingM3                             NA         NA      NA       NA    
MarketingM4                             NA         NA      NA       NA    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for quasipoisson family taken to be 1.012757)

    Null deviance: 5160.1  on 13503  degrees of freedom
Residual deviance: 5037.1  on 13444  degrees of freedom
  (28055 observations deleted due to missingness)
AIC: NA

Number of Fisher Scoring iterations: 13

And:

Call:
glm(formula = TPL ~ DrivAge + DrivGender + MaritalStatus + BonusMalus + 
    LicenceNb + PayFreq + JobCode + VehAge + VehClass + VehPower + 
    VehGas + VehUsage + Garage + Area + Region + Channel + Marketing, 
    family = quasipoisson, data = train)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-0.6924  -0.3969  -0.3545  -0.3144   4.7259  

Coefficients: (3 not defined because of singularities)
                                  Estimate Std. Error t value Pr(>|t|)    
(Intercept)                      -3.758964   0.805871  -4.664 3.12e-06 ***
DrivAge                           0.008306   0.003166   2.623  0.00871 ** 
DrivGenderM                      -0.059227   0.072369  -0.818  0.41314    
MaritalStatusDivorced             0.237533   0.286279   0.830  0.40671    
MaritalStatusMarried              0.069704   0.083210   0.838  0.40222    
MaritalStatusSingle               0.249467   0.101559   2.456  0.01405 *  
MaritalStatusWidowed             -0.110189   0.204761  -0.538  0.59049    
BonusMalus                        0.009858   0.002356   4.183 2.89e-05 ***
LicenceNb                        -0.006905   0.050557  -0.137  0.89137    
PayFreqHalf-yearly                0.027229   0.078711   0.346  0.72940    
PayFreqMonthly                    0.198096   0.157291   1.259  0.20790    
PayFreqQuarterly                 -0.020713   0.145221  -0.143  0.88658    
JobCodeFarmer                     0.565334   0.295674   1.912  0.05590 .  
JobCodeOther                      0.213664   0.238004   0.898  0.36934    
JobCodePrivate employee           0.210541   0.191649   1.099  0.27197    
JobCodePublic employee            0.331952   0.194939   1.703  0.08862 .  
JobCodeRetailer                   1.000891   0.612073   1.635  0.10202    
JobCodeRetiree                    0.366832   0.227306   1.614  0.10659    
VehAge                           -0.014329   0.007741  -1.851  0.06420 .  
VehClassCheaper                  -0.013083   0.111105  -0.118  0.90627    
VehClassCheapest                 -0.016907   0.149586  -0.113  0.91001    
VehClassExpensive                 0.041037   0.390048   0.105  0.91621    
VehClassMedium                    0.177040   0.186507   0.949  0.34252    
VehClassMedium high              -0.436590   0.310360  -1.407  0.15953    
VehClassMedium low               -0.068154   0.137396  -0.496  0.61987    
VehClassMore expensive           -0.810244   0.688764  -1.176  0.23947    
VehClassMost expensive            0.060609   0.704527   0.086  0.93145    
VehPowerP11                       0.152053   0.130469   1.165  0.24387    
VehPowerP12                       0.148002   0.140920   1.050  0.29362    
VehPowerP13                       0.196015   0.168605   1.163  0.24502    
VehPowerP14                       0.489279   0.198917   2.460  0.01392 *  
VehPowerP15                       0.210457   0.297820   0.707  0.47979    
VehPowerP16                       0.535802   0.422429   1.268  0.20468    
VehPowerP17                      -0.089318   1.062724  -0.084  0.93302    
VehPowerP2                      -10.935041 345.147440  -0.032  0.97473    
VehPowerP4                      -11.316674 344.768434  -0.033  0.97382    
VehPowerP5                       -0.888187   0.513750  -1.729  0.08386 .  
VehPowerP7                        0.291672   0.260037   1.122  0.26203    
VehPowerP8                        0.344021   0.139878   2.459  0.01393 *  
VehPowerP9                        0.094111   0.133048   0.707  0.47937    
VehGasRegular                    -0.094493   0.077936  -1.212  0.22537    
VehUsageProfessional              0.380818   0.218267   1.745  0.08105 .  
VehUsageProfessional run          0.565450   0.305062   1.854  0.06382 .  
GarageClosed zbox                 0.181757   0.096416   1.885  0.05943 .  
GarageOpened collective parking   0.095442   0.113857   0.838  0.40190    
GarageStreet                      0.247142   0.119960   2.060  0.03940 *  
AreaA12                          -0.032918   1.228840  -0.027  0.97863    
AreaA2                           -0.458926   0.717346  -0.640  0.52234    
AreaA3                           -0.451137   0.716336  -0.630  0.52885    
AreaA4                           -0.243743   0.720159  -0.338  0.73502    
AreaA5                           -0.277866   0.713694  -0.389  0.69703    
AreaA6                           -0.517108   0.775208  -0.667  0.50475    
AreaA7                           -0.302781   0.724004  -0.418  0.67581    
AreaA8                           -0.048111   0.731303  -0.066  0.94755    
AreaA9                           -0.102986   0.722589  -0.143  0.88667    
RegionHeadquarters                0.082624   0.114706   0.720  0.47135    
RegionParis area                  0.173735   0.122637   1.417  0.15660    
RegionSouth West                 -0.838719   0.709602  -1.182  0.23724    
ChannelB                          0.262125   0.244031   1.074  0.28278    
ChannelL                         -0.109997   0.081158  -1.355  0.17533    
MarketingM2                             NA         NA      NA       NA    
MarketingM3                             NA         NA      NA       NA    
MarketingM4                             NA         NA      NA       NA    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for quasipoisson family taken to be 0.9951371)

    Null deviance: 5123.8  on 13616  degrees of freedom
Residual deviance: 5008.5  on 13557  degrees of freedom
  (27942 observations deleted due to missingness)
AIC: NA

Number of Fisher Scoring iterations: 12
$\endgroup$
1
$\begingroup$

I'd suggest you to invest some more time in your models. Here are some issues that might be partly responsible for the unstable results:

1) Throwing away 2/3 of rows due to missing values is very bad practice. Try to retain these lines.

2) Why investing just one parameter in variables like DrivAge but many more for variables like area or power? The decision on how many parameters to use should not be based on the variable type of the data.

3) Why no interactions?

4) Three regressors are perfectly correlated with the others. That should not be happen.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.