0
$\begingroup$

I have run a logistic regression after variable subsetting using LASSO in glmnet( glmnet.cv,100 fold cross validation for $\lambda\ min estimation) but model summary is as below:

Call:
glm(formula = expression_logit_binom_mdl_1, family = binomial(logit), 
    data = model_data)

Deviance Residuals: 
       Min          1Q      Median          3Q         Max  
-2.6566592  -0.9787929   0.5198135   0.9386671   3.9041465  

Coefficients: (2 not defined because of singularities)
                                           Estimate                 Std. Error   z value               Pr(>|z|)    
(Intercept)                       -32.1356866731516            3.4194372490387  -9.39795 < 0.000000000000000222 ***
Eastings                            0.0000623039832            0.0000038474432  16.19361 < 0.000000000000000222 ***
Northings                           0.0000016375636            0.0000008817602   1.85715             0.06328938 .  
Agri_Fema                       59525.0373485190721        55482.8711864717407   1.07285             0.28333649    
Agri_Male                      -11669.9226195544579         5787.4071968829885  -2.01643             0.04375465 *  
aster_30m_aspect                    0.0005566141057            0.0000520968203  10.68422 < 0.000000000000000222 ***
aster_30m_slope                     0.0059387270303            0.0005644049763  10.52210 < 0.000000000000000222 ***
ASTER_DEM_30m                       0.0066396418545            0.0002960719856  22.42577 < 0.000000000000000222 ***
AVG_SIZE_HH                        -1.0042435928826            0.3005582277152  -3.34126             0.00083399 ***
Beat_office_dst                     0.0000525679521            0.0000024935951  21.08119 < 0.000000000000000222 ***
BF_Oct_201_dst                     -0.0000160199166            0.0000037366561  -4.28723   0.000018091223503917 ***
Blocks_line_dst                     0.0001113032815            0.0000200785437   5.54339   0.000000029666407139 ***
CHT_Agricultur_dst                 -0.0010768819943            0.0000238456270 -45.16057 < 0.000000000000000222 ***
CHT_Settlemen_dst                   0.0001731184848            0.0000098952668  17.49508 < 0.000000000000000222 ***
DNTW_F                          66934.5046721813414         3504.3132424908772  19.10061 < 0.000000000000000222 ***
DNTW_M                          58461.4234597916802         3793.1699486042680  15.41229 < 0.000000000000000222 ***
Eastwardnes_dst                    -0.0000258434505            0.0000034067044  -7.58606   0.000000000000032979 ***
ELECTRICITY_CONNECTION             -0.0884108359228            0.0063949965097 -13.82500 < 0.000000000000000222 ***
Empled_F                       -90741.5910881050804        55925.0856079266741  -1.62256             0.10468432    
Empled_M                       -28019.6867687456070         4799.2411915049570  -5.83836   0.000000005271788054 ***
FEMALE_POP_Density       -17288186636.9353904724121   3198700732.3671927452087  -5.40475   0.000000064897844970 ***
GLG_av_dst                         -0.0000460144615            0.0000028788448 -15.98365 < 0.000000000000000222 ***
GLG_cs_dst                          0.0000374175971            0.0000020058627  18.65412 < 0.000000000000000222 ***
GLG_QTd_dst                         0.0000137273219            0.0000028460499   4.82329   0.000001412100124083 ***
GLG_QT_dst                          0.0000535969536            0.0000032706943  16.38703 < 0.000000000000000222 ***
GLG_Tb_dst                          0.0000064335339            0.0000022537868   2.85454             0.00430986 ** 
GLG_T_dst                           0.0000129552797            0.0000009036316  14.33690 < 0.000000000000000222 ***
GrowthCenter_dst                    0.0000633482579            0.0000032754507  19.34032 < 0.000000000000000222 ***
H0_4                                3.4415405933093            0.8420485679638   4.08710   0.000043679017997170 ***
H10_14                              3.3466109099697            0.8474295606144   3.94913   0.000078435203114894 ***
H15_19                              3.8056634660638            0.8393599778209   4.53401   0.000005787528890741 ***
H20_24                              4.2099787316225            0.8307369053324   5.06776   0.000000402515642009 ***
H25_29                              3.9772799890839            0.8525604380154   4.66510   0.000003084664805557 ***
H30_49                              3.5995699141873            0.8369450462281   4.30084   0.000017014896680267 ***
H50_59                              2.9188273412766            0.8334553935371   3.50208             0.00046164 ***
H5_9                                4.1123480110458            0.8332525132073   4.93530   0.000000800291857122 ***
H60_64                              2.4085303493605            0.8632932337245   2.78993             0.00527189 ** 
H65                                 3.9813662853575            0.8500919185198   4.68345   0.000002820809596916 ***
HH_Density                     -69180.8369553794619        11690.6207158202960  -5.91764   0.000000003266019549 ***
HH_F                            66934.4596642923425         3504.3125579536800  19.10060 < 0.000000000000000222 ***
HH_M                            58461.4193739516631         3793.1688699018760  15.41229 < 0.000000000000000222 ***
HT_OWNED                           -5.4608945780607            1.5196272776268  -3.59357             0.00032617 ***
HT_RENTED                          -5.5532572493708            1.5171863600408  -3.66023             0.00025198 ***
HT_RENT_FREE                       -5.4603795914887            1.5200582501753  -3.59222             0.00032788 ***
Indstry_F                       59525.0301969924039        55482.8714780337614   1.07285             0.28333655    
Indstry_M                      -11669.9212021074090         5787.4073513157891  -2.01643             0.04375468 *  
LITERACY_RATE_BOTH                 -0.9272351392832            0.3334173834136  -2.78100             0.00541910 ** 
LITERACY_RATE_FEMALE                0.6555647064214            0.1629491524990   4.02312   0.000057431149169749 ***
LITERACY_RATE_MALE                  0.4127581573265            0.1730673141709   2.38496             0.01708110 *  
LITERATE_BOTH_YES               -1551.1446317942173         1146.2131327413126  -1.35328             0.17596694    
LITERATE_FEMALE_YES              1551.1520899004947         1146.2130198312602   1.35328             0.17596482    
LITERATE_MALE_YES                1551.1347550583953         1146.2131720394727   1.35327             0.17596971    
LknJB_F                         66934.5416586664214         3504.3156039965802  19.10060 < 0.000000000000000222 ***
LknJB_M                         58461.4240233603778         3793.1702170626863  15.41229 < 0.000000000000000222 ***
MALE_POP_Density         -17288174348.2915115356445   3198694677.2836837768555  -5.40476   0.000000064895531709 ***
Ntscl7_B                       -11308.5793378026319         1750.9535711728306  -6.45853   0.000000000105727631 ***
Ntscl7_F                       -55625.8852490456629         3222.4131486291899 -17.26218 < 0.000000000000000222 ***
Ntscl7_M                       -47152.8521907492395         3499.8215155937669 -13.47293 < 0.000000000000000222 ***
NtShl_7_B                       98151.0431035905203         6051.8804664108693  16.21827 < 0.000000000000000222 ***
NtShl_7_F                                        NA                         NA        NA                     NA    
NtShl_7_M                                        NA                         NA        NA                     NA    
POP_DENSITY               17288192006.8518753051758   3198698500.4909262657166   5.40476   0.000000064895871824 ***
Pourashava_cityCor_dst              0.0000834546741            0.0000022088843  37.78137 < 0.000000000000000222 ***
Roads_nationa_dst                  -0.0000395028971            0.0000027157471 -14.54587 < 0.000000000000000222 ***
Roads_regional_feede_dst           -0.0000566110721            0.0000027713904 -20.42696 < 0.000000000000000222 ***
Roads_regiona_dst                  -0.0000402535647            0.0000033501616 -12.01541 < 0.000000000000000222 ***
Service_F                       59524.9918790496813        55482.8716042225569   1.07285             0.28333686    
Service_M                      -11669.9277198423060         5787.4071055766972  -2.01643             0.04375455 *  
SEX_RATIO                          -0.1073882834614            0.0069930281094 -15.35648 < 0.000000000000000222 ***
Small_Hat_Bazar_dst                 0.0000967299711            0.0000037116779  26.06098 < 0.000000000000000222 ***
STRUCTURE_JHUPRI                    1.9118364584795            1.3887541242794   1.37666             0.16861868    
STRUCTURE_KUTCHA                    1.9262823338151            1.3892127198644   1.38660             0.16556377    
STRUCTURE_PUCKA                     1.5407728963395            1.3861487013134   1.11155             0.26633190    
STRUCTURE_SEMI_PUCKA                2.2255109545882            1.3880169917684   1.60337             0.10885201    
Upazilla_lin_dst                    0.0000670825793            0.0000050933297  13.17067 < 0.000000000000000222 ***
Zilla_lin_dst                       0.0000488043007            0.0000034479423  14.15462 < 0.000000000000000222 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 245860.02  on 177385  degrees of freedom
Residual deviance: 203410.02  on 177312  degrees of freedom
AIC: 203558.02

Number of Fisher Scoring iterations: 6

More background of my model can be found below links:

here and here

Now my question is why some coefficients are so varied?


Update

After using vifstep from usdm package I collected only 40 from 70+ variables. Afterward I ran LASSO with again and found coefficient as below:

41 x 1 sparse Matrix of class "dgCMatrix"
                                          1
(Intercept)              -1.550453712088479
Agri_Fema                 0.022839833392058
Agri_Male                -0.001644509136187
aster_30m_aspect          0.000512895998527
aster_30m_slope           0.005733959652405
ASTER_DEM_30m             0.006493690977654
Beat_office_dst           0.000044844700473
BF_Oct_201_dst           -0.000017637409224
Blocks_line_dst           0.000069773041947
CHT_Agricultur_dst       -0.001125024039569
CHT_Settlemen_dst         0.000069580040999
DNTW_M                    0.011432546528745
Eastwardnes_dst           0.000010904529097
ELECTRICITY_CONNECTION   -0.081578215424478
GLG_av_dst                0.000005532344286
GLG_cs_dst                0.000008260268395
GLG_QTd_dst              -0.000011097490408
GLG_Tb_dst               -0.000020101684698
GrowthCenter_dst          0.000064357864646
H20_24                    0.182427325999767
H65                       0.459375819924272
HH_M                     -0.008006633888449
HT_RENTED                -0.186416737714288
HT_RENT_FREE             -0.060092989358567
Indstry_F                 0.036439882944596
Indstry_M                -0.002359706219711
LITERATE_MALE_YES        -0.001379936201357
LknJB_F                   0.026845017905640
LknJB_M                   0.009395604488612
Pourashava_cityCor_dst    0.000063851429584
Roads_nationa_dst        -0.000040879721599
Roads_regional_feede_dst -0.000028409960055
Roads_regiona_dst        -0.000074577418293
Service_F                -0.014833316029951
Service_M                -0.007457890434070
Small_Hat_Bazar_dst       0.000111206754901
STRUCTURE_JHUPRI         -0.024295995035423
STRUCTURE_PUCKA          -0.141639844445959
STRUCTURE_SEMI_PUCKA      0.176215393388445
Upazilla_lin_dst          0.000067759282061
Zilla_lin_dst            -0.000017633887672

Then I even ran Ridge and Elastic net along with LASSO and compared their coefficient in the below graph: comparision

$\endgroup$
  • 1
    $\begingroup$ Why did you run the logistic regression after the lasso? Why not just use the lasso estimated coefficients? $\endgroup$ – Matthew Drury Apr 1 '17 at 8:33
  • $\begingroup$ I used LASSO for variable selection. The selected variable will be input to logit. Is not it a established practice of variable selection using LASSO. I read somewhere that LASSO does not do anything with COLLINEARITY. So i used vifstep of usdm package to remove all columns have score>10 the I ran cv.glmnet and fittd(glmnet) but this also give me another willdy shoot. Is not this the right way I am going along esp for the collinearity checking. could you help? I am new in this area. $\endgroup$ – SIslam Apr 1 '17 at 9:05
0
$\begingroup$

This is almost always caused by near-collinearity between some of your variables. Given the names of the variables, they do look potentially co-linear. If you care about prediction quality rather than interpreting coefficients, you can add in a regulariser which will damp the coefficients significantly. A $L_2$ regulariser (ridge regression), or even use the lasso again but with a smaller regularization constant.

If some of your features are only active for very few instances, you can also see large weights. This is quite easy to check.

$\endgroup$
  • $\begingroup$ Using logistic regression I need to order explanatory variables based on their effect(coefficient), i.e. which factor is the most culprit for deforestation and which is less etc.Which variables seem to have near collinearity from mere name , could you help?Could you come into chat if you need more please? $\endgroup$ – SIslam Apr 1 '17 at 9:04
  • $\begingroup$ There is some advice here, I would also check for rarely active features by taking the column-wise mean and checking if the problematic variables have very small values compared to the rest. $\endgroup$ – AaronDefazio Apr 1 '17 at 11:30
  • $\begingroup$ In terms of the problematic features, it looks like POP_DENSITY, MALE_POP_Density, and FEMALE_POP_Density are highly collinear. Based off of the names, I'm guessing that POP_DENSITY=MALE_POP_Density+FEMALE_POP_Density. $\endgroup$ – AaronDefazio Apr 1 '17 at 11:36
  • $\begingroup$ Updated the question. See is it OK. Now how can I test if lasso coefficients are correct?Another note, I just divided each parcels population by area to standardise the variable e.g. POP of x parcel is transformed into (POP of x)/(Area of x in Hactres) is it ok $\endgroup$ – SIslam Apr 1 '17 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.