# Logistic regression coefficients are wildly

I have run a logistic regression after variable subsetting using LASSO in glmnet( glmnet.cv,100 fold cross validation for $\lambda\ min estimation) but model summary is as below: Call: glm(formula = expression_logit_binom_mdl_1, family = binomial(logit), data = model_data) Deviance Residuals: Min 1Q Median 3Q Max -2.6566592 -0.9787929 0.5198135 0.9386671 3.9041465 Coefficients: (2 not defined because of singularities) Estimate Std. Error z value Pr(>|z|) (Intercept) -32.1356866731516 3.4194372490387 -9.39795 < 0.000000000000000222 *** Eastings 0.0000623039832 0.0000038474432 16.19361 < 0.000000000000000222 *** Northings 0.0000016375636 0.0000008817602 1.85715 0.06328938 . Agri_Fema 59525.0373485190721 55482.8711864717407 1.07285 0.28333649 Agri_Male -11669.9226195544579 5787.4071968829885 -2.01643 0.04375465 * aster_30m_aspect 0.0005566141057 0.0000520968203 10.68422 < 0.000000000000000222 *** aster_30m_slope 0.0059387270303 0.0005644049763 10.52210 < 0.000000000000000222 *** ASTER_DEM_30m 0.0066396418545 0.0002960719856 22.42577 < 0.000000000000000222 *** AVG_SIZE_HH -1.0042435928826 0.3005582277152 -3.34126 0.00083399 *** Beat_office_dst 0.0000525679521 0.0000024935951 21.08119 < 0.000000000000000222 *** BF_Oct_201_dst -0.0000160199166 0.0000037366561 -4.28723 0.000018091223503917 *** Blocks_line_dst 0.0001113032815 0.0000200785437 5.54339 0.000000029666407139 *** CHT_Agricultur_dst -0.0010768819943 0.0000238456270 -45.16057 < 0.000000000000000222 *** CHT_Settlemen_dst 0.0001731184848 0.0000098952668 17.49508 < 0.000000000000000222 *** DNTW_F 66934.5046721813414 3504.3132424908772 19.10061 < 0.000000000000000222 *** DNTW_M 58461.4234597916802 3793.1699486042680 15.41229 < 0.000000000000000222 *** Eastwardnes_dst -0.0000258434505 0.0000034067044 -7.58606 0.000000000000032979 *** ELECTRICITY_CONNECTION -0.0884108359228 0.0063949965097 -13.82500 < 0.000000000000000222 *** Empled_F -90741.5910881050804 55925.0856079266741 -1.62256 0.10468432 Empled_M -28019.6867687456070 4799.2411915049570 -5.83836 0.000000005271788054 *** FEMALE_POP_Density -17288186636.9353904724121 3198700732.3671927452087 -5.40475 0.000000064897844970 *** GLG_av_dst -0.0000460144615 0.0000028788448 -15.98365 < 0.000000000000000222 *** GLG_cs_dst 0.0000374175971 0.0000020058627 18.65412 < 0.000000000000000222 *** GLG_QTd_dst 0.0000137273219 0.0000028460499 4.82329 0.000001412100124083 *** GLG_QT_dst 0.0000535969536 0.0000032706943 16.38703 < 0.000000000000000222 *** GLG_Tb_dst 0.0000064335339 0.0000022537868 2.85454 0.00430986 ** GLG_T_dst 0.0000129552797 0.0000009036316 14.33690 < 0.000000000000000222 *** GrowthCenter_dst 0.0000633482579 0.0000032754507 19.34032 < 0.000000000000000222 *** H0_4 3.4415405933093 0.8420485679638 4.08710 0.000043679017997170 *** H10_14 3.3466109099697 0.8474295606144 3.94913 0.000078435203114894 *** H15_19 3.8056634660638 0.8393599778209 4.53401 0.000005787528890741 *** H20_24 4.2099787316225 0.8307369053324 5.06776 0.000000402515642009 *** H25_29 3.9772799890839 0.8525604380154 4.66510 0.000003084664805557 *** H30_49 3.5995699141873 0.8369450462281 4.30084 0.000017014896680267 *** H50_59 2.9188273412766 0.8334553935371 3.50208 0.00046164 *** H5_9 4.1123480110458 0.8332525132073 4.93530 0.000000800291857122 *** H60_64 2.4085303493605 0.8632932337245 2.78993 0.00527189 ** H65 3.9813662853575 0.8500919185198 4.68345 0.000002820809596916 *** HH_Density -69180.8369553794619 11690.6207158202960 -5.91764 0.000000003266019549 *** HH_F 66934.4596642923425 3504.3125579536800 19.10060 < 0.000000000000000222 *** HH_M 58461.4193739516631 3793.1688699018760 15.41229 < 0.000000000000000222 *** HT_OWNED -5.4608945780607 1.5196272776268 -3.59357 0.00032617 *** HT_RENTED -5.5532572493708 1.5171863600408 -3.66023 0.00025198 *** HT_RENT_FREE -5.4603795914887 1.5200582501753 -3.59222 0.00032788 *** Indstry_F 59525.0301969924039 55482.8714780337614 1.07285 0.28333655 Indstry_M -11669.9212021074090 5787.4073513157891 -2.01643 0.04375468 * LITERACY_RATE_BOTH -0.9272351392832 0.3334173834136 -2.78100 0.00541910 ** LITERACY_RATE_FEMALE 0.6555647064214 0.1629491524990 4.02312 0.000057431149169749 *** LITERACY_RATE_MALE 0.4127581573265 0.1730673141709 2.38496 0.01708110 * LITERATE_BOTH_YES -1551.1446317942173 1146.2131327413126 -1.35328 0.17596694 LITERATE_FEMALE_YES 1551.1520899004947 1146.2130198312602 1.35328 0.17596482 LITERATE_MALE_YES 1551.1347550583953 1146.2131720394727 1.35327 0.17596971 LknJB_F 66934.5416586664214 3504.3156039965802 19.10060 < 0.000000000000000222 *** LknJB_M 58461.4240233603778 3793.1702170626863 15.41229 < 0.000000000000000222 *** MALE_POP_Density -17288174348.2915115356445 3198694677.2836837768555 -5.40476 0.000000064895531709 *** Ntscl7_B -11308.5793378026319 1750.9535711728306 -6.45853 0.000000000105727631 *** Ntscl7_F -55625.8852490456629 3222.4131486291899 -17.26218 < 0.000000000000000222 *** Ntscl7_M -47152.8521907492395 3499.8215155937669 -13.47293 < 0.000000000000000222 *** NtShl_7_B 98151.0431035905203 6051.8804664108693 16.21827 < 0.000000000000000222 *** NtShl_7_F NA NA NA NA NtShl_7_M NA NA NA NA POP_DENSITY 17288192006.8518753051758 3198698500.4909262657166 5.40476 0.000000064895871824 *** Pourashava_cityCor_dst 0.0000834546741 0.0000022088843 37.78137 < 0.000000000000000222 *** Roads_nationa_dst -0.0000395028971 0.0000027157471 -14.54587 < 0.000000000000000222 *** Roads_regional_feede_dst -0.0000566110721 0.0000027713904 -20.42696 < 0.000000000000000222 *** Roads_regiona_dst -0.0000402535647 0.0000033501616 -12.01541 < 0.000000000000000222 *** Service_F 59524.9918790496813 55482.8716042225569 1.07285 0.28333686 Service_M -11669.9277198423060 5787.4071055766972 -2.01643 0.04375455 * SEX_RATIO -0.1073882834614 0.0069930281094 -15.35648 < 0.000000000000000222 *** Small_Hat_Bazar_dst 0.0000967299711 0.0000037116779 26.06098 < 0.000000000000000222 *** STRUCTURE_JHUPRI 1.9118364584795 1.3887541242794 1.37666 0.16861868 STRUCTURE_KUTCHA 1.9262823338151 1.3892127198644 1.38660 0.16556377 STRUCTURE_PUCKA 1.5407728963395 1.3861487013134 1.11155 0.26633190 STRUCTURE_SEMI_PUCKA 2.2255109545882 1.3880169917684 1.60337 0.10885201 Upazilla_lin_dst 0.0000670825793 0.0000050933297 13.17067 < 0.000000000000000222 *** Zilla_lin_dst 0.0000488043007 0.0000034479423 14.15462 < 0.000000000000000222 *** --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for binomial family taken to be 1) Null deviance: 245860.02 on 177385 degrees of freedom Residual deviance: 203410.02 on 177312 degrees of freedom AIC: 203558.02 Number of Fisher Scoring iterations: 6  More background of my model can be found below links: here and here Now my question is why some coefficients are so varied? Update After using vifstep from usdm package I collected only 40 from 70+ variables. Afterward I ran LASSO with again and found coefficient as below: 41 x 1 sparse Matrix of class "dgCMatrix" 1 (Intercept) -1.550453712088479 Agri_Fema 0.022839833392058 Agri_Male -0.001644509136187 aster_30m_aspect 0.000512895998527 aster_30m_slope 0.005733959652405 ASTER_DEM_30m 0.006493690977654 Beat_office_dst 0.000044844700473 BF_Oct_201_dst -0.000017637409224 Blocks_line_dst 0.000069773041947 CHT_Agricultur_dst -0.001125024039569 CHT_Settlemen_dst 0.000069580040999 DNTW_M 0.011432546528745 Eastwardnes_dst 0.000010904529097 ELECTRICITY_CONNECTION -0.081578215424478 GLG_av_dst 0.000005532344286 GLG_cs_dst 0.000008260268395 GLG_QTd_dst -0.000011097490408 GLG_Tb_dst -0.000020101684698 GrowthCenter_dst 0.000064357864646 H20_24 0.182427325999767 H65 0.459375819924272 HH_M -0.008006633888449 HT_RENTED -0.186416737714288 HT_RENT_FREE -0.060092989358567 Indstry_F 0.036439882944596 Indstry_M -0.002359706219711 LITERATE_MALE_YES -0.001379936201357 LknJB_F 0.026845017905640 LknJB_M 0.009395604488612 Pourashava_cityCor_dst 0.000063851429584 Roads_nationa_dst -0.000040879721599 Roads_regional_feede_dst -0.000028409960055 Roads_regiona_dst -0.000074577418293 Service_F -0.014833316029951 Service_M -0.007457890434070 Small_Hat_Bazar_dst 0.000111206754901 STRUCTURE_JHUPRI -0.024295995035423 STRUCTURE_PUCKA -0.141639844445959 STRUCTURE_SEMI_PUCKA 0.176215393388445 Upazilla_lin_dst 0.000067759282061 Zilla_lin_dst -0.000017633887672  Then I even ran Ridge and Elastic net along with LASSO and compared their coefficient in the below graph: • Why did you run the logistic regression after the lasso? Why not just use the lasso estimated coefficients? – Matthew Drury Apr 1 '17 at 8:33 • I used LASSO for variable selection. The selected variable will be input to logit. Is not it a established practice of variable selection using LASSO. I read somewhere that LASSO does not do anything with COLLINEARITY. So i used vifstep of usdm package to remove all columns have score>10 the I ran cv.glmnet and fittd(glmnet) but this also give me another willdy shoot. Is not this the right way I am going along esp for the collinearity checking. could you help? I am new in this area. – SIslam Apr 1 '17 at 9:05 ## 1 Answer This is almost always caused by near-collinearity between some of your variables. Given the names of the variables, they do look potentially co-linear. If you care about prediction quality rather than interpreting coefficients, you can add in a regulariser which will damp the coefficients significantly. A$L_2\$ regulariser (ridge regression), or even use the lasso again but with a smaller regularization constant.

If some of your features are only active for very few instances, you can also see large weights. This is quite easy to check.

• Using logistic regression I need to order explanatory variables based on their effect(coefficient), i.e. which factor is the most culprit for deforestation and which is less etc.Which variables seem to have near collinearity from mere name , could you help?Could you come into chat if you need more please? – SIslam Apr 1 '17 at 9:04
• There is some advice here, I would also check for rarely active features by taking the column-wise mean and checking if the problematic variables have very small values compared to the rest. – AaronDefazio Apr 1 '17 at 11:30
• In terms of the problematic features, it looks like POP_DENSITY, MALE_POP_Density, and FEMALE_POP_Density are highly collinear. Based off of the names, I'm guessing that POP_DENSITY=MALE_POP_Density+FEMALE_POP_Density. – AaronDefazio Apr 1 '17 at 11:36
• Updated the question. See is it OK. Now how can I test if lasso coefficients are correct?Another note, I just divided each parcels population by area to standardise the variable e.g. POP of x parcel is transformed into (POP of x)/(Area of x in Hactres) is it ok – SIslam Apr 1 '17 at 12:02