0
$\begingroup$

I was trying to help a student and was foxed by this Logistic Regression problem and seek your explanation.

Here's some economic data. All we have to do is create a model to predict whether the country falls in debt. The response variable is "Dependent.variable"

I analyzed using 'R' and the glm function and what I found was:

  • All the variables showed a 'p-value' of 1, indicating none of the variables are significant.
  • I plotted each variable versus the dependent - and sure enough - the variable is not co-related to the response. The feature values are scattered all over the place
  • And yet - the model fits perfectly! On a test set - it predicted all cases perfectly
  • The ROC curve is a rock solid square
  • I then took all the "coefficients estimates" and built the equation and computed probabilities using the expanded form of e/(1+e) - and it does give near perfect probabilities

How is this possible? What am I missing here!

Data > https://drive.google.com/open?id=0B6dzZ1cZd6PKeTVDM1RWaWpBUWs

Edit: Here is the model and the output

Call:
glm(formula = Dependent.variable ~ ., family = binomial, data = dfTrain)

Deviance Residuals: 
       Min          1Q      Median          3Q         Max  
-4.107e-05  -2.100e-08  -2.100e-08   2.100e-08   4.278e-05  

Coefficients:
                                 Estimate Std. Error z value Pr(>|z|)
(Intercept)                    -3.238e+01  6.458e+05   0.000    1.000
Inflation.CPI.                  1.931e+00  4.966e+03   0.000    1.000
Debt.GDP                        2.590e-01  1.830e+03   0.000    1.000
OfficialForexReserves.US.Bil.. -2.575e+00  2.538e+03  -0.001    0.999
GrossInvestment.GDP             7.396e+00  6.158e+04   0.000    1.000
Gov.Revenue.GDP                -5.396e+00  3.016e+04   0.000    1.000
Net.FDI.GDP                     8.927e-01  1.135e+04   0.000    1.000

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 6.4924e+01  on 50  degrees of freedom
Residual deviance: 6.0091e-09  on 44  degrees of freedom
AIC: 14

Number of Fisher Scoring iterations: 25

Rajesh

Improve this question
$\endgroup$
6
  • $\begingroup$ Can you provide the exact model you used for your question? $\endgroup$
    – timfaber
    Commented Oct 10, 2017 at 11:35
  • $\begingroup$ Hi - do you mean the code I used? lrModel <- glm (Dependent.variable ~ ., data = dfTrain, family = binomial) summary(lrModel) $\endgroup$
    – RajeshS
    Commented Oct 10, 2017 at 11:39
  • $\begingroup$ Hm, there might be several things, year is not distributed per level of the DV. But more importantly, the SE is quite high producing small z-values and hence p-values of 1. Multicollinearity is not the issue looking at the correlations. Consider moving this to CV $\endgroup$
    – timfaber
    Commented Oct 10, 2017 at 12:13
  • 3
    $\begingroup$ Your model is overfit. You have 72 observations, of which 24 are events. The general rule of thumb I use when fitting a logistic regression is one degree of freedom for every 10 events. So your data can support 2--maybe 3--degrees of freedom. It looks like you are using six. You should consider using fewer independent variables. $\endgroup$
    – Benjamin
    Commented Oct 10, 2017 at 13:32
  • 7
    $\begingroup$ p-values don't measure model fitness; they measure the size of a coefficient relative to its standard error $\endgroup$
    – Sycorax ♦
    Commented Oct 10, 2017 at 14:41

1 Answer 1

Reset to default
12
$\begingroup$

If you look carefully at the output, you should have read in the console

Warning messages: 
1: glm.fit: algorithm did not converge
2: glm.fit: fitted probabilities numerically 0 or 1 occurred

The output of glm cannot be relied upon. The problem seems due to complete separation of variables. Running a penalized logistic regression with Jeffreys prior penalty yields MAP values:

logistf::logistf(formula = Dependent.variable ~ . - Country -  Year, data = dat)
Model fitted by Penalized ML
Confidence intervals and p-values by Profile Likelihood 

                                       coef   se(coef)  lower 0.95  upper 0.95       Chisq            p
(Intercept)                     3.369863624 4.46210240 -11.4364696 25.63004559  0.37821187 5.385618e-01
Inflation.CPI.                  0.207995387 0.21203529  -0.6598256  1.19474042  0.73101619 3.925540e-01
Debt.GDP                        0.004909774 0.01848469  -0.0494921  0.09541694  0.06350028 8.010468e-01
OfficialForexReserves.US.Bil.. -0.191903652 0.07689628  -0.8971011 -0.06585468 22.38454885 2.231622e-06
GrossInvestment.GDP             0.138778700 0.16639479  -0.2263430  1.44319711  0.51060932 4.748752e-01
Gov.Revenue.GDP                -0.254483400 0.09539591  -0.6933168 -0.09166023 10.29617094 1.333065e-03
Net.FDI.GDP                     0.021054572 0.09666093  -0.3040144  0.40133013  0.04559491 8.309139e-01

Likelihood ratio test=65.10526 on 6 df, p=4.105605e-12, n=69

and the P-values are not ones anymore. See the post How to deal with perfect separation in logistic regression? for more information

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the explanations - extremely helpful. @Benjamin: I had dropped features and run it once. However it still produced perfect results on the test set and hence had posted the question. $\endgroup$
    – RajeshS
    Commented Oct 10, 2017 at 16:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.