# Evaluating a logistic regression model

I've been working on a logistic model and I'm having some difficulties evaluating the results. My model is a binomial logit. My explanatory variables are: a categorical variable with 15 levels, a dichotomous variable, and 2 continuous variables. My N is large >8000.

I am trying to model the decision of firms to invest. The dependent variable is investment (yes/no), the 15 level variables are different obstacles for investments reported by managers. The rest of the variables are controls for sales, credits and used capacity.

Below are my results, using the rms package in R.

  Model Likelihood     Discrimination    Rank Discrim.
Ratio Test            Indexes          Indexes
Obs          8035    LR chi2     399.83    R2       0.067    C       0.632
1           5306    d.f.            17    g        0.544    Dxy     0.264
2           2729    Pr(> chi2) <0.0001    gr       1.723    gamma   0.266
max |deriv| 6e-09                          gp       0.119    tau-a   0.118
Brier    0.213

Coef    S.E.   Wald Z Pr(>|Z|)
Intercept -0.9501 0.1141 -8.33  <0.0001
x1=10     -0.4929 0.1000 -4.93  <0.0001
x1=11     -0.5735 0.1057 -5.43  <0.0001
x1=12     -0.0748 0.0806 -0.93  0.3536
x1=13     -0.3894 0.1318 -2.96  0.0031
x1=14     -0.2788 0.0953 -2.92  0.0035
x1=15     -0.7672 0.2302 -3.33  0.0009
x1=2      -0.5360 0.2668 -2.01  0.0446
x1=3      -0.3258 0.1548 -2.10  0.0353
x1=4      -0.4092 0.1319 -3.10  0.0019
x1=5      -0.5152 0.2304 -2.24  0.0254
x1=6      -0.2897 0.1538 -1.88  0.0596
x1=7      -0.6216 0.1768 -3.52  0.0004
x1=8      -0.5861 0.1202 -4.88  <0.0001
x1=9      -0.5522 0.1078 -5.13  <0.0001
d2         0.0000 0.0000 -0.64  0.5206
f1        -0.0088 0.0011 -8.19  <0.0001
k8         0.7348 0.0499 14.74  <0.0001


Basically I want to assess the regression in two ways, a) how well the model fits the data and b) how well the model predicts the outcome. To assess goodness of fit (a), I think deviance tests based on chi-squared are not appropriate in this case because the number of unique covariates approximates N, so we cannot assume a X2 distribution. Is this interpretation correct?

I can see the covariates using the epiR package.

require(epiR)
logit.cp <- epi.cp(logit.df[-1]))

id n x1   d2 f1 k8
1 1 13 2030 56  1
2 1 14  445 51  0
3 1 12 1359 51  1
4 1  1 1163 39  0
5 1  7  547 62  0
6 1  5 3721 62  1
...
7446


I have also read that the Hosmer-Lemeshow GoF test is outdated, as it divides the data by 10 in order to run the test, which is rather arbitrary.

Instead I use the le Cessie–van Houwelingen–Copas–Hosmer test, implemented in the rms package. I not sure exactly how this test is performed, I have not read the papers about it yet. In any case, the results are:

Sum of squared errors    Expected value|H0           SD             Z            P
1711.6449914         1712.2031888    0.5670868    -0.9843245    0.3249560


P is large, so there isn't sufficient evidence to say that my model doesn't fit. Great! However....

When checking the predictive capacity of the model (b), I draw a ROC curve and find that the AUC is 0.6320586. That doesn’t look very good.

So, to sum up my questions:

1. Are the tests I run appropriate to check my model? What other test could I consider?

2. Do you find the model useful at all, or would you dismiss it based on the relatively poor ROC analysis results?

• Are you sure that your x1 should be taken as a single categorical variable? That is, does every case have to have 1, & only 1, 'obstacle' to investing? I would think that some cases could be confronted with 2 or more of the obstacles, & some cases have none. Oct 24, 2017 at 18:42