# How to interpret statistics (Emax, D, U, Q, B, etc.) of bootstrap validation of logistic regression

I'm only a linguist, so my knowledge of statistics is very basic.

I fitted a logistic regression model with R (with lrm(formula, y=T, x=T)), and when I use the option validate(lrm), I get some statistics I don't really understand.

index.orig   training   test optimism index.corrected     n
Dxy           0.5984   0.6112  0.5461   0.0651          0.5333 40
R2            0.3258   0.3676  0.2929   0.0747          0.2511 40
Intercept     0.0000   0.0000 -0.0105   0.0105         -0.0105 40
Slope         1.0000   1.0000  0.8427   0.1573          0.8427 40
Emax          0.0000   0.0000  0.0399   0.0399          0.0399 40
D             0.2713   0.3176  0.2394   0.0782          0.1931 40
U            -0.0177  -0.0177  0.0092  -0.0269          0.0092 40
Q             0.2890   0.3353  0.2302   0.1051          0.1839 40
B             0.1864   0.1772  0.1972  -0.0201          0.2064 40
g             1.4632   1.6642  1.3460   0.3182          1.1449 40
gp            0.2840   0.3011  0.2703   0.0308          0.2532 40


I don't really understand most of that. I think R2 and Dxy are supposed to be statistics of how good the predictors are, but I'm not sure how I should interpret the values, does the corrected Dyx = 0.651 mean that there is a strong correlation, while the corrected R2 = 0.0747 means that the correlation is very weak? I think the model is overfitted, but I'm not sure if I'm right.

Also, the other statistics are totally strange to me. What are Emax, D, U, Q, B, g, and gp?

-

You need to consider several things. First, when you get into a new field it is advisable to study the field before using its methods. You probably need to take several stat courses or read several stat books to be able to make the right choices and to make the right interpretations. Second, you did not specify the model you used. Did it allow continuous predictors to be nonlinear? Did you fully pre-specify the model as the R rms package's validate function assumes when you don't specify bw=TRUE? In other words did you do any model selection that the bootstrap validation needs to re-run at each bootstrap re-sample? In terms of interpreting the output, see rms course notes at http://biostat.mc.vanderbilt.edu/CourseBios330; click on Handouts. Then read http://biostat.mc.vanderbilt.edu/wiki/pub/Main/FrankHarrell/logistCal.pdf. Also, run validate with B=200.

-
Thank you for your answer. I agree with you, I should have a better understanding of statistics, but I don't have it yet, and I have to do this. About the other questions, the predictors were all categorical variables. I'm not sure what you mean by the second question. And thanks for the links, I'll take a look. –  mguzmann Mar 29 '13 at 13:24
That other question is crucial. Did the validate function have access to all the features/predictors/independent variables that were ever examined in building the model? In other words was the model 100% pre-specified as the software (the way you are using it) assumed? Were any "insignificant" variables removed? Was there any univariate screening or stepwise variable selection? –  Frank Harrell Mar 29 '13 at 20:25
I did remove some predictors that the lrm told me were not significant, and I did not pass the original model to validate, but the latter one without those predictors. Am I doing it wrong? Should I pass the model with all predictors? Also I should note that what I'm trying to do is to determine which grammatical factors have a significant correlation with construction A or B, and if this correlation could be extended beyond the data I have. I don't really need to go beyond that. Thanks again for your answers. –  mguzmann Mar 30 '13 at 21:15
The data are seldom able to reliably tell you which factors are truly correlated with the outcome. But besides that, resampling model validation us ruined if you pretend that the model after deleting "insignificant" variables (at what $\alpha$ level?) was the model at the beginning. The result will be an inflation of the model's value, along with all the tremendous damage that stepwise variable selection does to the model. You have to let the resampling procedure run the same $Y$-related analyses repeatedly. Specify all candidate variables to validate with bw=TRUE + stopping rule. –  Frank Harrell Mar 31 '13 at 13:29
Thank you for your explanation, I will do that. –  mguzmann Apr 1 '13 at 14:12