3
$\begingroup$

I am implementing a logistic regression on a 250 x 20 dataset (250 observations of 20 variables) with a dichotomous response. In this proces I have encoutered some different problems, namely:

1. Group sizes

I am aware that small group sizes can be troublesome, although it is not completely clear to me when the problem is to be taking serious. Most of my predictor variables are fairly evenly distributed, but one continuous predictor only has very few observations (1-3) in extreme cases. Moreover, in two categorical predictors taking values 0 and 1 only about 10 % are 1's. Taking the size of the dataset into account, could this be a potential problem?

2. Correlations

There is a high correlation between some of the predictors (correlation coefficients ranging from .6 to .8) which seem to fall in two seperate clusters. I am using stepAIC (in R) and LASSO for model selection, and one particular predictor (which is highly correlated with another simliar predictor) always ends up in the final models with a positive contribution, although it being negative would clearly be the most intuitive. The second predictor however, always shows a negative contribution. Could this be caused by the high correlation between these two predictors?

Also, can you include non numerical predictors (say Yes/No values) in a correlation-diagram by simply assigning numerical values to them, like 0 and 1?

3. Diagnostics for logistic regression

I have come across a lot of different suggestions of how to perform diagnostics for logistic regression models. To me, one of the mroe appealing approaches was residuals plots.

The plots below show the predictors (all continuous) plotted against the residuals of a final model chosen through stepAIC. My initial thought is that there are no clear patterns in the residuals that would suggest adding more complexity to the model (through interaction terms or variable transformations). Are you sharing this point of view, or am I missing something?

Residuals

The deviance test for assessing goodness of fit for the model (call it m) is carried out in R as follows (correct me if I am wrong):

1 - pchisq(deviance(m), df.residual(m)).

For my particular model, i get a p-value of .87. This should be telling me how plausible it is for the model to have generated the data, but is it assessing the linear assumption of the log odds given in the logistic regression model? Are there other ways to assess the linearity assumption?

4. 'Pseudo' R-squared

Is it common practice to report a 'pseudo' R-squared for predictive performance, when implementing logistic regression? The McFadden and Cox & Snell to me seems the most recommended ones, should you choose to report one.

Thanks in advance!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The main thing that's missing from this question is a description of why you're fitting this model (e.g. exploration, prediction, or hypothesis testing), which really strongly affects how you should do your model-building. If you're hypothesis testing, then you very probably shouldn't be doing model selection at all, as eloquently stated by Frank Harrell (also in his book, Regression modeling strategies ...) $\endgroup$
    – Ben Bolker
    Dec 20, 2015 at 16:45
  • $\begingroup$ Good point, I never thought of it that way. The main goal I would say, is to explore which covariates are related to the response. I am not hypothesis testing, my model selection is based on the AIC. Of course I would like the final model to have some predictive power, but it should not be too complex (include all available covariates). $\endgroup$
    – AfBM
    Dec 24, 2015 at 11:12

1 Answer 1

Reset to default
1
$\begingroup$

There are really too many questions here. Given that you're working in a predictive context, I think your questions boil down to/can be rephrased as "are any of these deviations from best practices large enough to cause problems with the validity of the approximation that a model chosen via stepwise AIC-based selection will be the one with the best expected predictive accuracy?"

  • group size (effective number of samples): less information = lower power = poorer predictive accuracy (and poorer ability to discriminate among models), but I don't see that there's much you can do about it. You might have a look at Frank Harrell's guidelines for effective sample size (reproduced here), and cut down your model if you seem to be violating them too badly.
  • correlation:
    • moderately strong correlation (say <0.95) is only really a problem for hypothesis testing - it is hard to distinguish the independent effects of correlated predictors - but not so much for prediction. It's a little bit of a problem for stepwise approaches in that they pick whichever of the correlated variables seems more predictive for a particular data set, which makes the results rather unstable, but this isn't a huge problem. Lasso should handle correlated variables OK.
      • it's fairly routine for the effect of a variable to be negative conditional on another variable, even though its marginal effect is positive.
      • it's fine to treat binary categorical variables as numeric dummy (0/1) variables
  • if you don't see clear patterns in the residuals you're usually OK. Sometimes it's helpful to add a smooth line (loess or GAM fit) to emphasize patterns
  • the excess deviance test doesn't really tell you a huge amount in the case of binary prediction; trust your graphical diagnostics
  • you're welcome to report a pseudo-$R^2$ if you like ...
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.