I have data that have been collected using case-control procedures, in which the population of positive cases is collected with a random sample of negative cases. This yields 62 positive cases and 179 controls. There are 58 possible predictor variables (mostly numeric, two factors).

My goal is not classification per se--I am in the social sciences and significance testing is a big deal in my discipline, for better or worse. Rather, my collaborators want to make some inferences about the predictors that increase the probability of being a positive case. Obviously, the data are unbalanced, non-randomly sampled, and I have low power. I am trying to select variables for a logistic regression.

As an exploratory option, I have run a random forest with various specifications to try and find important variables. My false positive rate is, as expected, only about .03, as there are many more positive cases than negative. My false negative rate is quite high, about .70. Adjusting the weights doesn't seem to help with this.

What would be a good procedure for variable selection that could then be used to estimate rare events logistic regressions using the reduced set of variables?

  • $\begingroup$ You may give RF one more chance with Boruta, yet be aware that the chance of overfitting any method is huge -- some validation is a must. $\endgroup$ – user88 Jul 7 '12 at 11:45
  • $\begingroup$ @mbq Thanks. I'll give it a shot. I also had some luck in reducing the bootstrap sample size in the randomForest() function, since the classwt option seemed to have no interpretable effect. $\endgroup$ – Trey Jul 7 '12 at 18:47
  • $\begingroup$ Possible duplicate of Does an unbalanced sample matter when doing logistic regression? $\endgroup$ – kjetil b halvorsen Nov 3 '16 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.