If I am dealing with a small sample size (n = 48; n = 29 have disease vs n = 19 without disease), what are the maximum numbers of the predictors I can include in my multivariable logistic regression model (I am building a predictive model)?

There are so many rules online and I am not sure which one I should use.

Any help is much appreciated!

  • $\begingroup$ See stats.stackexchange.com/questions/502150/… $\endgroup$ Nov 24 at 19:38
  • 3
    $\begingroup$ If you are building a predictive model then you can probably include as many variables as you think are relevant and use regularisation (L1 or L2 or both) to avoid over-fitting (don't do feature selection, it is more likely to make things worse rather than better if you have regularization) $\endgroup$ Nov 24 at 19:42
  • $\begingroup$ @DikranMarsupial Can you pls share the R code for L2 or L2 method? Any sample example would be much appreciated $\endgroup$
    – R Beginner
    Nov 24 at 20:36
  • $\begingroup$ @RBeginner unfortunately I don't use R, but any good implementation of logistic regression ought to provide L2 regularisation (c.f. ridge regression). There are undoubtedly packages for L1 regularisation (look for "LASSO") and both L1 and L2 regularisation (look for "elastic net"). $\endgroup$ Nov 25 at 7:38
  • $\begingroup$ @DikranMarsupial No worries! So, in your opinion, we don't have to worry about the p-value for inclusion into my predictive model (e.g. only included predictors in my final model with p<0.05)? $\endgroup$
    – R Beginner
    Nov 25 at 18:29

A useful rule of thumb for logistic regression is to limit yourself to about 1 unpenalized predictor per 15 cases of the minority class. See Section 4.4 of Frank Harrell's course notes, for example. That's when you have a

typical problem in medicine, epidemiology, and the social sciences in which the signal:noise ratio is small.

See for example, this page linked in a comment from kjetil b halvorsen, and this page. If your signal:noise ratio is higher, you can get away with fewer cases per predictor.

I highlighted the word "unpenalized" above because you don't have to throw out all except 1 or 2 of your predictors. A penalized method ("regularization" mentioned in one of the comments) allows you to use more predictors than that rule of thumb.

The regression coefficients of the predictors are penalized to lower magnitudes than they would be in a standard regression, to help avoid overfitting. The penalty that provides best performance is typically chosen by cross-validation. Ridge regression ("L2 regularization") provides coefficients for all predictors. LASSO ("L1 regularization") provides penalized coefficients for some predictors and sets coefficients of others to 0. My guess is that you would be better served by ridge regression here, perhaps after you apply your knowledge of the subject matter to reduce the effective number of predictors. See Harrell's notes for ideas on how to implement data reduction, to cut down on the numbers of predictors without using the outcomes.

For logistic regression, penalization is implemented in the glmnet package.

  • $\begingroup$ Thank you for the great contribution. So, if I decide to use an unpenalized approach, I can roughly fit 2 only predictors in my data (cases mean participants with a disease, correct)? $\endgroup$
    – R Beginner
    Nov 24 at 20:34
  • $\begingroup$ @RBeginner that's correct. The limiting number is the size of the minority class regardless of how you name the classes, so in your case it's limited by the 19 without disease. Thus even 2 predictors would be pushing it and could overfit. I recommend that you try ridge regression with glmnet(), illustrated in Section 6.6.1 of ISLR for ordinary least squares but directly applicable to your logistic regression with a family="binomial" argument. $\endgroup$
    – EdM
    Nov 24 at 21:49
  • $\begingroup$ Does the minority class mean the group with the smaller sample size? $\endgroup$
    – R Beginner
    Nov 25 at 1:24
  • $\begingroup$ Can you pls share a R code with the glmnet( )? Is it similar to the logistic regression one (e.g. glm)? $\endgroup$
    – R Beginner
    Nov 25 at 1:25
  • $\begingroup$ @RBeginner the minority group is the one with the smaller sample size. ISLR in my prior comment is freely available and the cited section 6.6.1 shows how to do ridge regression with glmnet(), including use of cv.glmnet() to find the optimum penalty. You specify the argument alpha = 0 to do ridge regression. If you also use a family="binomial" argument it will do logistic ridge regression. Unlike other R functions, you have to provide separate outcome and predictor vectors/matrices, however. $\endgroup$
    – EdM
    Nov 25 at 2:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.