Sample size for logistic regression?

Question

I want to make a logistic model from my survey data. It is a small survey of four residential colonies in which only 154 respondents were interviewed. My dependent variable is "satisfactory transition to work". I found that, of the 154 respondents, 73 said that they have satisfactorily transitioned to work, while the rest did not. So the dependent variable is binary in nature and I decided to use logistic regression. I have seven independent variables (three continuous and four nominal). One guideline suggest that there should be 10 cases for each predictor / independent variable (Agresti, 2007). Based on this guideline I feel that it is OK to run logistic regression.

Am I right? If not please let me know how to decide the number of independent variables?

Answer 1

There are several issues here.

Typically, we want to determine a minimum sample size so as to achieve a minimally acceptable level of statistical power. The sample size required is a function of several factors, primarily the magnitude of the effect you want to be able to differentiate from 0 (or whatever null you are using, but 0 is most common), and the minimum probability of catching that effect you want to have. Working from this perspective, sample size is determined by a power analysis.

Another consideration is the stability of your model (as @cbeleites notes). Basically, as the ratio of parameters estimated to the number of data gets close to 1, your model will become saturated, and will necessarily be overfit (unless there is, in fact, no randomness in the system). The 1 to 10 ratio rule of thumb comes from this perspective. Note that having adequate power will generally cover this concern for you, but not vice versa.

The 1 to 10 rule comes from the linear regression world, however, and it's important to recognize that logistic regression has additional complexities. One issue is that logistic regression works best when the percentages of 1's and 0's is approximately 50% / 50% (as @andrea and @psj discuss in the comments above). Another issue to be concerned with is separation. That is, you don't want to have all of your 1's gathered on one extreme of an independent variable (or some combination of them), and all of the 0's at the other extreme. Although this would seem like a good situation, because it would make perfect prediction easy, it actually makes the parameter estimation process blow up. (@Scortchi has an excellent discussion of how to deal with separation in logistic regression here: How to deal with perfect separation in logistic regression?) With more IV's, this becomes more likely, even if the true magnitudes of the effects are held constant, and especially if your responses are unbalanced. Thus, you can easily need more than 10 data per IV.

One last issue with that rule of thumb, is that it assumes your IV's are orthogonal. This is reasonable for designed experiments, but with observational studies such as yours, your IV's will almost never be roughly orthogonal. There are strategies for dealing with this situation (e.g., combining or dropping IV's, conducting a principal components analysis first, etc.), but if it isn't addressed (which is common), you will need more data.

A reasonable question then, is what should your minimum N be, and/or is your sample size sufficient? To address this, I suggest you use the methods @cbeleites discusses; relying on the 1 to 10 rule will be insufficient.

Answer 2

I typically use a 15:1 rule (ratio of min(events, non-events) to number of candidate parameters in the model). More recent work found that for a more rigorous validation 20:1 is needed. More information may be found in my course handouts (linked from here), in particular an argument for a minimum sample size of 96 just to estimate the intercept. But the sample size requirement is more nuanced, and an even more recent paper addresses this more comprehensively.

Answer 3

Usually, too few cases wrt. the model complexity (number of parameters) means that the models are unstable. So if you want to know whether you sample size / model complexity is OK, check whether you obtain a reasonably stable model.

There are (at least) two different kinds of instability:

1. The model parameters vary a lot with only slight changes in the training data.

2. The predictions (for the same case) of models trained with slight changes in the training data vary a lot.

You can measure 1. by looking how much your model coefficients vary if the training data is slightly perturbed. A suitable bunch of models can be calculated e.g. during bootstrap or (iterated) cross validation procedures.

For some types of models or problems, varying parameters do not imply varying predictions. You can directly check instability 2. by looking at the variation of predictions for the same case (regardless of whether they are correct or not) calculated during out-of-bootstrap or iterated cross validation.

Answer 4

There is no strict rules, but you can include all independent variables so long as the nominal variables dont have too many categories. You need one "beta" for all except one of the class for each nominal variable. So if a nominal variable was say "area of work" and you have 30 areas, then you'd need 29 betas.

One way to overcome this problen it to regularise the betas - or penalise for large coefficients. This helps ensure that you model doesn't overfit the data. L2 and L1 regularisation are popular choices.

Another issue to consider is how representative your sample is. What population do you want to make inference of? do you have all the different types of people in the sample that there is in the population? it will be difficult to make accurate inference if your sample has "holes" (eg no females aged 35-50 in the sample or no high income workers etc)

Answer 5

Here is the actual answer from the MedCalc website user41466 wrote about

http://www.medcalc.org/manual/logistic_regression.php

Sample size considerations

Sample size calculation for logistic regression is a complex problem, but based on the work of Peduzzi et al. (1996) the following guideline for a minimum number of cases to include in your study can be suggested. Let p be the smallest of the proportions of negative or positive cases in the population and k the number of covariates (the number of independent variables), then the minimum number of cases to include is: N = 10 k / p For example: you have 3 covariates to include in the model and the proportion of positive cases in the population is 0.20 (20%). The minimum number of cases required is N = 10 x 3 / 0.20 = 150 If the resulting number is less than 100 you should increase it to 100 as suggested by Long (1997).

Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR (1996) A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology 49:1373-1379.

Answer 6

Results from any logistic model with the number of observations per independent variable ranging from at least five to nine are reliable, especially so if results are statistically significant (Vittinghoff & McCulloch, 2007).

Vittinghoff, E., & McCulloch, C. E. 2007. Relaxing the rule of ten events per variable in logistic and Cox regression. American Journal of Epidemiology, 165(6): 710–718.