# Sample size for logistic regression?

I want to make a logistic model from my survey data. It is a small survey of four residential colonies in which only 154 respondents were interviewed. My dependent variable is "satisfactory transition to work". I found that, of the 154 respondents, 73 said that they have satisfactorily transitioned to work, while the rest did not. So the dependent variable is binary in nature and I decided to use logistic regression. I have seven independent variables (three continuous and four nominal). One guideline suggest that there should be 10 cases for each predictor / independent variable (Agresti, 2007). Based on this guideline I feel that it is OK to run logistic regression.

Am I right? If not please let me know how to decide the number of independent variables?

• I have never really understood the rule of thumb that says "10 cases for each predictor" (and unfortunately I don't have access to the book written by Agresti). What I mean is: if I have 100 subjects of which 10 are cases (the 1's) and 90 non-cases (the 0's), then the rule says "include only 1 predictor". But what if I model the 0's instead of the 1's and then I take the reciprocal of the estimated odds ratios? Would I be allowed to include 9 predictors? That makes no sense to me. – boscovich Apr 7 '12 at 10:13
• Dear Andrea, I have said the same thing that you mean. Out of 154 respondents there are 73 cases (the 1's and rest 0's). Could you throw some light on my question.Thanks! – Braj-Stat Apr 7 '12 at 15:56
• In a commentary i have read that one has to look at the minimum of the number of events and non-events. So in the example of 10/100 you end up with one predictor irrespective of how you code it. – psj Apr 8 '12 at 11:08
• @psj that sounds reasonable. Do you have any references? – boscovich Apr 12 '12 at 7:30
• There is a related discussion here: minimum-number-of-observations-for-logistic-regression. – gung Dec 13 '12 at 14:20

There is no strict rules, but you can include all independent variables so long as the nominal variables dont have too many categories. You need one "beta" for all except one of the class for each nominal variable. So if a nominal variable was say "area of work" and you have 30 areas, then you'd need 29 betas.

One way to overcome this problen it to regularise the betas - or penalise for large coefficients. This helps ensure that you model doesn't overfit the data. L2 and L1 regularisation are popular choices.

Another issue to consider is how representative your sample is. What population do you want to make inference of? do you have all the different types of people in the sample that there is in the population? it will be difficult to make accurate inference if your sample has "holes" (eg no females aged 35-50 in the sample or no high income workers etc)

Usually, too few cases wrt. the model complexity (number of parameters) means that the models are unstable. So if you want to know whether you sample size / model complexity is OK, check whether you obtain a reasonably stable model.

There are (at least) two different kinds of instability:

1. The model parameters vary a lot with only slight changes in the training data.

2. The predictions (for the same case) of models trained with slight changes in the training data vary a lot.

You can measure 1. by looking how much your model coefficients vary if the training data is slightly perturbed. A suitable bunch of models can be calculated e.g. during bootstrap or (iterated) cross validation procedures.

For some types of models or problems, varying parameters do not imply varying predictions. You can directly check instability 2. by looking at the variation of predictions for the same case (regardless of whether they are correct or not) calculated during out-of-bootstrap or iterated cross validation.

There are several issues here.

Typically, we want to determine a minimum sample size so as to achieve a minimally acceptable level of statistical power. The sample size required is a function of several factors, primarily the magnitude of the effect you want to be able to differentiate from 0 (or whatever null you are using, but 0 is most common), and the minimum probability of catching that effect you want to have. Working from this perspective, sample size is determined by a power analysis.

Another consideration is the stability of your model (as @cbeleites notes). Basically, as the ratio of parameters estimated to the number of data gets close to 1, your model will become saturated, and will necessarily be overfit (unless there is, in fact, no randomness in the system). The 1 to 10 ratio rule of thumb comes from this perspective. Note that having adequate power will generally cover this concern for you, but not vice versa.

The 1 to 10 rule comes from the linear regression world, however, and it's important to recognize that logistic regression has additional complexities. One issue is that logistic regression works best when the percentages of 1's and 0's is approximately 50% / 50% (as @andrea and @psj discuss in the comments above). Another issue to be concerned with is separation. That is, you don't want to have all of your 1's gathered on one extreme of an independent variable (or some combination of them), and all of the 0's at the other extreme. Although this would seem like a good situation, because it would make perfect prediction easy, it actually makes the parameter estimation process blow up. (@Scortchi has an excellent discussion of how to deal with separation in logistic regression here: How to deal with perfect separation in logistic regression?) With more IV's, this becomes more likely, even if the true magnitudes of the effects are held constant, and especially if your responses are unbalanced. Thus, you can easily need more than 10 data per IV.

One last issue with that rule of thumb, is that it assumes your IV's are orthogonal. This is reasonable for designed experiments, but with observational studies such as yours, your IV's will almost never be roughly orthogonal. There are strategies for dealing with this situation (e.g., combining or dropping IV's, conducting a principal components analysis first, etc.), but if it isn't addressed (which is common), you will need more data.

A reasonable question then, is what should your minimum N be, and/or is your sample size sufficient? To address this, I suggest you use the methods @cbeleites discusses; relying on the 1 to 10 rule will be insufficient.

• Can you provide a reference for the statement "One issue is that logistic regression works best when the percentages of 1's and 0's is approximately 50% / 50%"? I've been wondering about this myself, as I have a dataset that is very far from 50/50 and I'm wondering the implications. (sorry to resurrect the thread) – Trevor Sep 6 '13 at 1:48
• I don't see any problem w/ resurrecting an old thread when it's appropriate, @Trevor. I think what you are looking for is something along the lines of this nice answer by conjugate prior: does-an-unbalanced-sample-matter-when-doing-logistic-regression. – gung Sep 6 '13 at 2:11
• +1 to Trevor's question. I believe that logistic regression will continue to benefit from new data, even if that data is of the same case (despite diminishing returns). That's actually something that has bothered me about machine learning techniques like random forests - that they can get worse by adding more relevant training data. Perhaps there's a point at which logistic regression would break down due to numerical considerations if the imbalance became too severe. Would be interested in learning more about this. – Ben Ogorek Dec 27 '16 at 19:17
• +1, perhaps this is implied by your answer I'm not sure, but I'm wondering how this works for categorical variables with different levels? Would it be suggested to have 10 observations per level? – baxx Apr 17 at 23:41
• It's a rule of thumb, @baxx, but yes, to do more than just estimate the percentages, you would need at least 45. – gung Apr 18 at 1:12

Here is the actual answer from the MedCalc website user41466 wrote about

Sample size considerations

Sample size calculation for logistic regression is a complex problem, but based on the work of Peduzzi et al. (1996) the following guideline for a minimum number of cases to include in your study can be suggested. Let p be the smallest of the proportions of negative or positive cases in the population and k the number of covariates (the number of independent variables), then the minimum number of cases to include is: N = 10 k / p For example: you have 3 covariates to include in the model and the proportion of positive cases in the population is 0.20 (20%). The minimum number of cases required is N = 10 x 3 / 0.20 = 150 If the resulting number is less than 100 you should increase it to 100 as suggested by Long (1997).

Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR (1996) A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology 49:1373-1379.

• So it is the same 10cases per independent variable (with floor) – seanv507 Mar 16 '16 at 20:08

I typically use a 15:1 rule (ratio of min(events, non-events) to number of candidate parameters in the model). More recent work found that for a more rigorous validation 20:1 is needed. More information may be found in my course handouts linked from http://biostat.mc.vanderbilt.edu/rms, in particular an argument for a minimum sample size of 96 just to estimate the intercept. But the sample size requirement is more nuanced, and an even more recent paper addresses this more comprehensively.

Results from any logistic model with the number of observations per independent variable ranging from at least five to nine are reliable, especially so if results are statistically significant (Vittinghoff & McCulloch, 2007).

Vittinghoff, E., & McCulloch, C. E. 2007. Relaxing the rule of ten events per variable in logistic and Cox regression. American Journal of Epidemiology, 165(6): 710–718.

• Note that it's not strictly the "number of observations per independent variable" that's in question, it's the number of "events." For a logistic regression, the number of "events" is the number of cases in the least-frequent of the two outcome classes. That will be no greater than 1/2 of the number of total observations, and in some applications a good deal lower than that. – EdM Dec 27 '16 at 19:01

## protected by kjetil b halvorsenSep 1 '18 at 8:21

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).