Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I want to make a logistic model from my survey data. It is a small survey of four residential colonies in which only 154 respondent was interviewed. My dependent variable is "satisfactory transition to work". I found that of the 154 respondent 73 said that they have satisfactorily transited to work, while the rest did not. So the dependent variable is binary in nature and I decided to use logistic regression. I have seven independent variables (three continuous and four nominal). One guideline suggest that there should be 10 cases for each predictor/independent variable (Agresti, 2007). Based on this guideline I feel that it is ok to run logistic regression.

Am I right? If not please let me know how to decide the number of independent variables?

share|improve this question
2  
I have never really understood the rule of thumb that says "10 cases for each predictor" (and unfortunately I don't have access to the book written by Agresti). What I mean is: if I have 100 subjects of which 10 are cases (the 1's) and 90 non-cases (the 0's), then the rule says "include only 1 predictor". But what if I model the 0's instead of the 1's and then I take the reciprocal of the estimated odds ratios? Would I be allowed to include 9 predictors? That makes no sense to me. –  boscovich Apr 7 '12 at 10:13
    
Dear Andrea, I have said the same thing that you mean. Out of 154 respondents there are 73 cases (the 1's and rest 0's). Could you throw some light on my question.Thanks! –  Braj-Stat Apr 7 '12 at 15:56
2  
In a commentary i have read that one has to look at the minimum of the number of events and non-events. So in the example of 10/100 you end up with one predictor irrespective of how you code it. –  psj Apr 8 '12 at 11:08
    
@psj that sounds reasonable. Do you have any references? –  boscovich Apr 12 '12 at 7:30
1  
There is a related discussion here: minimum-number-of-observations-for-logistic-regression. –  gung Dec 13 '12 at 14:20

4 Answers 4

There are several issues here.

Typically, we want to determine a minimum sample size so as to achieve a minimally acceptable level of statistical power. The sample size required is a function of several factors, primarily the magnitude of the effect you want to be able to differentiate from 0 (or whatever null you are using, but 0 is most common), and the minimum probability of catching that effect you want to have. Working from this perspective, sample size is determined by a power analysis.

Another consideration is the stability of your model (as @cbeleites notes). Basically, as the ratio of parameters estimated to the number of data gets close to 1, your model will become saturated, and will necessarily be overfit (unless there is, in fact, no randomness in the system). The 1 to 10 ratio rule of thumb comes from this perspective. Note that having adequate power will generally cover this concern for you, but not vice versa.

The 1 to 10 rule comes from the linear regression world, however, and it's important to recognize that logistic regression has additional complexities. One issue is that logistic regression works best when the percentages of 1's and 0's is approximately 50% / 50% (as @andrea and @psj discuss in the comments above). Another issue to be concerned with is separation. That is, you don't want to have all of your 1's gathered on one extreme of an independent variable (or some combination of them), and all of the 0's at the other extreme. Although this would seem like a good situation, because it would make perfect prediction easy, it actually make the parameter estimation process blow up. (@Scortchi has an excellent discussion of how to deal with separation in logistic regression here: How to deal with perfect separation in logistic regression?) With more IV's, this becomes more likely, even if the true magnitudes of the effects are held constant, and especially if your responses are unbalanced. Thus, you can easily need more than 10 data per IV.

One last issue with that rule of thumb, is that it assumes your IV's are orthogonal. This is reasonable for designed experiments, but with observational studies such as yours, your IV's will almost never be roughly orthogonal. There are strategies for dealing with this situation (e.g., combining or dropping IV's, conducting a principal components analysis first, etc.), but if it isn't addressed (which is common), you will need more data.

A reasonable question then, is what should your minimum N be, and/or is your sample size sufficient? To address this, I suggest you use the methods @cbeleites discusses; relying on the 1 to 10 rule will be insufficient.

share|improve this answer
1  
Can you provide a reference for the statement "One issue is that logistic regression works best when the percentages of 1's and 0's is approximately 50% / 50%"? I've been wondering about this myself, as I have a dataset that is very far from 50/50 and I'm wondering the implications. (sorry to resurrect the thread) –  Trevor Sep 6 '13 at 1:48
1  
I don't see any problem w/ resurrecting an old thread when it's appropriate, @Trevor. I think what you are looking for is something along the lines of this nice answer by conjugate prior: does-an-unbalanced-sample-matter-when-doing-logistic-regression. –  gung Sep 6 '13 at 2:11

Usually, too few cases wrt. the model complexity (number of parameters) means that the models are unstable. So if you want to know whether you sample size / model complexity is OK, check whether you obtain a reasonably stable model.

There are (at least) two different kinds of instability:

  1. The model parameters vary a lot with only slight changes in the training data.

  2. The predictions (for the same case) of models trained with slight changes in the training data vary a lot.

You can measure 1. by looking how much your model coefficients vary if the training data is slightly perturbed. A suitable bunch of models can be calculated e.g. during bootstrap or (iterated) cross validation procedures.

For some types of models or problems, varying parameters do not imply varying predictions. You can directly check instability 2. by looking at the variation of predictions for the same case (regardless of whether they are correct or not) calculated during out-of-bootstrap or iterated cross validation.

share|improve this answer

There is no strict rules, but you can include all independent variables so long as the nominal variables dont have too many categories. You need one "beta" for all except one of the class for each nominal variable. So if a nominal variable was say "area of work" and you have 30 areas, then you'd need 29 betas.

One way to overcome this problen it to regularise the betas - or penalise for large coefficients. This helps ensure that you model doesn't overfit the data. L2 and L1 regularisation are popular choices.

Another issue to consider is how representative your sample is. What population do you want to make inference of? do you have all the different types of people in the sample that there is in the population? it will be difficult to make accurate inference if your sample has "holes" (eg no females aged 35-50 in the sample or no high income workers etc)

share|improve this answer

This article presents a formula for determining the appropriate sample size for a logistic regression analysis based on information form the number of covariates and the proportion of 1's and 0's in the dataset.

http://www.medcalc.org/manual/logistic_regression.php

share|improve this answer
1  
It would be helpful if you added the formula in case this link goes dead. –  Dimitriy V. Masterov Mar 7 at 1:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.