8
$\begingroup$

I'm running a binary logistic regressions with 3 numerical variables. I'm suppressing the intercept in my models as the probability should be zero if all input variables are zero.

What's minimal number of observations I should use?

$\endgroup$
  • 8
    $\begingroup$ You would need an intercept of $-\infty$ to do that! An intercept of 0 corresponds to a probability of $\frac{1}{1 + \exp(-0)} = 1/2$, not $0$, when all independent variables are zero. $\endgroup$ – whuber Jun 8 '11 at 18:57
  • 2
    $\begingroup$ There is a related discussion here: sample-size-for-logistic-regression. $\endgroup$ – gung Dec 13 '12 at 14:18
17
$\begingroup$

There is one way to get at a solid starting point. Suppose there were no covariates, so that the only parameter in the model were the intercept. What is the sample size required to allow the estimate of the intercept to be precise enough so that the predicted probability is within 0.1 of the true probability with 95% confidence, when the true intercept is in the neighborhood of zero? The answer is n=96. What if there were one covariate, and it was binary with a prevalence of 0.5? One would need 96 subjects with x=0 and 96 with x=1 to have an upper bound on the margin of error for estimating Prob[Y=1 | X=x] not exceed 0.1. The general formula for the sample size required to achieve a margin of error of $\delta$ in estimating a true probability of $p$ at the 0.95 confidence level is $n = (\frac{1.96}{\delta})^{2} \times p(1-p)$. Set $p = 0.5$ for the worst case.

$\endgroup$
  • $\begingroup$ I appreciate your help in this forum. I have ~90000 events and ~2000000 non-events. I need to a logistic model with 65 predictors. Now how and how many sample can I take. in fact my question is related to stats.stackexchange.com/questions/268201/… $\endgroup$ – SIslam Mar 22 '17 at 13:12
  • $\begingroup$ No problem with fitting 65 simultaneously with your effective sample size. $\endgroup$ – Frank Harrell Mar 22 '17 at 14:00
  • $\begingroup$ but I was suggested that too many samples can cause problem since I was getting psudo r squared as low. $\endgroup$ – SIslam Mar 22 '17 at 14:31
  • 2
    $\begingroup$ Are you kidding? If the $R^2$ is low using a large sample, that is the most accurate estimate of the true $R^2$ and dropping observations will not improve the performance of the model; it will only make it worse. Supplement the $R^2$ with other easier to understand metrics such as the $c$-index (concordance probability; ROC area). And above all, ignore any advice to "balance" the outcome category frequencies. $\endgroup$ – Frank Harrell Mar 22 '17 at 15:29
  • $\begingroup$ Do I need use glmnet for this to find most useful predictor at this stage? $\endgroup$ – SIslam Mar 23 '17 at 11:39
9
$\begingroup$

There isn't really a minimum number of observations. Essentially the more observations you have the more the parameters of your model are constrained by the data, and the more confident the model becomes. How many observations you need depends on the nature of the problem and how confident you need to be in your model. I don't think it is a good idea to rely too much on "rules of thumb" about this sort of thing, but use the all the data you can get and inspect the confidence/credible intervals on your model parameters and on predictions.

$\endgroup$
  • $\begingroup$ no minimum number! I have ~90000 event and ~2000000 non-events. I need to a logistic model with 65 regressor. I am told that this is too many samples, since I am taking this whole ~90000 events and ~90000 non events randomly selected from ~2000000 , try to lessen sample while samples are representative. at this stage how many sample can I take and how. In fact I am referring stats.stackexchange.com/questions/268201/… $\endgroup$ – SIslam Mar 22 '17 at 13:31
  • 3
    $\begingroup$ No don't do that $\endgroup$ – Frank Harrell Mar 22 '17 at 15:30
  • 1
    $\begingroup$ I agree with @FrankHarrell (possibly for a different reason?). The "class imbalance" problem tends to go away the more data you collect, and if you artificially balance the training data you are telling the model that the operational class frequencies are 50-50, which probably isn't true, and you will over classify the minority class in operational use. If you do do this, then post-process the output probabilities to adjust for the difference in training and operational class frequencies (at which point you will probably get essentially the same result as training with all the data). $\endgroup$ – Dikran Marsupial Mar 23 '17 at 8:25
0
$\begingroup$

Update: I didn't see the above comment, by @David Harris, which is pretty much like mine. Sorry for that. You guys can delete my answer if it is too similar.

I'd second Dikran Marsupail post and add my two cents.

Take in consideration your prior knowledge about the effects that you expect from your independent variables. If you expect small effects, than you will need a huge sample. If the effects are expected to be big, than a small sample can do the job.

As you might know, standard errors are a function of sample size, so the bigger the sample size, the smaller the standard errors. Thus, if effects are small, i.e., are near zero, only a small standard error will be able to detect this effect, i.e, to show that it is significantly different from zero. On the other hand, if the effect is big (far from zero), than even a large standard error will produce significant results.

If you need some reference, take a look at Andrew Gelmans' Blog.

$\endgroup$
  • 1
    $\begingroup$ Gelman's blog is getting pretty big :-). Do you have a particular post in mind? $\endgroup$ – whuber Jun 9 '11 at 14:06
  • $\begingroup$ @Whuber, you are right, I should have pointed to something more specific. He has some recent talk presentations about small effects and multiple comparisons, but I think the following link is enough: stat.columbia.edu/~gelman/research/published/power4r.pdf $\endgroup$ – Manoel Galdino Jun 9 '11 at 21:45
  • $\begingroup$ that link in the comments is dead and there's no reference to a particular post of the mentioned blog $\endgroup$ – baxx Apr 17 at 23:44
0
$\begingroup$

It seems that to get an acceptable estimation we have to apply the rules that have been examined by other researchers. I agree with the two rules of thumb above (10 obs for each var. and the formula by Harrell). Here, there is another question that the data are revealed or stated preference. Hosmer and Lemeshow in their book have provided a rule for revealed and Louviere and Hensher in their book (The methods of Stated preference) provided a rule for stated preference data

$\endgroup$
  • 2
    $\begingroup$ This would benefit from a fuller explanation and also complete and precise references. $\endgroup$ – Nick Cox Jul 20 '13 at 22:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.