4
$\begingroup$

In logistic regression and cox regression, a general recommendation is to include a maximum of one independent variable (IV) for every 10 events in order to avoid overfitting. I have seen some studies which recommend that it might be acceptable with a lower figure than 10, but that's not the point here, so for the sake of discussion, we can assume 10 events per IV as the rule.

Does this also apply to the other independent variables? Consider a situation where we have n = 1000, and a dependent variable with 40 events. This should allow us to include 4 IVs according to the rule above. But what if one of the IVs have very few events, say only 15 or so? To clarify, perhaps we have a binary IV on whether the subject has had a non-fatal overdose or not, which is not very common in our sample. Only 15 out of 1000 individuals in the sample has had an non-fatal overdose. Intuitively, it seems to me that the requirement of 40 events should apply to the IVs as well, but I may be wrong.

So what does the expertise say? Is an IV with very few events a problem that restricts the use of that IV in this way, or are we free to use the IVs regardless of number of events, as long as we stick to 10 events of the DV per IV included?

$\endgroup$
2
  • $\begingroup$ It appears as if you may be using the word "cases" in two different senses in your question, once to refer to the number of rows of the design matrix and once to refer to the number of '1's for an indicator variable. If that's not the case, could you clarify what you do mean by "case" in your question? $\endgroup$
    – Glen_b
    Dec 7, 2015 at 23:48
  • $\begingroup$ I'm talking about cases in the sens of events, the number of '1's. To my understanding, the rule of 10 refers to number of events. I'm sorry about the confusion. $\endgroup$
    – JonB
    Dec 8, 2015 at 7:58

3 Answers 3

2
$\begingroup$

On the average 15 events per candidate variable is a good rough rule of thumb. This applies to both model building and to parameter estimation, hence the use of the word candidate. But you need 96 observations just to estimate the intercept in a binary logistic model such that the margin of error in the predicted risk with 0.95 confidence does not exceed $\pm 0.1$.

A more recent paper here suggests 20:1 events:candidate variable, and a much higher figure when machine learning is used.

$\endgroup$
2
  • $\begingroup$ The link is broken for me. What paper is that for the 20:1 rule of thumb? $\endgroup$ Mar 28, 2021 at 8:09
  • 1
    $\begingroup$ van der Ploeg, T., Austin, P.C. & Steyerberg, E.W. (2014). Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol 14, 137. doi.org/10.1186/1471-2288-14-137 $\endgroup$ Mar 28, 2021 at 13:35
0
$\begingroup$

There's a few thing's I'd try:

  1. I'd try bootstrap with those n=1000 and 4 IVs. I may find that the regression equation is very unstable, especially for the coefficient for that binary IV. If the equations are unstable, it could indicate overfitting.
  2. I can look at the 2x2 table between the DV and the binary IV. If for example, most of the 15 cases of fatal overdose had an event in the DV, then that IV may be crucial in predicting the DV. However, I think it is unlikely to happen in usual cases and I suspect that IV is nearly useless.
  3. I can compare the improvement in the model's ability to predict the DV events by calculating the sensitivity and specificity of classifications with and without that IV, using the jackknife approach.
$\endgroup$
6
  • $\begingroup$ I'm talking about something like n = 100, number of events in the DV = 40 and number of '1's in one of the four IVs = 15. $\endgroup$
    – JonB
    Dec 8, 2015 at 7:59
  • $\begingroup$ Read the question now. Perhaps it's more clear. $\endgroup$
    – JonB
    Dec 8, 2015 at 19:44
  • $\begingroup$ Ok. I made changes $\endgroup$
    – Hotaka
    Dec 10, 2015 at 21:29
  • $\begingroup$ Thank you. You don't really answer my question, but perhaps there is no clear answer? I have a grasp on what bootstrapping is but not how to use it. And jackknife is a term I've only heard at some occasion without understanding it at all. Would you mind explaining the bootstrap and the jackknife methods briefly and how they relate to my question? I would also be happy if you could provide some R code if you work in R. $\endgroup$
    – JonB
    Dec 10, 2015 at 21:44
  • $\begingroup$ As for (2), if n = 1000, the DV has 40 events and the IV has 15 '1's, it's enough if three of the cases with a '1' on the IV has an event on the DV for the p-value to be below 0.05 (p = 0.02). In this case, I wouldn't agree that it's useless, but the chance that it will be useful in the regression model is low. How can I determine whether to keep it or not? $\endgroup$
    – JonB
    Dec 10, 2015 at 21:51
0
$\begingroup$
  • Hotaka has the right approach I think. Bootstrapping is a great way of looking at the stability of model.
  • This is a perhaps subtle distinction, but the 10/variable rule of thumb is meant to guide model building rather than variable estimation.
  • If you have very few outcomes for a variable - the model will give you large SE/CI for the coefficient estimate for that variable. You can fit the model and the estimates are accurate, the estimates may just not be useful given the lack of data. This is different from when you try to fit too many variable to a model, where the estimates may be bias, rather than accurate but not useful.

* - if this is of importance you could run a simulation to see if your model is reasonable. I wouldn't trust either my coding or statistical know how. But assuming only one binary predictor looking at parameter estimate. here the true value is 1.5-->

overfit2<-function(n){
# Parameters
      beta0 <- -3.218876
      beta1 <- 1.5

      # Initialize
      beta0hat<-rep(NA, 100000)
      beta1hat<-rep(NA, 100000)
      event<-rep(NA, 100000)
      outcome<-rep(NA, 100000)
      both<-rep(NA, 100000)
      # Simulation
      for(i in 1:100000)
      {
        #data generation
        prev<-runif(1, min=0.005, max=0.1)
        x1 <- rbinom(n=n, size=1, prob=prev)
        linpred<-beta0+x1*beta1
        pi <- exp(linpred) / (1 + exp(linpred))
        y <- rbinom(n=n, size=1, prob=pi) 
        data <- data.frame(x1=x1, y=y)
        data$both<-ifelse(x1==1 & y==1,1,0)
        #fit the logistic model
        mod <- glm(y ~ x1, family="binomial", data=data)
        #save the estimates
        beta0hat[i] <- mod$coef[1]
        beta1hat[i] <- mod$coef[2]
        event[i]<-sum(x1)
        outcome[i]<-sum(y)
        both[i]<-sum(data$both)
      }
      data.frame(x1=beta1hat, event=event, outcome=outcome, x0=beta0hat, both=both)
    }

xx<-overfit.mod(1000)
xx<-xx[which(xx$both>0),]

I'm just looking at parameter estimates. But you could include coverage as well.

p1<- ggplot(xx, aes(factor(event), x1))+geom_boxplot() 
p1+ geom_hline(yintercept=1.5)+coord_cartesian(xlim=c(0,40), ylim = c(0, 4))

enter image description here

$\endgroup$
7
  • $\begingroup$ Thanks. I'm not sure I understand your distinction between model building and estimation. $\endgroup$
    – JonB
    Dec 8, 2015 at 19:45
  • $\begingroup$ Are you saying that few events for an IV doesn't matter, other than that the standard errors will be large? And if so, why do too few events of the DV lead to estimation bias and too few cases of an IV lead "only" to large standard errors? Shouldn't few events of an IV lead to biased estimates (for that IV) as well? $\endgroup$
    – JonB
    Dec 9, 2015 at 20:46
  • $\begingroup$ (1) Not sure I'm making sense. (2) I would use "prevalence" of the IV. I think this is what you mean. (3) this is a fair review: aje.oxfordjournals.org/content/165/6/710.full. (4) But in practical model building this is less often an issue: the confidence intervals/SE expand and fairly obvious that you shouldn't put much weight on finding. often model won't converge when you have outcomes are rare as above $\endgroup$
    – charles
    Dec 9, 2015 at 21:46
  • $\begingroup$ (5) you might want to focus question: are you concerned about parameter estimate, model stability, or predictive ability? $\endgroup$
    – charles
    Dec 9, 2015 at 21:52
  • $\begingroup$ Thank you. (2): I'm talking about the absolute number of '1's, regardless of number of rows in the data, so I don't mean prevalence. (3): Thanks, I've read it before (in fact, this was one of the papers I was thinking about previously) but to my understanding, it only concerns low numbers of events in the DV, not in the IV. (4): I don't have the impression that models don't converge, I have yet to encounter that and I have done quite a few regression models on a DV with few events (and thus restricted the number of IV's accordingly). $\endgroup$
    – JonB
    Dec 9, 2015 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.