Regression models: cases per independent variable?

Question

In logistic regression and cox regression, a general recommendation is to include a maximum of one independent variable (IV) for every 10 events in order to avoid overfitting. I have seen some studies which recommend that it might be acceptable with a lower figure than 10, but that's not the point here, so for the sake of discussion, we can assume 10 events per IV as the rule.

Does this also apply to the other independent variables? Consider a situation where we have n = 1000, and a dependent variable with 40 events. This should allow us to include 4 IVs according to the rule above. But what if one of the IVs have very few events, say only 15 or so? To clarify, perhaps we have a binary IV on whether the subject has had a non-fatal overdose or not, which is not very common in our sample. Only 15 out of 1000 individuals in the sample has had an non-fatal overdose. Intuitively, it seems to me that the requirement of 40 events should apply to the IVs as well, but I may be wrong.

So what does the expertise say? Is an IV with very few events a problem that restricts the use of that IV in this way, or are we free to use the IVs regardless of number of events, as long as we stick to 10 events of the DV per IV included?

Answer 1

On the average 15 events per candidate variable is a good rough rule of thumb. This applies to both model building and to parameter estimation, hence the use of the word candidate. But you need 96 observations just to estimate the intercept in a binary logistic model such that the margin of error in the predicted risk with 0.95 confidence does not exceed $\pm 0.1$.

A more recent paper here suggests 20:1 events:candidate variable, and a much higher figure when machine learning is used.

Answer 2

There's a few thing's I'd try:

1. I'd try bootstrap with those n=1000 and 4 IVs. I may find that the regression equation is very unstable, especially for the coefficient for that binary IV. If the equations are unstable, it could indicate overfitting.
2. I can look at the 2x2 table between the DV and the binary IV. If for example, most of the 15 cases of fatal overdose had an event in the DV, then that IV may be crucial in predicting the DV. However, I think it is unlikely to happen in usual cases and I suspect that IV is nearly useless.
3. I can compare the improvement in the model's ability to predict the DV events by calculating the sensitivity and specificity of classifications with and without that IV, using the jackknife approach.

Answer 3

• Hotaka has the right approach I think. Bootstrapping is a great way of looking at the stability of model.
• This is a perhaps subtle distinction, but the 10/variable rule of thumb is meant to guide model building rather than variable estimation.
• If you have very few outcomes for a variable - the model will give you large SE/CI for the coefficient estimate for that variable. You can fit the model and the estimates are accurate, the estimates may just not be useful given the lack of data. This is different from when you try to fit too many variable to a model, where the estimates may be bias, rather than accurate but not useful.

* - if this is of importance you could run a simulation to see if your model is reasonable. I wouldn't trust either my coding or statistical know how. But assuming only one binary predictor looking at parameter estimate. here the true value is 1.5-->

overfit2<-function(n){
# Parameters
      beta0 <- -3.218876
      beta1 <- 1.5

      # Initialize
      beta0hat<-rep(NA, 100000)
      beta1hat<-rep(NA, 100000)
      event<-rep(NA, 100000)
      outcome<-rep(NA, 100000)
      both<-rep(NA, 100000)
      # Simulation
      for(i in 1:100000)
      {
        #data generation
        prev<-runif(1, min=0.005, max=0.1)
        x1 <- rbinom(n=n, size=1, prob=prev)
        linpred<-beta0+x1*beta1
        pi <- exp(linpred) / (1 + exp(linpred))
        y <- rbinom(n=n, size=1, prob=pi) 
        data <- data.frame(x1=x1, y=y)
        data$both<-ifelse(x1==1 & y==1,1,0)
        #fit the logistic model
        mod <- glm(y ~ x1, family="binomial", data=data)
        #save the estimates
        beta0hat[i] <- mod$coef[1]
        beta1hat[i] <- mod$coef[2]
        event[i]<-sum(x1)
        outcome[i]<-sum(y)
        both[i]<-sum(data$both)
      }
      data.frame(x1=beta1hat, event=event, outcome=outcome, x0=beta0hat, both=both)
    }

xx<-overfit.mod(1000)
xx<-xx[which(xx$both>0),]

I'm just looking at parameter estimates. But you could include coverage as well.

p1<- ggplot(xx, aes(factor(event), x1))+geom_boxplot() 
p1+ geom_hline(yintercept=1.5)+coord_cartesian(xlim=c(0,40), ylim = c(0, 4))