I am running a logistic regression, similar to the following:

Pr(Y = 1) = B0 + B1*X1 + B2*X2 + B3*X3 + e

X1 is an indicator variable. I find B1 is statistically insignificant. A colleague is worried that B1 may be insignificant because I have too few observations where X1 = 1 (about 0.50% of my sample of 10,000 observations). His suggestion is to randomly draw control observations to match the number of treatment observations and rerun the regression.

Alternatively, could I address this concern by simply bootstrapping my standard errors in my original regression and sample? Is bootstrapping a valid way to address power concerns?

  • 3
    $\begingroup$ The formula you posted in the code block isn't a logistic regression, it's a linear regression. Is this really what you used? $\endgroup$ – jbowman Dec 21 '18 at 22:20
  • $\begingroup$ @jbowman I'm using logistic. That is just the convention of how it is written in my profession. $\endgroup$ – user4951834 Dec 22 '18 at 1:46

His suggestion is to randomly draw control observations to match the number of treatment observations and rerun the regression.

This is not bootstrap, but oversampling. With bootstrap, you would be sampling from your data randomly with replacement, $n$ cases out of sample of size $n$. So, with bootstrap, you would end up with same sample size and approximately similar distribution to your initial data, so bootstrap would not cure anything about insufficient sample size.

When confronted with unbalanced data (see also other questions tagged as ), oversampling (randomly sampling more observations from the minority class and appending them to the dataset) is one of the possible approaches. People use it, but the fundamental problem is that by oversampling the minority class you don't really get more data, since you are just looking at the same data multiple times. This means that you could easily overfit to the available data. So even if you could use it to virtually adjust the sample size, this does not solve your problem.

  • 1
    $\begingroup$ One way to see it is that oversampling the minority class creates dependent observations, which invalidates p-values. $\endgroup$ – Michael M Dec 22 '18 at 8:09

So just from what you have told me, a quick back of the envelope calculation shows your minimum detectable effect of 0.79. According to Cohen's 1988 paper, that is fairly large. If you think the effect of the covariate (on the log odds scale) is quite small, you may not have enough power to detect it.

My suggestion is not to bootstrap your regression. Instead, hypothesize the size of the effect and do some simulations to see if your sample size and prevalence of exposure lead to acceptable power.

Also, what are the confidence intervals like? Are the really narrow or really wide?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.