1
$\begingroup$

I am attempting to do a logistic regression bootstrap with R. The problem is I get high SE's. I'm not sure what to do about this or what it means. Does it mean that bootstrap does not work well for my particular data? Here is my code:

get.coeffic = function(data, indices){
  data    = data[indices,]
  mylogit = glm(F~B+D, data=data, family="binomial")
  return(mylogit$coefficients)
}

Call:
boot(data = Pres, statistic = logit.bootstrap, R = 1000)

Bootstrap Statistics :
       original      bias    std. error
t1* -10.8609610 -23.0604501  338.048398
t2*   0.2078474   0.4351766    6.387781

I also want to know that after bootstrapping, how would this help with my final regression model? That is, how do I find what regression coefficient do I use in my final model?

> fit <- glm(F ~ B + D , data = President, family = "binomial")
> summary(fit)
Call:
glm(formula = F ~ B + D, family = "binomial", data = President)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.7699  -0.5073   0.1791   0.8147   1.2836  

Coefficients:
             Estimate Std. Error z value Pr(>|z|)  
(Intercept) -14.57829    8.98809  -1.622   0.1048  
B             0.15034    0.14433   1.042   0.2976  
D             0.13385    0.08052   1.662   0.0965 .
- --
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 23.508  on 16  degrees of freedom
Residual deviance: 14.893  on 14  degrees of freedom
AIC: 20.893

Number of Fisher Scoring iterations: 5
$\endgroup$
6
  • $\begingroup$ How many lines do you have? Could you please also add the output of 'summary' of the original model? Maybe the reason is perfect separation. $\endgroup$
    – Michael M
    Aug 3 '15 at 20:43
  • $\begingroup$ Hi Michael, please see edit. Also, could you explain "perfect separation"? Thanks! $\endgroup$
    – ali.hash
    Aug 3 '15 at 20:51
  • $\begingroup$ The bootstrap should not be used to tell you which coefficients to use in the final model. The final model should be the model that you pre-specify, if you want to preserve all aspects of statistical inference and minimize bias. $\endgroup$ Aug 3 '15 at 21:19
  • $\begingroup$ Are B & D highly correlated? The model as a whole is clearly significant 1-pchisq(23.508-14.893, 2), # [1] 0.01346718. $\endgroup$ Aug 3 '15 at 21:22
  • 1
    $\begingroup$ Bootstrap can help one to estimate bias and how much a predictive method falls apart. It is not generally used to devise new estimators. It is especially not good for variable selection unless you use a double bootstrap and you also have no collinearity. $\endgroup$ Aug 4 '15 at 17:50
2
$\begingroup$

I don't follow your code, you call your data different things in different places, I don't see your function being used anywhere, etc. Setting that aside, I'm not sure there is a big problem with your model other than the fact that you don't have much data (I gather N = 17, which is pretty small). I don't think your standard errors would be that problematic if you had a more typical sample size.

Moreover, your model seems impressively good to me for a logistic regression model with so few data to work with. The reason neither variable is significant is clearly because they are correlated. This will expand your SEs, but wouldn't be bad if you had more data. As it is, your SEs are about one third larger than they would have been if your data were perfectly uncorrelated:

1/(1-.49^2)
# [1] 1.315963

That means the model doesn't know which of the two variables should be given credit for predicting the response. Nonetheless, there is good predictive ability amongst those variables somewhere, as can be seen by their combined significance:

1-pchisq(23.508-14.893, 2)
# [1] 0.01346718

As far as bootstrapping goes, it is used to get an estimate of the nature of the sampling distribution that doesn't rely on assumptions about normality. It may help you to read this excellent CV thread: Explaining to laypeople why bootstrapping works.

$\endgroup$
6
  • $\begingroup$ I was also ending up with N = 17 (didn't get an answer to my question above)... (+1) for your answer. I was thinking why the bootstrap SE are so much larger than the original ones. Maybe it is because the number of 0 or 1 is so small that we have almost perfect separation in some bootstrap samples (with huge corresponding SE). Clearly I would not trust a logistic regression with 17 obs and 3 parameters very much. And absolutely mistrust the bootstrap results. I don't know why people think that bootstrap is a solution to small samples. It is not! $\endgroup$
    – Michael M
    Aug 4 '15 at 11:25
  • $\begingroup$ Thanks @MichaelM. Bootstrapping may not work, but then, nothing may with insufficient sample sizes. I don't think separation is likely to be the issue here, though. Neither the estimated coefficient nor the SE are that large (they would tend towards infinity) & the number of Fisher Scoring iterations is only 5 (you often have ~20 or more w/ separation). I think this is just somewhat correlated variables w/ very few data. $\endgroup$ Aug 4 '15 at 13:13
  • 2
    $\begingroup$ @MichaelM, " I don't know why people think that bootstrap is a solution to small samples". I thought bootstrap would help with small samples. I've read examples where people use to estimate regression coefficients for their models (which is what I am trying to do). Could you explain why that is wrong? Thank you. $\endgroup$
    – ali.hash
    Aug 4 '15 at 20:52
  • 2
    $\begingroup$ @ali.hash, to understand the bootstrap, it may help you to read the thread I linked in my answer. The bootstrap takes your data as an estimate of the population. If your dataset is too small, it is impossible for it to do a decent job of reflecting the population. $\endgroup$ Aug 4 '15 at 22:03
  • 1
    $\begingroup$ Bootstrapping isn't for the point estimate of your coefficients; it is for the SE / confidence intervals. $\endgroup$ Aug 4 '15 at 22:14
2
$\begingroup$

The problem here is the extremely small sample size (n=17). When you bootstrap two things can occur:

  1. You are going to get bootstrap samples with fewer ones for the dependent variable
  2. Therefore, in most samples your logistic estimates will be unstable—Resulting in very biased estimates

This might explain why you get huge standard errors. Tip: Check the distribution of ‘1’s in some of the bootstrap samples. With N=17,even the normal logistic is not guaranteed to converge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.