0
$\begingroup$

When I run a regression analysis in SPSS, one of my predictor variables just fails to reach significance, p = .06. When I apply bootstrapping, the output tells me the predictor has a significant effect p =.012, albeit the confidence intervals contains zero. My sample is rather small: N = 49, and the analyses includes five predictors and an interaction between two of them (so six predictors total).

My questions is, what is the justification for (not) using bootstrapping when applying regression analyses? Are the bootstrap results always superior to the 'normal' output you get from SPSS?

$\endgroup$

1 Answer 1

4
$\begingroup$

There is randomness in the bootstrap procedure. So my first step after seeing such a change in results would be to increase the number of replications and see if it remains stable (I suspect not).

Also how are the p-values and confidence intervals computed? You can have bootstrap test, but bootstrap confidence intervals are much more common. Most often when I see software reporting p-values they don't report the bootstrap test, but an approximation assuming a normal sampling distribution for the statistics in question, which defeats the very purpose of a bootstrap test. When in doubt I would look at the confidence interval (after first increasing the number of replications).

In general, you have less than 10 observations per variable. So you should be prepared to live with non-significant results. No amount of statistical trickery can change that.

$\endgroup$
3
  • $\begingroup$ Any signal that a predictor is not needed in a model can be treated as good news too: a simpler model than you feared appears adequate. $\endgroup$
    – Nick Cox
    Commented Oct 21, 2014 at 10:36
  • $\begingroup$ Thanks Maarten and Nick for your replies. The associated confidence interval is around [.029,.404], using either 1000, 2000 or 5000 replications. So it seems that the bootstrap results are rather stable in indicating the predictor is significant. The general question I have is what the justification is for (not) using these bootstrap results over the general results you get from SPSS. Should one always go for the bootstrap results given that they provide a more accurate test? $\endgroup$
    – Frank
    Commented Oct 22, 2014 at 0:06
  • $\begingroup$ There is never a method that solves all problems. The bootstrap is good, but it can fail. If you have different results, I would try to track down why that is the case. If you know that, you can make an informed decision. Basically this involves a lot of model checking and a lot of reading. $\endgroup$ Commented Oct 22, 2014 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.