How to interpret Bootstrap?

Question

I'm a real newbie when it comes to statistics so please don't judge me and my question ;)

I'm doing a linear regression analysis with SPSS and since my data is neither normally distributed nor shows homoscedasticity, I decided to use bootstrapping.

Now, I'm really confused when it comes to the interpretation of the output. SPSS offers me the "normal" model summary and coefficients as well the bootstrap summary and bootstrap coefficients. Do I now, only interpret the bootstrap part? Or is the F-value for example still relevant, meaning that if F is not significant, I also can't interpret the bootstrap interval even though it is significant?

Answer 1

The intuitive idea behind the bootstrap is this: if your original dataset was a random draw from the full population, then if you take subsample from the sample (with replacement), then that too represents a draw from the full population. You can then estimate your model on all of those bootstrapped datasets. This gives you a large number of estimates and so you can e.g. look at the standard deviations of your estimates - it turns out that often this gives a good guess of the standard error of the estimates. Actually, the standard error of the estimates can be thought of excactly as this if you take the many datasets from the true population.

Suppose for example there is one outlier in your dataset: then in many of your bootstrapped datasets that observation is not included and so for those datasets, you see the estimated coefficients change by a lot.

Similarly, you can look at the F statistic for each of the bootstrap datasets. You could for example see how many times the model was rejected. But I am not sufficiently familiar with SPSS to know what it reports as the F stat: is it the average F statistic?

Answer 2

As @Superpronker mentioned it really depends on what SPSS is doing with the bootstrap. Including your code and the output would help a great deal. Also the bootstrap is a subject with a vast amount of literature. You could see this by simply looking at the bibliography in my 2007 edition of Bootstrap Methods published by Wiley. So I think you really also need at least a basic tutorial on the bootstrap. Sometimes going to Wikipedia helps with this sort of thing.

In regression there are various ways to deal with issues like heteroskedasticity and non-normality. If the F test you refer to is from the OLS solution to linear regression where normality and homoskedasticity is ignored and by non-significance you mean that the F test can't tell you that any of the regression coefficients are different from 0, it may be that you should just ignore it and apply a different approach.

The bootstrap can be one approach to deal with the problem. In regression there are two common bootstrap approaches. One is called bootstrapping residuals and the other is called bootstrapping vectors. You should want to find out which one SPSS is using. There is some literature that says bootstrapping vectors is more robust in the sense that it requires fewer assumptions. The vector is the set of observed values of $(Y, X_1, X_2, \ldots, X_k)$ where $Y$ is the dependent variable and the $X_j$ are the $k$ predictor variables in your model. From your problem description we do not know if $k$ is $1$ or $>1$. For each $j$ there is associated with $X_j$ a regression parameter $b_j$ that is estimated.

The bootstrapping residuals method takes the $n$ residuals, where $n$ is your sample size, and it samples with replacement from this set of residuals. In the computer program this is done by the Monte Carlo method.

The model is $Y=b_1 X_1 + b_2 X_2 + \ldots + b_k X_k +e$ where $e$ is an error term. You initially get n residuals by taking $y_i - \hat{b}_1 x_{1i} - \hat{b}_2 x_{2i}- \ldots -\hat{b}_k x_{ki}$ to be the $i$th residual. Here $\hat{b}_j$ denotes the estimate of the regression parameter $b_j$. We use the notation $y_i$ and $x_{ji}$ to represent the $i$th observed value of the dependent variable and the $i$th observed value of the $j$th predictor variable respectively.

As this gets complicated, I suggest you look at a reference on bootstrapping residuals The 1993 Chapman and Hall text by Efron and Tibshirani is one possibility. The end results are bootstrap distributions for each regression parameter and one of several possible bootstrap confidence intervals could be used. Efron's percentile method is the most likely possibility. If the confidence interval does not contain 0 the regression parameter is considered significant.

Answer 3

As a quick summary, the general bootstrap in SPSS Statistics is described thusly in the help.

The Simple method is case resampling with replacement from the original dataset. The Stratified method is case resampling with replacement from the original dataset, within the strata defined by the cross-classification of strata variables.

Some procedures have other options.

The Algorithms manual, which is available online, covers details for jackknife, case, stratified, residual, and wild resampling.

As for the user's original question, the question says "my data is neither normally distributed nor shows homoscedasticity", which could reflect a misconception about what the normality assumption means in regression. It is about the error term, not the variables in the equation.

And a question for Michael: your books on bootstrapping are priced on Amazon for Kindle from 107 to 237 dollars! Why? I'd love to read one of these, but the cost is phenomenal. Unfortunately, I don't have a good library as an alternative to purchasing.