I am trying to understand the intuition behind the wild-bootstrap. What is it actually doing? I need to be able to understand what it is trying to do compared to a conventional regression.

My data has heteroskedasticity, and the method I use does 5000 replications.

How does it generate 5000 extra data?


Let's say you have a training set $\mathcal{T}$ of $n$ example pairs $(y_i, \vec{x}_i)$.

A normal bootstrap is a set $\mathcal{B}$ of $n$ example pairs $(y_{r_i}, \vec{x}_{r_i})$, where $r_i$ is a sequence of $n$ random integers sampled uniformly from 1 to $n$. In particular, note that every example in $\mathcal{B}$ is exactly the same as one of the examples from $\mathcal{T}$, and some are repeated. But this is a little weird, especially when the response variable is continuous, because if we re-sampled the original population, we would almost surely not get even one exact duplicate, while a bootstrap is likely to have many.

To avoid duplicates, we need the examples of $\mathcal{B}$ not to be carbon copies of examples from $\mathcal{T}$, but rather synthetic examples that look more like what we would get it we sampled from the original population. This requires making an assumption about the distribution of the original population.

If we assume homoskedasticity and fit a linear model to $\mathcal{T}$ which has residuals $e_i$ then we can construct new, synthetic examples by replacing the fitted residual from each example with the residual from a different training example. If the residuals are truly i.i.d., there should be no problem swapping one for another. We do this replacement by subtracting the residual found for the training example $(y_i, \vec{x}_i)$ and adding the residual for some other example:

$$ y^*_i = y_{r_i} - e_{r_i} + e_{r'_i} \tag{1} $$

Where $r_i$ and $r'_i$ are two different and independent resamplings. We can then form the bootstrap in the usual way:

$$ \mathcal{B} = \{\, (y^*_i, \vec{x}_i)\, \}_{i=1}^n \tag{2} $$

This is called the residual bootstrap and can be thought of as choosing new residuals from the empirical distribution function of residuals.

To relax the i.i.d. and homoskedasticity assumptions further, we can use a wild bootstrap, where we calculate the new response variable even more randomly by multiplying the randomly chosen residual by yet another random variable $v_i$.

$$ y^*_i = y_{r_i} - e_{r_i} + v_i e_{r'_i} \tag{3} $$

Often the standard normal distribution $ v_i \sim \mathcal{N}(0, 1) $ is used but other choices are possible. For example, sometimes $v_i$ is simply chosen with equal probability from $\{-1,1\}$, which simply randomly flips the sign half the time, forcing the residual distribution to be symmetric. The point is to get training examples which are closer to what we would have drawn from the original population without the artificial replication introduced by the bootstrap.

  • $\begingroup$ So basically, we generate errors that behave in the same as the actual residuals and then get actual data that performs in the same way as actual data? Any textbooks to recommond? $\endgroup$ – Francis Origi May 17 at 7:37
  • $\begingroup$ then what, what do we do with all this extra data? How do we caluclate the t-statistics etc? $\endgroup$ – Francis Origi May 27 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.