12
$\begingroup$

I am trying to understand the intuition behind the wild-bootstrap. What is it actually doing? I need to be able to understand what it is trying to do compared to a conventional regression.

My data has heteroskedasticity, and the method I use does 5000 replications.

How does it generate 5000 extra data?

$\endgroup$

1 Answer 1

13
$\begingroup$

Let's say you have a training set $\mathcal{T}$ of $n$ example pairs $(y_i, \vec{x}_i)$.

A normal bootstrap is a set $\mathcal{B}$ of $n$ example pairs $(y_{r_i}, \vec{x}_{r_i})$, where $r_i$ is a sequence of $n$ random integers sampled uniformly from 1 to $n$. In particular, note that every example in $\mathcal{B}$ is exactly the same as one of the examples from $\mathcal{T}$, and some are repeated. But this is a little weird, especially when the response variable is continuous, because if we re-sampled the original population, we would almost surely not get even one exact duplicate, while a bootstrap is likely to have many.

To avoid duplicates, we need the examples of $\mathcal{B}$ not to be carbon copies of examples from $\mathcal{T}$, but rather synthetic examples that look more like what we would get it we sampled from the original population. This requires making an assumption about the distribution of the original population.

If we assume homoskedasticity and fit a linear model to $\mathcal{T}$ which has residuals $e_i$ then we can construct new, synthetic examples by replacing the fitted residual from each example with the residual from a different training example. If the residuals are truly i.i.d., there should be no problem swapping one for another. We do this replacement by subtracting the residual found for the training example $(y_i, \vec{x}_i)$ and adding the residual for some other example:

$$ y^*_i = y_{r_i} - e_{r_i} + e_{r'_i} \tag{1} $$

Where $r_i$ and $r'_i$ are two different and independent resamplings. We can then form the bootstrap in the usual way:

$$ \mathcal{B} = \{\, (y^*_i, \vec{x}_i)\, \}_{i=1}^n \tag{2} $$

This is called the residual bootstrap and can be thought of as choosing new residuals from the empirical distribution function of residuals.

To relax the i.i.d. and homoskedasticity assumptions further, we can use a wild bootstrap, where we calculate the new response variable even more randomly by multiplying the randomly chosen residual by yet another random variable $v_i$.

$$ y^*_i = y_{r_i} - e_{r_i} + v_i e_{r'_i} \tag{3} $$

Often the standard normal distribution $ v_i \sim \mathcal{N}(0, 1) $ is used but other choices are possible. For example, sometimes $v_i$ is simply chosen with equal probability from $\{-1,1\}$, which simply randomly flips the sign half the time, forcing the residual distribution to be symmetric. The point is to get training examples which are closer to what we would have drawn from the original population without the artificial replication introduced by the bootstrap.

$\endgroup$
4
  • $\begingroup$ So basically, we generate errors that behave in the same as the actual residuals and then get actual data that performs in the same way as actual data? Any textbooks to recommond? $\endgroup$ May 17, 2019 at 7:37
  • $\begingroup$ then what, what do we do with all this extra data? How do we caluclate the t-statistics etc? $\endgroup$ May 27, 2019 at 12:26
  • $\begingroup$ @FrancisOrigi no doubt not needed at this point but in case it helps others: You use these synthetic datasets to perform whatever inferential tasks you want. For example, compute the variance of your estimator across synthetic datasets and use that estimated variance to compute t-statistics, calculate p-values, generate confidence intervals, etc $\endgroup$ Nov 3, 2023 at 13:48
  • $\begingroup$ I believe @olooney's response is incorrect. The wild bootstrap involves rescaling the actual residual for observation $i$ (or the residual under a restricted model in which a coefficient is fixed) by a random factor. If it were "a randomly chosen residual" as in the residual bootstrap, then the wild bootstrap would not relax the homoskedasticity assumption as claimed. $\endgroup$ Nov 3, 2023 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.