Bootstrap based bias correction

Question

Assume we have a probablistic model $f_{\theta}(x)$ and try to estimate the parameter $\theta$ based on data $x$ with some procedure that yields a biased estimator $$E[\hat{\theta}]=\theta + \eta,$$ where $\eta$ represents the bias.

As far as I know we can estimate the bias $\eta$ via bootstrap, applying some procedure as follows:

1) Run the model in order to obtain $\hat{\theta}$.

2) Draw $B$ bootstrap samples $X_1,...,X_B$ and re-estimate the model $B$ times in order to obtain paramter estimates $\hat{\theta}_1,...,\hat{\theta}_B$.

3) Estimate the bias via $\hat{\eta}=\hat{\theta}-(n^{-1}\sum_{i=1}^B\hat{\theta}_i-\hat{\theta})$.

However, this procedure implies that the estimate of the bias is $unbiased$ itself. I am, however confronted with a situation where $$E[\hat{\theta}]=\theta + \eta(\theta),$$ and hence the bias a function of the unkown $\theta$ we cannot obtain valid estimates for $\eta$ since simply pluggin-in $\hat{\theta}$ for $\theta$ does not work as we cannot hope to appoximate $\eta(\theta)$ with $\eta(\hat{\theta})$. Do you have any suggestions how to solve such a problem (at least approximately)?

Answer 1

Validity of bootstrapped estimates of parameter values depends on a critical assumption about the distribution of estimates. As AdamO put it:

But one important assumption is that such a distribution is pivotal. This means that if the underlying parameter changes, the shape of the distribution is only shifted by a constant, and the scale does not necessarily change. This is a strong assumption!

That fundamental assumption does not hold in your case. In fairness, though, that problem is inherent in all non-parametric bootstrapping. As Davison and Hinkley put it on page 33:

In nonparametric problems the situation is more complicated. It is now unlikely (but not strictly impossible) that any quantity can be exactly pivotal.

Thus much depends on how far your term $\eta(\theta)$ removes the distribution from being pivotal: providing not just a change of location, which would be OK, but also of scale/shape.

Pivot plots, described by Canty et al, display how the distribution of bootstrapped estimates changes with the value of the parameter being estimated. They can be used to evaluate the magnitude of the issue and to try to find transformations of data that can bring you closer to pivotal.