6
$\begingroup$

Consider an experiment with $m$ subjects and $n$ words. Every subject rates every word, producing an $(m \times n)$ data matrix $\hat{X}$. I am interested in forming a confidence interval around $f(\hat{X})$ where $f$ is some arbitrary scalar function. I'd like this confidence interval to take into account both the sampling error that comes from the random choice of subjects and the sampling error that comes from the random choice of the words.

What I'm inclined to do is to bootstrap both subjects and words: On each bootstrap iteration, randomly resample the words (with replacement) and the same for subjects (using the same set of resampled words for all resampled subjects). This generates a resampled data matrix $X^\ast_i$, and a bootstrap estimate $f(X^\ast_i)$. The vector of bootstrap estimates (e.g., 10,000 of them) is then used to form a bootstrap confidence interval (e.g., percentile bootstrap) as if these estimates were standard, single-factor bootstrap estimates.

  1. Is this legit or am I violating some implicit assumption of the bootstrap?
  2. Is there a more principled way of dealing with this problem?
  3. Is there any R package that implements such a procedure? I can easily write the resampling code but calculating advanced bootstrap intervals (e.g., BCa) isn't trivial. boot seems to assume a single random effect.
$\endgroup$
  • 1
    $\begingroup$ Do you have any fixed effects? $\endgroup$ – Dave Oct 4 at 15:58
  • $\begingroup$ $f(\cdot)$ measures the performance difference between two models predicting the ratings. You can think about the two models as a fixed effect. $\endgroup$ – Trisoloriansunscreen Oct 4 at 16:21
  • 1
    $\begingroup$ It seems you're assuming that ratings given to each word are independent. But, is that really the case? It seems possible that subjects might rate words differently depending on the order they're presented in, the overall collection of other words, the history of recently presented words, etc. $\endgroup$ – user20160 Oct 4 at 18:12
  • 1
    $\begingroup$ Have you looked into the different options available through the bootMer() function? rdocumentation.org/packages/lme4/versions/1.1-21/topics/bootMer $\endgroup$ – Matt Barstead Oct 6 at 12:29
  • 1
    $\begingroup$ The procedure looks legit to me. $\endgroup$ – amoeba says Reinstate Monica Oct 9 at 11:05
2
+100
$\begingroup$

I think in this case it is recommended to do a parametric bootstrap: the mixed effect model gives you an estimate of the variance of the effects of words and subjects, so you can generate new random deviates from their distribution (thus without actually resampling the estimated values). It is not difficult to write the code yourself, but if you used the lme4 package to estimate the model then I think you should be able to do it via the function bootMer. If I understood correctly your problem, you could just write a wrapper function that computes predicted values $X^*$ from the bootstrapped model and calculate the function $f(X^*)$; and pass it to bootMer. Once you have a boostrapped distribution for $f(X^*)$ you can use any methods to calculate a confidence interval (e.g. percentile). If you are interested in BCa intervals I have some code that calculate that here (it is part of a small package that contains my frequently used custom functions available here)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.