How to bootstrap F-statistics in a repeated measures ANOVA?

Question

I have a repeated measures design. Each subject was tested in 2 conditions (RE, re) with different cognitive load levels (low, mid, high). I originally analysed this dataset using a repeated measures ANOVA (mu_N1 ~ lang*load + Error (subj/lang * load)). This is an unbalanced design, i.e. the number of observations per each cell is slightly different (e.g. subject "KOS" has fewer observations in "ER" and "low" than, say, subject "GRU"). Now I want to bootstrap F-statistics. The question is how do I resample the data? In the simple case of comparing the means of two samples (unpaired t-test), the bootstrap is straightforward: put all the observations into one pile, then take two random samples of the original size from that pile, compute t-statistic, repeat many times. With my data, there are "nested" levels (i.e. each subject was subjected to ER and re, and within RE and ER subjected to low, mid, high. Below is a rough slice of my data (disregard the column CLred). Would it be reasonable to leave the the order of columns "subj", "lang", "load" unchanged, but only reshuffle (resample with replacement) the numbers in column mu_N1 at each bootstrap resample?

"subj" "lang" "CLred" "load" "mu_N1"
"1" "KOS" "RE" 0.21077 "low" 2.039479
"2" "KOS" "RE" 0.50301 "mid" 7.050688
"3" "KOS" "RE" 0.50301 "mid" 6.321543
"4" "KOS" "RE" 0.30664 "low" -7.347104
"5" "KOS" "RE" 0.5031 "mid" 4.850391
"1412" "KOS" "er" 0.22893 "low" -3.867834
"1413" "KOS" "er" 0.77563 "mid" -2.939469
"1414" "KOS" "er" 1.1121 "mid" 17.26057
"1415" "KOS" "er" 1.669 "mid" -4.96131
"1416" "KOS" "er" 1.9141 "high" 11.5676
"1417" "KOS" "er" 1.3287 "mid" 2.687416
"1418" "KOS" "er" 1.2408 "mid" -9.099418
"6835" "GRU" "er" 4.0121 "mid" -9.006017
"6836" "GRU" "er" 4.4279 "mid" -18.26898
"6837" "GRU" "er" 4.782 "mid" -10.48242
"6838" "GRU" "er" 4.9231 "mid" 17.72892
"6839" "GRU" "er" 5.3071 "high" -7.620954
"6840" "GRU" "er" 5.7038 "high" 0.7890218
"6841" "GRU" "er" 6.4355 "high" 3.29691
"6842" "GRU" "er" 7.0723 "high" -7.693555
"7218" "GRU" "RE" 2.0459 "mid" 2.386349
"7219" "GRU" "RE" 2.3365 "mid" -3.752826
"7220" "GRU" "RE" 2.7053 "high" -0.5608671
"7221" "GRU" "RE" 2.7053 "high" -1.798806
"7222" "GRU" "RE" 1.8141 "mid" -2.58064
"7223" "GRU" "RE" 2.2764 "mid" 2.066321
"7224" "GRU" "RE" 2.2764 "mid" 2.385393
"7225" "GRU" "RE" 1.9051 "mid" 0.9645662
"7226" "GRU" "RE" 1.907 "mid" -3.036294
"7227" "GRU" "RE" 1.907 "mid" 1.307736
"26" "KOS" "RE" 0.42917 "mid" -5.46118
"27" "KOS" "RE" 0.22734 "low" -15.93142
"28" "KOS" "RE" 0.35941 "low" -11.60953
"29" "KOS" "RE" 0.62127 "mid" -16.10649
"30" "KOS" "RE" 0.62127 "mid" -4.435908

Answer 1

In general, probably not. The proper procedure is going to depend on the assumptions you are comfortable making about what is independent. In this case, I suspect that you would be comfortable stating that individuals are independent of eachother, but that within individuals things are (at least potentially) related to eachother.

If that is the case, then you should resample at the level of individuals rather than at the level of responses. That way, the relationship within individuals is preserved, and the randomness that is a result of the specific sample you got from the population will be approximated by the resampling.

Answer 2

That is a permutation test, not a bootstrap. They are both resampling statistics. They both provide good finite-sample inference that relies on relatively few assumptions.