4
$\begingroup$

Although reading quite a bunch of books, I'm still not sure, which method to use and how to implement it, therefore I appreciate any help!

I have 4 different groups (treatments) with 50 participants each. Each participant's action is observed 5 times under the same condition. The 5 different values are collected within one "round", where each participant has to select 5 items at an arbitrary time. So I know that the different values for each participant are not independent of each other. However, time effects are not of interest for me and I don't assume that learning effects are apparent. I just collect 5 values for each participant in order to get more data. All the data within one group (for all users and their choices) is then taken together and a complex aggregated measure (a number) is calculated for all groups. Now the question is, whether there are significant differences for this measure between the groups. As there is no unbiased estimator for the measure and as my data is not normal, I know that I need some form of sampling. So the Welch or Levene tests, for instance, would not work here.

I'm just not sure whether bootstrapping or permutation tests seem to be more appropriate. Second, could I "ignore" the repeated measures and permute all values together without regard to the participants who provided it? If not, am I supposed to create vectors for each participant and permute these? What happens if I have an unbalanced design (in case some observations are missing)? Do you know any software where such a method is already implemented?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

If you permute the individual values then you are testing the combined null hypothesis that there is no difference between groups and that there is no structure within values from the same individual. So if you reject the null you don't know if it is because the groups differ or because there is structure within an individual.

I would suggest permuting individuals and keeping all values with the individual.

I don't know of a program that does this out of the box, but writing a permutation test in R is often easier/quicker than figuring out a precanned program.

Here is a simple example:

library(nlme)

Odist <- split(Orthodont$distance, Orthodont$Subject)

out <- replicate( 1999, { tmp <- sample(Odist);
    mean( unlist( tmp[1:16] ) ) - mean( unlist( tmp[17:27] ) ) } )

out <- c(out, mean(unlist(Odist[1:16])) - mean(unlist(Odist[17:27])) )

hist(out)
abline( v=out[2000] )

mean(out >= out[2000])
Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you so much for your help! As I'm not so familiar with the R syntax, this works only for fixed group sizes, doesn't it? Is there also an existing function for comparing the permuted values with the observed one and giving back the p-value? $\endgroup$
    – Pat
    Commented May 13, 2011 at 22:32
  • $\begingroup$ The above would work for unequal numbers of measurements per subject. In the above code it would weight each observation equally, but it could be rewritten to weight individuals equally (replace unlist with sapply and mean). As long as you can write a function to compute your complex aggregate measure using unequal sizes, this idea will work. $\endgroup$
    – Greg Snow
    Commented May 16, 2011 at 14:23
1
$\begingroup$

You can do permutations tests of both your within and between tests. Just make sure that you permute your values nested within your participants. So, if I'm a participant you can permute all the within conditions I ran in, but it's still all my data. Then you can take each participant and permute them through the between conditions.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks a lot for your help! That's a very interesting idea, I didn't know that this is also allowed. Just by chance, do you have a reference in mind who did such a double between/within permutation? $\endgroup$
    – Pat
    Commented May 13, 2011 at 22:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.