Although reading quite a bunch of books, I'm still not sure, which method to use and how to implement it, therefore I appreciate any help!
I have 4 different groups (treatments) with 50 participants each. Each participant's action is observed 5 times under the same condition. The 5 different values are collected within one "round", where each participant has to select 5 items at an arbitrary time. So I know that the different values for each participant are not independent of each other. However, time effects are not of interest for me and I don't assume that learning effects are apparent. I just collect 5 values for each participant in order to get more data. All the data within one group (for all users and their choices) is then taken together and a complex aggregated measure (a number) is calculated for all groups. Now the question is, whether there are significant differences for this measure between the groups. As there is no unbiased estimator for the measure and as my data is not normal, I know that I need some form of sampling. So the Welch or Levene tests, for instance, would not work here.
I'm just not sure whether bootstrapping or permutation tests seem to be more appropriate. Second, could I "ignore" the repeated measures and permute all values together without regard to the participants who provided it? If not, am I supposed to create vectors for each participant and permute these? What happens if I have an unbalanced design (in case some observations are missing)? Do you know any software where such a method is already implemented?