The key question for me is still regarding "If you don't want to infer anything about the time factor [...]" If you do want to infer something about the time factor, wouldn't this imply a possibly totally different design and/or test optimized to test time specifically? Does just because you have multiple measurements justify a repeated measures ANOVA? In my case I take 8 measurements over the course of 50 minutes, however, I do not control fully for the nature of them (they differ slightly). I take 8 measurements exactly because I want to have a meaningful mean. What if I would take 100?

Related question regarding repeated measures versus one-way ANOVA design.

In addition, don't you run into the risk of interpreting significant effects incorrectly? If the hypothesis is to expect a significant difference in the mean values, and there are only a limited amount of measurements, ... what does a significant effect even say in this case? I thought hypotheses had to be phrased up front, and tested subsequently.

This is already helpful, thanks! I am in a similar situation where I have 8 measurements over the course of 50 minutes, within-subjects (so 16 total). Therefore I am wondering whether you could elaborate on "If there is no difference between the answer order, then the whole model will be the same as taking the mean."?

NATO's technical report really helped me along. It's a great reading on the subject. Thanks!

Wow. :O Thanks for your time! Will look into it in detail tomorrow.

You are right about "the greater familiarity with the problem set", although it is my understanding this is partially reduced by using a balanced setup. However, when not using a within-subjects design, computer experience can play a major factor in the outcome no? My goal was to eliminate this by using within-subjects.