0
$\begingroup$

I'm hoping I can get some help with the design of a study, I've looked at threads on similar topics but haven't found exactly what I need. We will be showing 6 videos to participants, and then asking them to rate the effectiveness of each video (so, each participant watches each video and then rates them all). They will be asked a number of ordinal questions (e.g. rate a certain quality of the video from 1 to 5, with 1 being not at all effective, and 5 being highly effective), and then asked to rank the videos from 1 to 6. Right now we have about 60 participants lined up.

My questions are as follows. I'm concerned that the order in which the participants will view the videos will affect their rankings. The natural thing is to randomize the order, but there are way more different ways to order than there are participants. It also seems unnecessarily complicated to give each a different order, so I thought of maybe breaking the group of 60 into a number of smaller subgroups and then show each subgroup the videos in the same order. Is that a natural thing to do? If so, what is a good number of subgroups, and what is a good way to construct the orderings shown to each group so that each video gets a fair chance? I remember once hearing about association schemes but I'm not sure if they are related to this question.

My other concern is the size of the study. I'm a little worried that 60 participants may be too small to meaningfully differentiate between the videos. Is there a good way to estimate the number of participants needed before starting the study?

$\endgroup$

1 Answer 1

0
$\begingroup$

My thoughts on your predicament are to randomly assign order of the videos. Then if you suspect there is an effect that is due to the order, create a second set of variables to capture this nuisance variable.

For example, say you give participant X your videos in the following order: 2, 6, 5, 3, 1, 4. But then you have a second dummy variable matrix that is a 6x6 (each place order is given a dummy variable). You then can capture the main effect of order (e.g. is the first video the person watched a statistically significant effect) as well as any interactions (e.g. watching video 3 first [or second] led to a statistically significant rise in its rating).

That is how I would do it and would probably work ok-ish given your sample size (the model cant get so complicated or it wont be able to be estimated).

$\endgroup$
6
  • $\begingroup$ Thank you for your answer. So, are you suggesting giving each participant a randomly generated order? Shouldn't I then worry about assigning the orders such that each video is viewed first by 10 out of 60 of the participants, also viewed 2nd by 10 out of 60, etc.? Because it could just happen that random choices for orders would not end up being fair across the 60 participants. $\endgroup$ Apr 25, 2017 at 6:45
  • $\begingroup$ well that would be part of your random assignment. For example, you don't really have 6 videos, but you also have their associated order. That is what you randomly assign. So, you are randomly assigning 10 first order video one, 10 1st order video two, 10 1st order video three, etc; where a participant can only get one of each order/video. That way you have a balanced design. $\endgroup$
    – JWH2006
    Apr 25, 2017 at 15:57
  • $\begingroup$ Thanks for your answers. I've explored a bit more, and I do think this is a problem for Latin squares or association schemes in combinatorics. $\endgroup$ Apr 28, 2017 at 10:22
  • $\begingroup$ yes! a latin square was exactly what I was suggesting. $\endgroup$
    – JWH2006
    Apr 28, 2017 at 12:57
  • $\begingroup$ Oh, it is? Cool, then, I guess we are on the same page. Thanks! $\endgroup$ Apr 30, 2017 at 20:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.