Varied experimental conditions in repeated measure designs

Question

I'm running an experiment with an e-learning system using a repeated measures with crossover design.

Subjects are shown two videos on a web page. In one session, a feature (a table of contents) is enabled, while the other session is a control:

C=Control F=Feature
subject     pretest1   video 1   posttest1    pretest2   video 2    posttest2
S1                        C                                  F
S2                        C                                  F
S3                        C                                  F
S4                        F                                  C
S5                        F                                  C
S6                        F                                  C

The amount learned is measured by subtracting pre-test result from post-test result.

My issue is that I can't swap around the order of the two videos for the crossover because Video 2 requires knowledge of Video 1. If I had chosen videos on separate topics I would be introducing variability in aptitude, interest and prior knowledge within the subjects.

It is also unlikely that both videos have equal difficulty, so the learning gains in one can't be directly compared to the learning gains in the other. I know that the crossover will cancel out this effect and allows me to get a mean difference, but this seems to prevent me from doing basic EDA like scatterplots, and I'm not sure how to proceed with statistical analysis from there.

Is there anything that I can do to improve this experiment design, and do you have any advice or resources on how I should proceed with analysis in this scenario?

Answer 1

It looks like you have potential confounding factors that you need to account for in your design. While the suggestion would make the need for a larger sample size, the following design would move you forward in addressing these confounders:

Do a 2x2 matrix design (sorry I'm new so I can't post a pic), so you actually have 4 conditions:

1. NFV1, NFV2
2. NFV1, FV2
3. FV1, NFV2
4. FV1, FV2

You may still have to answer the question of how close pre- and post- tests are to each other. A way to address close (in time) pre- and post- tests are to ask different questions but they address the same concepts....but those often need to be checked with an SME and can bring their own difficulties if not done correctly