The main thing is study design and perhaps less statistics.
Your data will probably result try to answer the following questions:
- Is interface A easier to understand than interface B
- Does experience with a previous interface improve the second run
- Does it matter if you tried system A first and then went on to system B
As I understand it you want to answer the first question but are worried that the 2 & 3 will blur the data. Now I don't think that there is any good way to see how you can explain that 3 is not a part of 1 and I would suggest that you avoid the cross-over design of your study since you lack knowledge of that part. In a real world setting the users will also only have one interface to relate to.
Your outcome variables are
- Time to complete a task or a group of tasks
- The experience that the user has with the system
It's always good practice to define which of the outcome variable that is your primary and that's the one you should do use for all your power calculations. If you have several tasks to compare I would suggest you create some compound variable (preferably that makes intuitive sense).
When choosing the subjective experience you should look for validated questionnaires. It is always good to use an already existing questionnaire and in medicine we very often use scores like EQ-5D that we have previously validated and most are familiar with. There are probably similar that perhaps can be used in your case. Scores have one nice feature and that is that you can calculate an average but this also is a disadvantage because an improvement of 10 points means nothing if you can't relate to the score. In EQ-5D we often compare surgery results to the general population and see how close we get with our interventions.
I would do a randomized trial without the cross-over. The important thing with randomization is... it has to be random! Yep, there are plenty of people peeping in the envelopes ruining their own experiments so please make sure you have a good randomization procedure:
- Use computer randomization or opaque envelopes
- Use blocked randomization so that you aim for equal group size (each block having 50% group A and 50 % group B)
- Use random block size, if you have sizes of 2, 4 & 6 you'll have a hard time knowing what interface will be next
- Only use stratification for 1 or maybe 2 variables, for instance gender, computer experience
- Randomize late, preferably when the subject is sitting by the computer
If word gets around about the systems you might want to have some check for previous knowledge of the systems. If you enroll your classmates you might have told them vital parts of your system by accident ruining the experiment. For obvious reasons you can't blind your subjects but you should try to do everything in your power to keep the subjects unknowing of what they're about to experience.
If your randomization works you don't need to worry about the confounders. Even though the theory says you don't need to most note down possible confounders for their subjects in case something doesn't work out with the randomization. If you have very small groups, less that 20-30 subjects, you might want to do the same.
Typical confounders in your case are probably:
- Previous computer experience
- Educational level
Calculating power (estimating number of subjects needed) is easy unless you want to go into details and I did my first calculations with Russ Lenth's power calculations that you can find here. You could also use R's package "pwr", you can find some help on that here.
Power calculation is a very rough estimate and you should always add 10-20 % for drop-outs. In medicine we use a significance level of 0.05 and aim for a power of 80% (0.8) by tradition but choosing interface is perhaps not as critical as choosing drug A for a cancer patient and therefore can allow for a significance of only 0.1.
You have to guess a lot when trying to calculate power which is one of the reasons that you should look at the numbers as guidance more than the truth. I also think that if you need more than 60-80 subjects per group then the difference that your looking for is probably very small and that is perhaps OK if you're looking for a better heart medication but less interesting if your designing interfaces.
If you have a well designed study this part is the least of your worries. I would say that most of the flaws are in study design and statistic measures differ usually in the decimals while the conclusions usually stay the same.
Tests for randomized trials:
For testing confounders:
- Regression analysis is an extremely powerful method that really allows you to do almost anything.
For learning the basics behind statistics I use Khan Academy that I also recommend to all my students.