Test to compare user interfaces For my thesis I will need to do an experimental user study to compare two user interfaces which both can perform the same tasks. They differ in how they support you in achieving your task. I'm not that familiar with statistics so please correct me if any of the following assumptions I already made are incorrect.
I'm pretty sure that I'll need to use a balanced within-subjects design. Reading through several papers on user interfaces seems to indicate that long in-field studies are advised. Due to time limitations, and the very experimental design of the interface this isn't an option.
The plan is to do both objective and subjective (questionnaire) measurements. Objective measurements could be e.g. time taken to complete a given task.
At first I figured I needed a paired t-test. However, I will most likely take multiple measurements and not just one. Some reading mentioned a repeated measure ANOVA for such a scenario, but I'm not sure about this at all.
Which test should I use? Which power analysis method should I use? I want to determine how many subjects I need (or can do) and what the power of my test will be.
 A: The main thing is study design and perhaps less statistics.
Hypothesis
Your data will probably result try to answer the following questions:


*

*Is interface A easier to understand than interface B

*Does experience with a previous interface improve the second run

*Does it matter if you tried system A first and then went on to system B


As I understand it you want to answer the first question but are worried that the 2 & 3 will blur the data. Now I don't think that there is any good way to see how you can explain that 3 is not a part of 1 and I would suggest that you avoid the cross-over design of your study since you lack knowledge of that part. In a real world setting the users will also only have one interface to relate to.
Outcome
Your outcome variables are


*

*Time to complete a task or a group of tasks

*The experience that the user has with the system


It's always good practice to define which of the outcome variable that is your primary and that's the one you should do use for all your power calculations. If you have several tasks to compare I would suggest you create some compound variable (preferably that makes intuitive sense). 
When choosing the subjective experience you should look for validated questionnaires. It is always good to use an already existing questionnaire and in medicine we very often use scores like EQ-5D that we have previously validated and most are familiar with. There are probably similar that perhaps can be used in your case. Scores have one nice feature and that is that you can calculate an average but this also is a disadvantage because an improvement of 10 points means nothing if you can't relate to the score. In EQ-5D we often compare surgery results to the general population and see how close we get with our interventions.
Design
I would do a randomized trial without the cross-over. The important thing with randomization is... it has to be random! Yep, there are plenty of people peeping in the envelopes ruining their own experiments so please make sure you have a good randomization procedure:


*

*Use computer randomization or opaque envelopes

*Use blocked randomization so that you aim for equal group size (each block having 50% group A and 50 % group B)

*Use random block size, if you have sizes of 2, 4 & 6 you'll have a hard time knowing what interface will be next

*Only use stratification for 1 or maybe 2 variables, for instance gender, computer experience

*Randomize late, preferably when the subject is sitting by the computer


If word gets around about the systems you might want to have some check for previous knowledge of the systems. If you enroll your classmates you might have told them vital parts of your system by accident ruining the experiment. For obvious reasons you can't blind your subjects but you should try to do everything in your power to keep the subjects unknowing of what they're about to experience.
Confounders
If your randomization works you don't need to worry about the confounders. Even though the theory says you don't need to most note down possible confounders for their subjects in case something doesn't work out with the randomization. If you have very small groups, less that 20-30 subjects, you might want to do the same.
Typical confounders in your case are probably:


*

*Previous computer experience

*Age

*Gender

*Educational level


Power calculation
Calculating power (estimating number of subjects needed) is easy unless you want to go into details and I did my first calculations with Russ Lenth's power calculations that you can find here. You could also use R's package "pwr", you can find some help on that here.
Power calculation is a very rough estimate and you should always add 10-20 % for drop-outs. In medicine we use a significance level of 0.05 and aim for a power of 80% (0.8) by tradition but choosing interface is perhaps not as critical as choosing drug A for a cancer patient and therefore can allow for a significance of only 0.1.
You have to guess a lot when trying to calculate power which is one of the reasons that you should look at the numbers as guidance more than the truth. I also think that if you need more than 60-80 subjects per group then the difference that your looking for is probably very small and that is perhaps OK if you're looking for a better heart medication but less interesting if your designing interfaces.
Statistics
If you have a well designed study this part is the least of your worries. I would say that most of the flaws are in study design and statistic measures differ usually in the decimals while the conclusions usually stay the same.
Tests for randomized trials:


*

*T-test for continuous outcome

*Cox regression for time dependent survival outcome

*Kaplan-Meier for visualizing survival

*Chi-square test for categorical outcome


For testing confounders:


*

*Regression analysis is an extremely powerful method that really allows you to do almost anything.


For learning the basics behind statistics I use Khan Academy that I also recommend to all my students.
A: My take on this, which is admittedly biased by techniques I'm familiar with.
Study design: In your case, I think you can get away with a simple randomized design. I hesitate to have each subject try both GUI setups for two reasons: It complicates analysis, and its possible that greater familiarity with the "problem set" or whatever will influence your results for the second GUI.
Power: Sadly, this is going to entirely depend on your question. If you're really inclined to calculate power beforehand, you need to do it for each type of analysis you're going to be conducting. Each test has their own quirks, although power calculation via simulation is probably the easiest method that can apply to any test you can come up with. Though be aware that power calculations have tons of assumptions built into them. When in doubt, add more subjects.
Analysis: Two ideas off the top of my head:


*

*If you're not interested in any covariates - and indeed, randomizing should get rid of the differences between your study groups if you do it right - then for continuous, normally distributed measures, you should be able to do a t-test. For more categorical measures, like things rated on scales, you're looking at contingency table analysis.

*UI design seems like a perfect application of survival analysis. Unless one of the GUIs is profoundly bad your users should be able to complete your test on either system. So the question is how long does it take them to complete your test? With two groups and a random design, comparing time-to-completion with something like a Kaplan-Meyer curve gives you both a cool picture, a good estimate of how much faster one UI is over the other, and is pretty straight-forward.


And as @whuber said, it's really nice to see people pondering this ahead of time.
A: Which test you have to use depends on the variables you intend to measure. A Likert type questionnaire needs a different test than time measurements. So first you need to know what the criteria are for the experimental interface to be better. Then you have to find out how to measure these and what measure constitutes a better interface. Only then can you determine the test to use.
As for power: You could also use an adaptive design. This is a kind of design where you don't have a pre-set sample size but define parameters for dynamically increasing the sample size. E.g. you start with 10 participants and see if you get significant results. If your p is below alpha, you stop: everything is fine. If your p is above alpha but below a certain threshold you continue to add participants (depending on your randomization scheme - e.g. batches of 4) until your p leaves the corridor between the upper threshold and the lower one (alpha). You stop if your time runs out or your funding. ;-) But don't take my word on this here. This is what I heard. I've never used it myself.
The problem with user interface studies is that it is often hard to get significant results with small sample sizes. You could interpret this in two ways: a) my study is bad (to little data) or b) the difference between the interfaces is not large enough to show a consistent improvement. 
You could also think about which is more important to you: a low type-I error or a low type-II error. A low type-I error would be great if there is some risk or cost attached to using your experimental interface and you really want to be sure that you have an effect there in order to justify the cost. Cost could be everything from higher cognitive load to major trade unions going on strike because they hate the new interface. A low type-II error (i.e. high power) would be great if there is some cost of NOT using the experimental interface and you want to justify that. (In text books this is often explained under "consumer risk" and "producer risk".)
