In a hypothetical usability experiment, two interface changes are tested against a baseline. The interface changes are cumulative, i.e. variant B is an extension of A. We measure usability according to its definition by measures for efficiency, effectivity and satisfaction. To make things worse, the interface variants are measured by having human participants process three different sets of tasks.

If the hypothesis is formulated as "variant A improves the usability over baseline", the metric to test would be an index of the usability measures over all three sets of tasks. Since all three measures are subject to noise, a signal in one of the measures might be missed.

As an alternative, we could examine each usability measure separately. Is it correct to assume that in the latter case we have to correct for alpha-inflation? If so, would it be better to state only few hypotheses a priori (e.g. "B improves efficiency over baseline"), report the results and examine other hypotheses in an exploratory analysis?

Also, how about the three different sets of tasks? Is it correct to assume that examining the measures for each set of tasks separately would result in even higher alpha inflation?

tl;dr: In a 3x3 design (variant, task set) measuring three variables, do we have to adjust for a family-wise error rate of $1-(1-p)^{3\cdot3\cdot3}$? If so, can we get around it by stating fewer, more clearly-define hypotheses and use the remaining hypotheses as a basis for an exploratory analysis?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.