1
$\begingroup$

I am trying to evaluate the usability of three prototypes of an application and want to see if there is a significant difference in terms of usability. For this I have collected responses from users for three sets of same likert scale (5-point) based questionnaires after they tested each prototype.

Since the likert scale is ordinal, I have decided to apply Friedman Test to see if there is a statistically significant difference across all the prototypes and in case there is a significant difference, I will compare the prototypes pairwise by using a Wilcoxons Signed Rank test with a Bonferroni correction. What do you think about this approach?

Another area where I am confused is that I have 12 questions on the questionnaire set and these questions are divided amongst sub-group such as attractiveness, efficiency etc and I want to evaluate these sub-parameters across different prototypes. My question: what is the right approach of combining/aggregating values for questions belonging to a sub-group before applying the Friedman test?

Thank you for your time and for your response in advance!

$\endgroup$
0
$\begingroup$

let's solve your question in parts.

First of all, if you have three prototypes, then we can consider that you have three treatments. For each prototype, you collected the responses of individuals using a 5-point Likert scale. Therefore, you have a complete block design.

Since the likert scale is ordinal, I have decided to apply Friedman Test to see if there is a statistically significant difference across all the prototypes and in case there is a significant difference

It's true that your scale is ordinal. However, it does not imply that you need to use a statistical test based on ranks, such as Friedman's.

I will compare the prototypes pairwise by using a Wilcoxons Signed Rank test with a Bonferroni correction

You need to test if your data meet the requirements for applying a parametric test (see, e.g., https://en.wikipedia.org/wiki/Analysis_of_variance#Assumptions). If your data meet all the requirements, then you can perform an ANOVA and a subsequent post-hoc test (usually the Tukey's test). However, if your data does not meet all the requirements, you may use Friedman's test and a subsequent post-hoc test (I recommend you to use the Nemeyi's test).


Another area where I am confused is that I have 12 questions on the questionnaire set and these questions are divided amongst sub-group such as attractiveness, efficiency etc and I want to evaluate these sub-parameters across different prototypes. My question: what is the right approach of combining/aggregating values for questions belonging to a sub-group before applying the Friedman test?

My first thought is to analyze each response separately. However, you may group them for each of the evaluated parameters (as you said, attractiveness, efficiency, etc), and then perform a single test with all the answers regarding a single parameter grouped. Therefore, you will analyze the responses for each parameter separately, instead of analyzing each response separately.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.