1
$\begingroup$

Given data:

30 datapoints in the form of usability scores (System Usability Scale) 2 groups within data (Group 1: 17, group 2: 13, independent, unequal sizes)

Objective:

See if there is any statistically significant difference between usability scores of these two groups

Question:

What is the appropriate approach here to investigate statistical significance comparing two independent groups?

Preliminary thoughts (based on some research as a non-statistician):

Some tests make assumptions about underlying data distribution, some don't. Visualizing data through histrograms and boxplots, my first impression is that it is not normally distributed. Therefore, e.g. Mann-Whitney-U test could be an option. Also, testing for normality using Shapiro-Wilk etc. is deemed to be useless due to little power to reject normality hypothesis, in particular on a small smaple size. Still, if such a test rejects the hypothesis (being: it is a normal distribution) one can indeed take this as an indication for non-normality.

Can anybody confirm or reject the conclusions I drew? Are there any better alternatives based on the information provided? I am looking for best practice in dealing with such data and would love to hear your opionions.

$\endgroup$

1 Answer 1

2
$\begingroup$

I would second your conclusion to use the Mann-Whitnet U test (MWUt).

The first reason for this is your use of Likert scale to collect your measurements. A Likert score is ordinal-scale data, not interval-scale data. It is technically not valid to perform arithmetic on Likert scale data because $2−1$ may not be equal to $3−2$, $3+2$ may not be equal to $4+1$, etc.. I am pretty sure you labelled your scale $1,2,3,4,5$, but you might as well have labelled them $1,3,6,7,9$, or for that matter $A,B,C,D,E$. You can see that performing ar4ithmetic will be challenging. Note that this topic (how to deal with Likert scores) has been highly debated (see e.g. this paper for a decent historical perspective), and there are many diverging opinions on this topic. The "general" agreement is that individual Likert scores should be treated as strictly ordinal, but that aggregations (i.e. sums, just as is the case for the SUS) of Likert scores can practically be treated as interval-scale data (but my personal opinion is that it makes things even worse; a sum of ordinal values even loses its ordinality $2+2$ could be more, or less than $3+1$, which could be different from $1+3$).

MWUt does not perform arithmetic (only comparisons, which are mathematically valid for ordinal scales. But your SUS score is already a sum!). It has good power, and by using the aggregate SUS score, it should not have too many ties (which make it lose power). As long as you interpret its significance properly (test of stochastic superiority, i.e. $P(Group1>Group2)>P(Group2>Group1)$: i.e. a random score from Group1 is significantly more often greater than a randomly drawn score from Group2), then it will tell you what you want (i.e. one of the groups gives higher SUS scores than the other).

As far as "normality", I honestly would not worry about it too much. It is not your 2 samples which need to be normal, it is the sampling distribution of the difference of means which needs to be. Now, you have no idea what that sampling distribution may look like (you only have 1 instance). So having both samples be normally distributed is a sufficient condition; but it is not a necessary condition. And while 13 and 17 are indeed "not very large" sample sizes, they are not ridiculously small either. So if you are tempted to use a t-test, unless your histograms look properly "ugly", I would not see anything objectionable. And you are correct that, at these sample sizes, normality tests lack power to reject; but if it does not reject, this says that your data is not too "ugly", as so the t-test will give you a reasonable estimate. But again, this relies on treating the SUS scores as interval-scale.

You could also simply bootstrap the difference of means; no need for normality. 13 and 17 might be too small sample sizes to convince your audience that the samples are truly representative of the populations. And again, this treats the SUS as interval-score.

In summary, your main problem is deciding (and justifying?) how to treat the SUS score; interval, ordinal? Theoretically speaking, you can not establish it is either. So you will need to address that question, in some manner.
If you decide to treat it as interval-scale (which I would not do, but you would not be the first, or only, or last person to do so), then honestly feel free to run a 2 sample t-test (Welch t-test, because your samples are unbalanced); you already hand-waved the issue of scale, so you can hand-wave the issue of normality (that last one being justifiable of several grounds).
If you decide to treat it as ordinal, then MWUt is your best option.
And if you decide it is neither, you will have to rethink completely how you measure usability... (which most likely will not happen, at least not for this particular study).
If from this asnwer, you deduce that I am not a big fan of SUS (and other such Likert-based scores), you would be 100% correct; I would use task performance metrics (time to complete, completion status, number of "errors" per task, etc.) which are objective measures, w/o ambiguity about how to treat them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.