0
$\begingroup$

For my thesis, I had to evaluate the usability of two desktop applications - A and B. 5 participants were asked to rate how much they agreed to a set of statements. A 5 point Likert scale was provided (Strongly disagree=1; Disagree=2; Neither agree nor disagree=3; Agree=4; Strongly Agree=5 ) to judge how much they agreed with each statement.

This questionnaire was first filled for application A and then for application B. Now, I have to analyse the Likert scale data of both the samples using some sort of parametric or non-parametric test. But I am a bit confused on how I should proceed and everything I have read online has not been of much help.

I understant I can conduct a t-test but I do not know how my data should look like? Do I have to feed in the data based on the mean results of each statement for each of the applications?

I just want to compare the results and show that 'X' application is ranked to be much more useable.

Improve this question
$\endgroup$
4
  • $\begingroup$ Welcome. So 5 participants were asked to complete a battery of questions in response to their experimentation with both applications (A and B)? Also, how many questionnaire items were respondents asked to complete? And finally, what software will you be using to analyze your findings? $\endgroup$
    – Thomas Bilach
    Commented Jun 25, 2020 at 15:05
  • $\begingroup$ How many people answered the survey? Literally 5? $\endgroup$
    – Huy Pham
    Commented Jun 25, 2020 at 15:13
  • $\begingroup$ @ThomasBilach yes, they answered the same survey first after experimenting with application A and then with application B. There were 15 items. I am using JASP. $\endgroup$
    – AntR
    Commented Jun 26, 2020 at 6:50
  • $\begingroup$ @HuyPham Yes, just 5. The user groups of the said applications are literally THAT small. $\endgroup$
    – AntR
    Commented Jun 26, 2020 at 6:51

2 Answers 2

Reset to default
-2
$\begingroup$

Given that you have used a likert scale, comparig means is a plausible way of seeing if the two apps are different, but because you only have 5 people in each group, you shouldn't use the t test because you don't have a large enough sample to assume that the mean will be normally distributed.

If the participants rating both apps then your best bet is a Wilcoxon signed-rank test which is like a paired t test but with looser assumptions.

If you had wanted to do a ranking, then you could have told the participants to rank the two, forcing them to choose which one they prefer. You can then simply compare that to a 50/50 even split which would have been what would have happened had they chosen randomly.

Some comments if I may: With 15 items, chances are you're going to get something that falsely appears significant. I genuinely think you've got to select only like 1 item that is the best comparison between the two apps and run the test on that. There are non parametric multivariate tests (i.e. they let you compare lots of means all at once, while having only a small sample), but it gets pretty complicated. You might not even need to hypothesis test honestly, why not just use some summary statistics.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks, I actually did end up doing a Wilcoxon test but I'll take your other comments into consideration. $\endgroup$
    – AntR
    Commented Jun 27, 2020 at 9:55
4
$\begingroup$

So McElreath (2020) talks about how Likert data is often the most mistreated type of data. While other methods may give you a reasonable answer, what you really should be using is an ordinal logistic regression. Here is a brief introduction that might be useful:

https://stats.idre.ucla.edu/r/dae/ordinal-logistic-regression/

but I'd recommend checking out an ebook copy of McElreath's book from your university library (they should have it). You can also watch this lecture on this type of data from his graduate seminar:

https://www.youtube.com/watch?v=zA3Jxv8LOrA&list=PLDcUM9US4XdNM4Edgs7weiyIguLSToZRI&index=14

McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. CRC press.

EDIT: Adding another citation here that will be relevant to those seeing this:

Bürkner, P. C., & Vuorre, M. (2019). Ordinal regression models in psychology: A tutorial. Advances in Methods and Practices in Psychological Science, 2(1), 77-101.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.