0
$\begingroup$

While testing some software I had users fill up a survey which asks questions in Likert type item(1-5) about the 3 different states of the software. These states were: one without aid, one with aid in a particular format and one with aid in another format. This was conducted on the same group of individuals in a "try this, then this, then this, now answer the survey" format.

The survey consisted of 20 questions each of which were independent variables. How do I go about analysing this data? While searching for answers I noticed some persons saying that I can just represent this data using bar charts to get the point across but I have doubts about this.

The software itself also collected information on the performance of users for a more objective approach. However this data is formatted into 3 groups. I originally thought about using a paired sample t-test to compare it with the following:

  • Without aid - Aid with 1 format
  • Without aid - Aid with the 2nd format
  • Aid with 1 format - Aid with the 2nd format

in order to determine if there was a significant difference. Would this t test be valid for comparisons like this in this scenario? Is there another option?

This is an example of the data where the values are scores for each format:

Without aid Aid(format 1) Aid(format 2)
5 6 7

Finally would you advise attempting to apply analysis to both the qualitative (likert) and quantiative data in order to link the two? Like for example Spearmans correlation. If yes what in particular would you suggest?

P.S: I'm a bit new to statistical analysis so I'm still learning about various tests.

The goal is to test a hypothesis that the inclusion of these formats do help the user.

Improve this question
$\endgroup$
4
  • $\begingroup$ I have two comments: you are using the term Likert scale incorrectly. You're confusing the item and the scale. This is a common mistake, check Wikipedia. Second, you didn't write what your goal is. A graphic presentation? Testing statistical hypotheses? $\endgroup$
    – Daniel Dostal
    Commented Aug 10, 2022 at 8:13
  • $\begingroup$ @DanielDostal Yeah its actually a likert type item and not the interval scale. The goal is to test a hypothesis that including the formats does make a difference $\endgroup$
    – Wiron Agroliv
    Commented Aug 10, 2022 at 8:21
  • $\begingroup$ Do you want to evaluate the entire test as a whole, or examine each item separately? What I mean by that is whether the items measure the same thing, and can therefore sum to one aggregate score, or whether you are interested in the information from each item separately. $\endgroup$
    – Daniel Dostal
    Commented Aug 10, 2022 at 8:30
  • $\begingroup$ I'm interested in individual items. I think I can better articulate those and point out key information. Some categories for the survey would be things like "Helpfulness, Cumbersomeness" etc. From there I'd try to point out any difference between the formats $\endgroup$
    – Wiron Agroliv
    Commented Aug 10, 2022 at 8:34

1 Answer 1

Reset to default
0
$\begingroup$

In my opinion, you have two choices. Either you assert that item scores are quantitative (metric) variables (which is not quite true) and use, say, a paired t-test.

The second option is that you treat the item scores as ordinal categories and model them using ordinal regression. A proportional odds model with mixed effects works very nicely here.

You need to reformat the data so that each respondent has three rows. The columns are person (levels 1 to n), aid (levels 0,1,2), and 20 columns item1, item2... item20. Then I recommend using library ordinal in R environment. You will model each item separately (so you will create 20 models).

You can use this code:

#let's create toy dataset as I don't have your original data
n = 50 #number of respondents
set.seed(2)
data = data.frame(person = rep(1:n, each = 3),aid = factor(0:2))
data$item1 = factor(rbinom(n*3, 4, as.numeric(data$aid)/4)+1)
data$item2 = factor(rbinom(n*3, 4, as.numeric(data$aid)/4)+1)
#...items 3 to 19
data$item20 = factor(rbinom(n*3, 4, as.numeric(data$aid)/4)+1)

#this is how your data look like
data

#use this library
library(ordinal)

#model the first item
fit = clmm(item1 ~ aid + (1|person), data=data, link = "logit")

#results
summary(fit)

#clearer results with odds ratios
cbind(oddsRatio = exp(coef(fit)), coef(summary(fit)))[-(1:4),]

#repeat for item 2, 3... 20
Improve this answer
$\endgroup$
4
  • $\begingroup$ I'm leaning a bit more on the paired t-test. I dont understand your comment "Either you assert that item scores are quantitative (metric) variables (which is not quite true)...". They are quantitative as they essentially keep track of user actions but it does not have a fixed interval $\endgroup$
    – Wiron Agroliv
    Commented Aug 10, 2022 at 9:11
  • $\begingroup$ Item scores are ordinal. This means that level 1 is less than level 2, level 2 is less than 3, etc. But it is not necessarily true that the distance between levels 1 and 2 is the same as between 2 and 3. Thus, mean and standard deviation are likely to be meaningless numbers, so the t-test is not a reliable tool here. $\endgroup$
    – Daniel Dostal
    Commented Aug 10, 2022 at 9:24
  • $\begingroup$ However, if you're going to use a t-test, you won't be the first or last to work with item scores as metric variables. In fact, it's a pretty common (bad) habit. $\endgroup$
    – Daniel Dostal
    Commented Aug 10, 2022 at 9:26
  • $\begingroup$ So from this should I conclude that using t -test would invalidate the hypothesis? I originally thought, based on readings, that if the mean average of scores collected for the unformatted version was higher than any of the other formats it would fail to reject the null hypothesis. With this info it seems all of this is also wrong $\endgroup$
    – Wiron Agroliv
    Commented Aug 10, 2022 at 9:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.