Following on from this question: Imagine that you want to test for differences in central tendency between two groups (e.g., males and females) on a 5-point Likert item (e.g., satisfaction with life: Dissatisfied to Satisfied). I think a t-test would be sufficiently accurate for most purposes, but that a bootstrap test of differences between group means would often provide more accurate estimate of confidence intervals. What statistical test would you use?
$\begingroup$
$\endgroup$
3
asked Jul 20, 2010 at 3:31
-
3$\begingroup$ A related question: People often use the nonparametric Mann-Whitney test for this kind of data. Since there are only five possible values, there will be lots of tied ranks. The Mann-Whitney test adjusts for tied ranks, but does this adjustment work when there are a huge number of ties? $\endgroup$– Harvey MotulskyCommented Aug 23, 2010 at 1:22
-
6$\begingroup$ You may be interested in this recent article published in PARE, Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon, j.mp/biLWrA. $\endgroup$– chlCommented Oct 6, 2010 at 17:05
-
1$\begingroup$ I am not sure if chi-square test is also appropriate, it tests whether there is any dependency between the groups and the items (different distribution between groups). $\endgroup$– pe-perryCommented Dec 15, 2014 at 4:02
Add a comment
|
6 Answers
$\begingroup$
$\endgroup$
3
Clason & Dormody discussed the issue of statistical testing for Likert items (Analyzing data measured by individual Likert-type items). I think that a bootstrapped test is ok when the two distributions look similar (bell shaped and equal variance). However, a test for categorical data (e.g. trend or Fisher test, or ordinal logistic regression) would be interesting too since it allows to check for response distribution across the item categories, see Agresti's book on Categorical Data Analysis (Chapter 7 on Logit models for multinomial responses).
Aside from this, you can imagine situations where the t-test or any other non-parametric tests would fail if the response distribution is strongly imbalanced between the two groups. For example, if all people from group A answer 1 or 5 (in equally proportion) whereas all people in group B answer 3, then you end up with identical within-group mean and the test is not meaningful at all, though in this case the homoscedasticity assumption is largely violated.
-
$\begingroup$ The Clason and Dormody article looks good. Your response distribution comments are interesting to contemplate. I agree that differences in distributions might be of interest. But if you were only interested in whether population group means were different, it would not necessarily matter what distributions gave rise to such equality. $\endgroup$ Commented Aug 19, 2010 at 10:33
-
$\begingroup$ In this case, you are assuming that your Likert scale (in other words, the perceived difference between, e.g. much satisfied and "just" satisfied) behaves ideally and is perceived as having the same meaning in both population. Thus you are implicitly making the assumption that this is a numeric scale, but I agree that this is often considered as such in applied research, especially if participants come from the same country. My point was just to emphasize the categorical data analysis perspective, as usually found in the Factor Analysis tradition, like in my reply to Question #10. $\endgroup$– chlCommented Aug 19, 2010 at 11:01
-
$\begingroup$ I assume that the mean of the sample responding to a Likert item is generally a meaningful summary of the group's position on the underlying dimension. It's interesting to think about when the meaning of a Likert item would vary systematically between groups. Of course, this issue extends beyond just Likert items, probably to any subjective measurement procedure. $\endgroup$ Commented Aug 20, 2010 at 2:03
$\begingroup$
$\endgroup$
Depending on the size of the dataset in question, a permutation test might be preferable to a bootstrap in that it may be able to provide an exact test of the hypothesis (and an exact CI).
answered Jul 20, 2010 at 8:05
$\begingroup$
$\endgroup$
1
IMHO you cannot use a t-test for Likert scales. The Likert scale is ordinal and "knows" only about relations of values of a variable: e.g. "totally dissatisfied" is worse than "somehow dissatisfied". A t-test on the other hand needs to calculate means and more and thus needs interval data. You can map Likert scale scores to interval data ("totally dissatisfied" is 1 and so on) but nobody guarantees that "totally dissatisfied" is the same distance to "somehow dissatisfied" as "somehow dissatisfied" is from "neither nor". By the way: what is the difference between "totally dissatisfied" and "somehow dissatisfied"? So in the end, you'd do a t-test on the coded values of your ordinal data but that just doesn't make any sense.
-
9$\begingroup$ ... and yet it is commonly done. One thing to point out, and yes this is a little pedantic, if you are using a single Likert-type item that isn't a Likert scale. The difference is meaningful (though the question asker is talking about a Likert item and the ordinality is an issue). A Likert scale is a consequence of summing or averaging several Likert items. This approach was developed specifically to offset the extent to which the ordinal data was actually ordinal and make it more reasonable to be treated as being on the interval scale. $\endgroup$ Commented Dec 6, 2010 at 9:31
$\begingroup$
$\endgroup$
1
If each single item in the questionnaire is ordinal, and I don't think that this point can be disputed given how there is no way of knowing whether the quantitative difference between "strongly agree" and "agree" is the same as that between "strongly disagree" and "disagree", then why would the summation of all these ordinal level scales produce a value that shares the properties of true interval level data?
For example, if we are interpreting the results from a depression inventory, it doesn't make sense (to me at least) to say that a person with a score of "20" is twice as depressed as a person with a score of "10". This is because each item in the questionnaire isn't measuring actual differences in levels of depression (assuming that depression is a stable, intenal, organic disorder) but rather the person's subjective rating of agreement with a particular statement. When asked, "how depressed would you say your mood is on a scale of 1-4, 1 being very depressed and 4 being not depresed at all", how do I know that one respondent's subjective rating of 1 is the same as another respondent's? Or how can I know if the difference between 4 and 3 is the same as that of 3 and 4 in terms of the person's current level of depression.If we can't know any of this, then it doesn't make any sense to treat the summation of all these ordinal items as interval level data. Even if the data do form a normal distribution, I don't think it is appropriate to treat the differences between scores as interval level data if they were computed by adding up all the responses to a likert-items. A normal distribution of data just means that the responses are probably representative of the greather population; it doesn't imply that the values obtained from the inventories share important properties of interval level data.
We need to be careful in the behavioural sciences about how we use statistics to speak to the latent variables we are studying, for since there is no direct way of measuring these hypothetical constructs, there are going to significant problems when we attempt to quantify subject them to parametric tests. Again, simply because we have assigned values to a set of responses doesn't mean that differences between these values are meaningful.
-
1$\begingroup$ If you are happy summing item scores, you have already assumed more than strictly ordinal level of measurement. Strictly speaking, ordinal measures cannot meaningfully be added or averaged (incidentally, Stevens is clear about that). Once you have done that, treating the resulting scores as interval level data is perfectly reasonable. $\endgroup$– GalaCommented Jun 25, 2013 at 15:44
$\begingroup$
$\endgroup$
1
Proportional odds ratio model is better then t-test for Likert item scale.
-
1$\begingroup$ Would you like to explain your reasons? I can see how such a model might provide a more precise model of observed responses. However, in the typical practical research situations that I have seen, researchers are interested in whether the two groups differ in terms of the mean (e.g., did the training group report greater performance than the control; was student satisfaction higher one year to the next). The proportional odds ratio model does not test this question exactly as far as I am aware. $\endgroup$ Commented Aug 20, 2010 at 1:53
$\begingroup$
$\endgroup$
I will try to explain proportional odds ratio model in this context since it was suggested and indicated in at least 2 answers to this question.
The score test of a proportional odds model is equivalent to the Wilcoxon rank sum test.
More precisely, the score test statistic for no effect of a single dichotomous covariate in a proportional odds cumulative logistic regression model (McCullagh 1980) for ordinal outcome was shown to be equal to the Wilcoxon rank sum test statistic. (Proof in An extension of the Wilcoxon Rank-Sum test for complex sample survey data.)
Just like Wilcoxon rank sum test, this test detect whether two samples were drawn from different distributions, regardless of the expected values.
This test is invalid if you only want to detect whether two samples were drawn from distributions with different expected values, just like Wilcoxon rank sum test.
