12
$\begingroup$

I've never liked how people typically analyze data from Likert scales as if error were continuous & Gaussian when there are reasonable expectations that these assumptions are violated at least at the extremes of the scales. What do you think of the following alternative:

If the response takes value $k$ on an $n$-point scale, expand that data to $n$ trials, $k$ of which have the value 1 and $n-k$ of which have the value 0. Thus, we're treating response on a Likert scale as if it is the overt aggregate of a covert series of binomial trials (in fact, from a cognitive science perspective, this is actually an appealing model for the mechanisms involved in such decision making scenarios). With the expanded data, you can now use a mixed effects model specifying respondent as a random effect (also question as a random effect if you have multiple questions) and using the binomial link function to specify the error distribution.

Can anyone see any assumption violations or other detrimental aspects of this approach?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Do you know of any published research that looks at the relative merits of using likert scales as intervals vs ordinal data? Perhaps, the flaws of treating them as interval level scales are not serious enough to warrant a complex approach. If that is the case then your approach may simply be a wild goose chase. $\endgroup$
    – user28
    Sep 6, 2010 at 1:12

6 Answers 6

Reset to default
17
$\begingroup$

I don't know of any articles related to your question in the psychometric literature. It seems to me that ordered logistic models allowing for random effect components can handle this situation pretty well.

I agree with @Srikant and think that a proportional odds model or an ordered probit model (depending on the link function you choose) might better reflect the intrinsic coding of Likert items, and their typical use as rating scales in opinion/attitude surveys or questionnaires.

Other alternatives are: (1) use of adjacent instead of proportional or cumulative categories (where there is a connection with log-linear models); (2) use of item-response models like the partial-credit model or the rating-scale model (as was mentioned in my response on Likert scales analysis). The latter case is comparable to a mixed-effects approach, with subjects treated as random effects, and is readily available in the SAS system (e.g., Fitting mixed-effects models for repeated ordinal outcomes with the NLMIXED procedure) or R (see vol. 20 of the Journal of Statistical Software). You might also be interested in the discussion provided by John Linacre about Optimizing Rating Scale Category Effectiveness.

The following papers may also be useful:

  1. Wu, C-H (2007). An Empirical Study on the Transformation of Likert-scale Data to Numerical Scores. Applied Mathematical Sciences, 1(58): 2851-2862.
  2. Rost, J and and Luo, G (1997). An Application of a Rasch-Based Unfolding Model to a Questionnaire on Adolescent Centrism. In Rost, J and Langeheine, R (Eds.), Applications of latent trait and latent class models in the social sciences, New York: Waxmann.
  3. Lubke, G and Muthen, B (2004). Factor-analyzing Likert-scale data under the assumption of multivariate normality complicates a meaningful comparison of observed groups or latent classes. Structural Equation Modeling, 11: 514-534.
  4. Nering, ML and Ostini, R (2010). Handbook of Polytomous Item Response Theory Models. Routledge Academic
  5. Bender R and Grouven U (1998). Using binary logistic regression models for ordinal data with non-proportional odds. Journal of Clinical Epidemiology, 51(10): 809-816. (Cannot find the pdf but this one is available, Ordinal logistic regression in medical research)
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Mixed effects ordinal logistic regression is also available in R with the ordinal package and clmm(). $\endgroup$
    – John
    Mar 31, 2012 at 14:39
9
$\begingroup$

If you really wish to abandon the assumption of interval level data for likert scales I would suggest that you assume the data to be a ordered logit or probit instead. Likert scales usually measure strength of response and hence higher values should indicate a stronger response on the underlying item of interest.

Suppose that you have a $H$ item scale and that $S$ represents the unobserved strength of response on the item of interest. Then you can assume the following response model:

$y = 1$ if $S \le \alpha_1$

$y = h\ $ if $\alpha_{h-1} < S\ \le \alpha_h$ for $h = 2, 3, ..H-1$

$y = H\ $ if $\alpha_{H-1} < S <\ \infty$

Assuming that $S$ follows a normal distribution with an unknown mean and variance would give a ordered probit model.

Improve this answer
$\endgroup$
3
$\begingroup$

One concern would be that by using this approach you're imposing a specific relationship between the mean $np$ and the variance $np(1-p)$ of the response. For the kind of surveys Likert scales are often used in - e.g. you choose one of five categories from between "Strongly agree" to "Strongly disagree" with respect to some statement or other - it feels wrong to me. For instance I'd expect a ten-point scale to give roughly the same distribution of responses as a five-point scale if you collapse adjacent pairs of categories: for a response $y$ & common $p$ $$\Pr_{n=4}(Y=y)\neq\Pr_{n=9}(Y=2y)+\Pr_{n=9}(Y=2y+1)$$ I recall some research that seems to bear this out: Coelho & Esteves (2006), “The choice between a five-point and a ten-point scale in the framework of customer satisfaction measurement”.

Improve this answer
$\endgroup$
1
$\begingroup$

You could use the binomial approximation in a 5-pt Likert scale if you combined the agree and strongly agree into one group & the disagree and strongly disagree into another. Of course, you still need to decide where the neutrals go. I would put the neutrals in any one group, use the normal approximation to the binomial (provided you have more than 40 responses), and develop confidence intervals on the proportions of each group (see any standard stat. text on how to get conf. intervals on proportions coming from a binomial distribution with the normal approximation). Then, I would put the neutrals in the other group, and redo the confidence intervals. If I get the same conclusion from both, then there is a potential conclusion. Otherwise, I don't see how the binomial can be used with Likert data.

Improve this answer
$\endgroup$
1
$\begingroup$

If I understood correctly, this paper suggests a very similar approach to what you have described, suggesting that yes, indeed, Likert-like data can arise from a binomial process.

Full Ref: Allik, J. (2014). A mixed-binomial model for Likert-type personality measures. Frontiers in Psychology, (5) 371

Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome to the site! Could you add a full reference for that paper? It's standard practice here because links tend to go dead. $\endgroup$
    – mkt
    Mar 8, 2018 at 16:36
-1
$\begingroup$

Actually I am preparing a paper in which I am using your approach of treating a response on a likert item as if it is the overt aggregate of a covert series of binomial trials.

In my paper the binomial distribution is used in order to explain the shape of the observed frequency distributions. The rationale behind this approach is given by two assumptions. In many applets, showing how the binomial distribution comes into existence, one has repeated independent Bernoulli trials by a single ball falling through an array of pins. Each time a ball falls onto a pin, it will bounce to the right (i.e. a success) with probability p or to the left (i.e. a failure) with probability 1-p. After the ball falls through the array, it lands in a bin labeled by the corresponding number of successes. In my paper the process of decision making is also seen as a series of repeated independent Bernoulli trials in which, at each trial, the subject decide to agree or not to agree to the statement in question. The two assumptions read as follows.

(i) At each independent Bernoulli trial the subject makes a decision to agree with probability p or not to agree (disagree) with probabiliity 1-p.

(ii) If five categories of response are available for the statement, then the number of times a Bernoulli decision is made regarding the decision to agree or not to agree (disagree) is equal to 4 (5-1).

The final choice for a specific response category is given by the following rules.

  • If in all (four) cases a Bernoulli decision of agreement is made, then the response 'strongly agree' will be given.

  • If in three cases a Bernoulli decision of agreement is made, then the response 'agree' will be given.

  • If in two cases a Bernoulli decision of agreement is made, then the response 'undecided' will be given.

  • If in only one case a Bernoulli decision of agreement is made, then the response 'disagree' will be given.

  • If in no case a Bernoulli decision of agreement is made, then the response 'stronglly disagree' will be given.

A similar reasoning can be given using 'disagree' decisions. In order to obtain a binomial distribution, the scoring of the response categories is as follows.

strongly disagree = 0, disagree = 1, neutral = 2, agree = 3, strongly agree = 4

These two assumptions lead to a binomial distribution for the response frequencies provided that there are no systematic differences between the respondents.

I hope you can agree. I would appriciate very much if you could improve my english in the above text.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I have removed your older reply. Please note that my comment was not intended as a negative remark; one-line replies are generally not very informative and arguable answers are to be preferred (but see our FAQ). $\endgroup$
    – chl
    Feb 2, 2012 at 13:25
  • 1
    $\begingroup$ This is an interesting & creative proposal, but I'm skeptical of it. In both your version & in ordinal logistic regression as suggested, eg, by others on this thread will require the proportional odds assumption. However, OLR will allow the thresholds / cutpoints b/t the categories to vary more flexibly, I believe, whereas they will be determined by the binomial parameters $p$ & $n$ in your scheme. This assumption would have to be verified against the data, & I suspect, would lead to problems. (BTW, the -1 did not come from me.) $\endgroup$
    – gung - Reinstate Monica
    Nov 2, 2012 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.