Which tests can I use to analyze dependent likert-type data?

Question

In a survey, where students were presented various stimuli, after each stimulus they were asked to rate the following:

• Quality, on a five-point scale from "Excellent" to "Bad", let's say $Q$
• Confidence in their quality ratings, on a four-point scale from "Very Confident" to "Not confident", i.e. $C$

Here are my assumptions:

• Obviously these samples are dependent, since they belong to exactly one stimulus and student pair (am I right assuming so?)

• Also, the questions are very similar to the Likert scale, therefore I'm unsure of whether to classify them as ordinal or interval-based (and others seem too). I'd say they're ordinal, but many researchers in my field seem to ignore that and treat them as interval-based.

Basically, what I'd like to find out is whether these ratings are dependent on each other. So, my questions are:

• Would that data be considered interval-based or ordinal?
• Which tests can I apply here? $\chi^2$? Wilcoxon signed rank test?

I've already come up with three dimensional plots that show the counts for each pair, e.g. to say "In ten cases, users chose Bad and were Not Confident about it". Or, $count(Q_1,C_4) = 10$ … But there's nothing I can statistically prove from that alone.

Answer 1

I think there are several challenges to consider.

In terms of how to visualize, the most accurate would be to use a mosaic plot, or a stacked barplot (which are practically the same in this case, but it might be easier to find a stacked barplot in excel or SPSS than the mosaic plot).

It might also be helpful to change the likert scale to a numerical (1-5) scale, and have a boxplot of each of the 4 categories of your second question. Since boxplots are based on percentiles, the meaning of the boxplot can be somewhat consistent (depending on how the quantiles are calculated when dealing with mid points) with the type of data you present.

In terms of how to analyse, there are different questions you can ask. The simplest will be "is there a correlation between the two?", that can easily be answered using the pearson correlation on the ranking of the numerical values of your scales. This correlation will actually give you the Spearman correlation measure (the correlation of the ranks). The ranking is important for cases where you will have ties (for example, the vector: 1,2,2,4 should actually become: 1,2.5,2.5,3).

The wilcoxon test is relevant if you want to answer the question if the ranks of one measure is different than the other measure. But from your question, it doesn't sound like an interesting question. You can also use the Chi-square test for a similar question, but it's power will probably be smaller.

Answer 2

In my opinion, this question has been done to death but surfaces again and again due to slightly different nomenclature. First off, it depends on what your scientific question is: are you interested in trend (averaged differential response comparing groups) or heterogeneity (piecewise comparisons of responses among all categories)? What are the outcomes of the study? What are your stimuli? Are the stimuli categorical or ordered somehow or continuous in nature? These are details that statisticians need to know.

Ordinal regression is the family of methods around coding ordinal responses by their numeric categories. The physical quantities you estimate are uselessly intepreted as "expected differences in ordinal response levels" (this is verbatim how you would report regression coefficients) which are often non-integral, however the statistics around those quantities do test for association in the response levels. So coding, "very poor" as 0 and "excellent" as 5 in the case of Likert data is sensible.

You are using the word dependent ambiguously here. Does your experimental design have repeated measures within individuals? Is one individual exposed to many stimuli and recorded for different responses at different times? This would be repeated measures. Otherwise, I think you are confused about dependence and each stimulus/outcome observation is pairwise independent from other observations.

Answer 3

A summary test statistic for multidimensional, non-metric (input and output) data is a tough pitch and likely of limited interpretable value.

You could run a proportional odds logistic regression model of one variable (input) on the other (target) - if you are willing to do the analysis as comparing the target's probability distribution over the input levels. It would show if the input variable has significantly different dependence over the input levels and target classes. It won't really matter for predictive probabilities which contrast scheme you use, however, for interpretation of the weights, you might like to use orthogonal polynomials. You will need to interpret with examples test cases and their predictive distributions on barplots. This is because along with the logit probability scale that the weights work on, there are cut-offs identified by the regression process - which makes it rather difficult to interpret logit scale quantities.

For example with R, your code would be

#input data
#quality <- scan()
#confidence <- scan()
Q <- length(unique(quality))
C <- length(unique(confidence))

require(MASS)
# tell R that the data is ordinal
quality <- factor(quality, levels = paste(1:Q), ordered = TRUE)
confidence <- factor(confidence, levels = paste(1:C), ordered = TRUE)

# train model, R will use orthogonal polynomials by default
polr.model <- polr(confidence ~ quality)

#plot probability predictions as pdf for each input level
lapply(
  unique(quality), 
  function(z) { 
    pdf(paste('Quality_predictive_probabilities-Confidence_',z,'.pdf',sep=''))
    probs <- predict(polr.model,newdata=list(quality=z), type='probs')
    barplot(probs,xlab=paste('Quality',z),ylab='Confidence')
    dev.off()
  }
)

which will save the probability predictions into your current working directory. You may want to be careful about calling your variables confidence and quality if the audience is statistically aware as these words mean something quite specific to the community.