# Which tests can I use to analyze dependent likert-type data?

In a survey, where students were presented various stimuli, after each stimulus they were asked to rate the following:

• Quality, on a five-point scale from "Excellent" to "Bad", let's say $Q$
• Confidence in their quality ratings, on a four-point scale from "Very Confident" to "Not confident", i.e. $C$

Here are my assumptions:

• Obviously these samples are dependent, since they belong to exactly one stimulus and student pair (am I right assuming so?)

• Also, the questions are very similar to the Likert scale, therefore I'm unsure of whether to classify them as ordinal or interval-based (and others seem too). I'd say they're ordinal, but many researchers in my field seem to ignore that and treat them as interval-based.

Basically, what I'd like to find out is whether these ratings are dependent on each other. So, my questions are:

• Would that data be considered interval-based or ordinal?
• Which tests can I apply here? $\chi^2$? Wilcoxon signed rank test?

I've already come up with three dimensional plots that show the counts for each pair, e.g. to say "In ten cases, users chose Bad and were Not Confident about it". Or, $count(Q_1,C_4) = 10$ … But there's nothing I can statistically prove from that alone.

-
I believe you will be receiving answers suggesting accurate tetrachoric and related alternatives for ordinal DV. Just want to comment that often researchers avoid the nominal scale as their association measures may be missunderstood and/or because they may yield different alternative measures. Going into interval scales allows standard association estimations at the price of the akward implications of an interval scale (consecutive categories are equally spaced). –  JDav Aug 7 '12 at 13:20

A summary test statistic for multidimensional, non-metric (input and output) data is a tough pitch and likely of limited interpretable value.

You could run a proportional odds logistic regression model of one variable (input) on the other (target) - if you are willing to do the analysis as comparing the target's probability distribution over the input levels. It would show if the input variable has significantly different dependence over the input levels and target classes. It won't really matter for predictive probabilities which contrast scheme you use, however, for interpretation of the weights, you might like to use orthogonal polynomials. You will need to interpret with examples test cases and their predictive distributions on barplots. This is because along with the logit probability scale that the weights work on, there are cut-offs identified by the regression process - which makes it rather difficult to interpret logit scale quantities.

For example with R, your code would be

#input data
#quality <- scan()
#confidence <- scan()
Q <- length(unique(quality))
C <- length(unique(confidence))

require(MASS)
# tell R that the data is ordinal
quality <- factor(quality, levels = paste(1:Q), ordered = TRUE)
confidence <- factor(confidence, levels = paste(1:C), ordered = TRUE)

# train model, R will use orthogonal polynomials by default
polr.model <- polr(confidence ~ quality)

#plot probability predictions as pdf for each input level
lapply(
unique(quality),
function(z) {
pdf(paste('Quality_predictive_probabilities-Confidence_',z,'.pdf',sep=''))
probs <- predict(polr.model,newdata=list(quality=z), type='probs')
barplot(probs,xlab=paste('Quality',z),ylab='Confidence')
dev.off()
}
)


which will save the probability predictions into your current working directory. You may want to be careful about calling your variables confidence and quality if the audience is statistically aware as these words mean something quite specific to the community.

-
I ran your script but swapped confidence ~ quality since I'm rather interested in the effect of experienced quality on user confidence (is that the correct notation?). Running your script I get newdata' had 1 rows but variable(s) found have 4938 rows four times, and four times the exact same plots output as PDF (note the missing .pdf extension in your script). The only thing differing in the plots is the xlab, the numbers and histogram (?) are the same. –  slhck Aug 25 '12 at 13:08
Oops, try newdata=list(quality=z).Did you also replace unique(confidence) with unique(quality)? –  Yoda Aug 25 '12 at 20:00
Yes, I updated the code and it produces graphics I can probably use. Thank you! However, I'm not sure how exactly interpret the numerical results (or the regression results specifically). The graphics are obvious, but what in polr.model is interesting for me? –  slhck Aug 25 '12 at 20:21
The signs and magnitude of the regression coefficients for the different quality levels indicates their tendency to move the distribution around the 'confidence' classes. You could look at the t-statistics for the quality variable, but it would need you to throw out the only variable if they are not significant. –  Yoda Aug 25 '12 at 21:15

I think there are several challenges to consider.

In terms of how to visualize, the most accurate would be to use a mosaic plot, or a stacked barplot (which are practically the same in this case, but it might be easier to find a stacked barplot in excel or SPSS than the mosaic plot).

It might also be helpful to change the likert scale to a numerical (1-5) scale, and have a boxplot of each of the 4 categories of your second question. Since boxplots are based on percentiles, the meaning of the boxplot can be somewhat consistent (depending on how the quantiles are calculated when dealing with mid points) with the type of data you present.

In terms of how to analyse, there are different questions you can ask. The simplest will be "is there a correlation between the two?", that can easily be answered using the pearson correlation on the ranking of the numerical values of your scales. This correlation will actually give you the Spearman correlation measure (the correlation of the ranks). The ranking is important for cases where you will have ties (for example, the vector: 1,2,2,4 should actually become: 1,2.5,2.5,3).

The wilcoxon test is relevant if you want to answer the question if the ranks of one measure is different than the other measure. But from your question, it doesn't sound like an interesting question. You can also use the Chi-square test for a similar question, but it's power will probably be smaller.

-