In a survey, where students were presented various stimuli, after each stimulus they were asked to rate the following:

  • Quality, on a five-point scale from "Excellent" to "Bad", let's say $Q$
  • Confidence in their quality ratings, on a four-point scale from "Very Confident" to "Not confident", i.e. $C$

Here are my assumptions:

  • Obviously these samples are dependent, since they belong to exactly one stimulus and student pair (am I right assuming so?)

  • Also, the questions are very similar to the Likert scale, therefore I'm unsure of whether to classify them as ordinal or interval-based (and others seem too). I'd say they're ordinal, but many researchers in my field seem to ignore that and treat them as interval-based.

Basically, what I'd like to find out is whether these ratings are dependent on each other. So, my questions are:

  • Would that data be considered interval-based or ordinal?
  • Which tests can I apply here? $\chi^2$? Wilcoxon signed rank test?

I've already come up with three dimensional plots that show the counts for each pair, e.g. to say "In ten cases, users chose Bad and were Not Confident about it". Or, $count(Q_1,C_4) = 10$ … But there's nothing I can statistically prove from that alone.

  • $\begingroup$ I believe you will be receiving answers suggesting accurate tetrachoric and related alternatives for ordinal DV. Just want to comment that often researchers avoid the nominal scale as their association measures may be missunderstood and/or because they may yield different alternative measures. Going into interval scales allows standard association estimations at the price of the akward implications of an interval scale (consecutive categories are equally spaced). $\endgroup$
    – JDav
    Aug 7, 2012 at 13:20

3 Answers 3


I think there are several challenges to consider.

In terms of how to visualize, the most accurate would be to use a mosaic plot, or a stacked barplot (which are practically the same in this case, but it might be easier to find a stacked barplot in excel or SPSS than the mosaic plot).

It might also be helpful to change the likert scale to a numerical (1-5) scale, and have a boxplot of each of the 4 categories of your second question. Since boxplots are based on percentiles, the meaning of the boxplot can be somewhat consistent (depending on how the quantiles are calculated when dealing with mid points) with the type of data you present.

In terms of how to analyse, there are different questions you can ask. The simplest will be "is there a correlation between the two?", that can easily be answered using the pearson correlation on the ranking of the numerical values of your scales. This correlation will actually give you the Spearman correlation measure (the correlation of the ranks). The ranking is important for cases where you will have ties (for example, the vector: 1,2,2,4 should actually become: 1,2.5,2.5,3).

The wilcoxon test is relevant if you want to answer the question if the ranks of one measure is different than the other measure. But from your question, it doesn't sound like an interesting question. You can also use the Chi-square test for a similar question, but it's power will probably be smaller.


In my opinion, this question has been done to death but surfaces again and again due to slightly different nomenclature. First off, it depends on what your scientific question is: are you interested in trend (averaged differential response comparing groups) or heterogeneity (piecewise comparisons of responses among all categories)? What are the outcomes of the study? What are your stimuli? Are the stimuli categorical or ordered somehow or continuous in nature? These are details that statisticians need to know.

Ordinal regression is the family of methods around coding ordinal responses by their numeric categories. The physical quantities you estimate are uselessly intepreted as "expected differences in ordinal response levels" (this is verbatim how you would report regression coefficients) which are often non-integral, however the statistics around those quantities do test for association in the response levels. So coding, "very poor" as 0 and "excellent" as 5 in the case of Likert data is sensible.

You are using the word dependent ambiguously here. Does your experimental design have repeated measures within individuals? Is one individual exposed to many stimuli and recorded for different responses at different times? This would be repeated measures. Otherwise, I think you are confused about dependence and each stimulus/outcome observation is pairwise independent from other observations.

  • $\begingroup$ To answer your initial questions: I just want to know how confident people feel depending on various factors, e.g. the quality. The stimuli are visually degraded video clips in this case, and they are impaired in different ways, i.e. with different "treatments" if you want. One individual is exposed to n stimuli, one after another, a total of n times. The observations are independent, then, but of course "quality" and "confidence" are sort of dependent (although I agree I might have used the wrong wording here). $\endgroup$
    – slhck
    Aug 21, 2012 at 18:17
  • $\begingroup$ If you are interested in the effect of your stimuli on the ordinal response levels as you've stated, then this is a standard "full factorial" design. For an example of a different approach, look at latin squares. Both are examples of repeated measures analyses and you're correct in assuming you need to take the correlated nature of the data into account. Repeated measures ANOVA would be the knee jerk reflex analysis for this type of data. No need for a Wilcoxon since there's no censored data. $\endgroup$
    – AdamO
    Aug 21, 2012 at 20:18
  • $\begingroup$ In our field, we can use latin square designs, but they're not that common for these types of experiments. As for the repeated measures ANOVA, could you maybe elaborate a little on what exactly to do in my specific case? Which would be the independent and dependent variable here? $\endgroup$
    – slhck
    Aug 25, 2012 at 12:33
  • $\begingroup$ Your indep and dep variable are determined by design: I don't know that, but you do. Repeated measures ANOVA lets you specify a cluster ID for values drawn repeatedly within the same individual. It lets your statistical technique know, "Hey, observations 14, 73, and 132 are all from the same individual, so consider them slightly more correlated". You specify independent and dependent variables much the same way as regular ANOVA. $\endgroup$
    – AdamO
    Aug 27, 2012 at 18:15

A summary test statistic for multidimensional, non-metric (input and output) data is a tough pitch and likely of limited interpretable value.

You could run a proportional odds logistic regression model of one variable (input) on the other (target) - if you are willing to do the analysis as comparing the target's probability distribution over the input levels. It would show if the input variable has significantly different dependence over the input levels and target classes. It won't really matter for predictive probabilities which contrast scheme you use, however, for interpretation of the weights, you might like to use orthogonal polynomials. You will need to interpret with examples test cases and their predictive distributions on barplots. This is because along with the logit probability scale that the weights work on, there are cut-offs identified by the regression process - which makes it rather difficult to interpret logit scale quantities.

For example with R, your code would be

#input data
#quality <- scan()
#confidence <- scan()
Q <- length(unique(quality))
C <- length(unique(confidence))

# tell R that the data is ordinal
quality <- factor(quality, levels = paste(1:Q), ordered = TRUE)
confidence <- factor(confidence, levels = paste(1:C), ordered = TRUE)

# train model, R will use orthogonal polynomials by default
polr.model <- polr(confidence ~ quality)

#plot probability predictions as pdf for each input level
  function(z) { 
    probs <- predict(polr.model,newdata=list(quality=z), type='probs')

which will save the probability predictions into your current working directory. You may want to be careful about calling your variables confidence and quality if the audience is statistically aware as these words mean something quite specific to the community.

  • $\begingroup$ I ran your script but swapped confidence ~ quality since I'm rather interested in the effect of experienced quality on user confidence (is that the correct notation?). Running your script I get newdata' had 1 rows but variable(s) found have 4938 rows four times, and four times the exact same plots output as PDF (note the missing .pdf extension in your script). The only thing differing in the plots is the xlab, the numbers and histogram (?) are the same. $\endgroup$
    – slhck
    Aug 25, 2012 at 13:08
  • $\begingroup$ Oops, try newdata=list(quality=z).Did you also replace unique(confidence) with unique(quality)? $\endgroup$
    – Yoda
    Aug 25, 2012 at 20:00
  • $\begingroup$ Yes, I updated the code and it produces graphics I can probably use. Thank you! However, I'm not sure how exactly interpret the numerical results (or the regression results specifically). The graphics are obvious, but what in polr.model is interesting for me? $\endgroup$
    – slhck
    Aug 25, 2012 at 20:21
  • $\begingroup$ The signs and magnitude of the regression coefficients for the different quality levels indicates their tendency to move the distribution around the 'confidence' classes. You could look at the t-statistics for the quality variable, but it would need you to throw out the only variable if they are not significant. $\endgroup$
    – Yoda
    Aug 25, 2012 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.