3
$\begingroup$

In my experiment, participants are shown 4 images (A, B, C, D) that they have to rate on a 5 point scale. So my data looks like this:

particpant1: A 5, B 3, C 3, D 1

p2: A 4, B 3, C 2, D 4

p3: A 4, B 3, C 2, D 4

etc...

I want to see if A and B are rated higher by participants, but I am not sure how to do this. I thought I could use a poisson regression, because I have count data. But I'm not sure and wouòd appreciate help.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Your design doesn't make things easy. Because all subjects rate all 4 images, you have to account for that correlation. Are you familliar at all with multinomial regression and random effects? $\endgroup$
    – Demetri Pananos
    Commented Apr 10, 2021 at 23:41
  • 1
    $\begingroup$ @DemetriPananos Multinomial or ordinal? I agree about the random effects. // I’m with Demetri that this is a deceptively challenging problem. // Unless those ratings are counts of objects/widgets/animals/whatever, you do not have a count variable just because it’s an integer. I suspect you have an ordinal variable that could go something like awful/bad/neutral/good/excellent and you happen to code those words as 1-5. $\endgroup$
    – Dave
    Commented Apr 11, 2021 at 0:09
  • $\begingroup$ How many experimental participants? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 11, 2021 at 0:57
  • $\begingroup$ This isn't really count data, it is more like ordinal data, so maybe change the title and retag? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 11, 2021 at 1:04
  • $\begingroup$ @Dave Yes, ordinal, my mistake. $\endgroup$
    – Demetri Pananos
    Commented Apr 11, 2021 at 1:10

2 Answers 2

Reset to default
7
$\begingroup$

Interesting thread! Demetri suggested one possibility for analyzing your data. But I wonder if something like what I outline below might get you closer to what you want?

It seems like you would like to see if A and B are rated higher than C and D. So why not compute the average rating given by each participant to the A and B images and the average rating given by that participant to the C and D images? (Computing average ratings from ordinal ratings is not ideal, but beggars can't be choosers.)

Then, you would end up with two response observations for each participant. For example, your first participant would have an average rating across images A and B of (5 + 3)/2 = 4 and an average rating across images C and D of (3 + 1)/2 = 2.

Then you could use a linear mixed effects model to model the average rating. The response variable in this model would be average rating across a 2-image set. The predictor variable would be image set, which is a dummy variable set to 1 for the set of images consisting of A and B and 0 for the set of images consisting of C and D. Your model would also include a random intercept for participant and, if necessary, a random slope for image set.

This model would allow you to investigate if the set of images consisting of A and B has tends to have higher average ratings than the set of images consisting of C and D.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Simplicity is a virtue. One consideration I feel OP should be made aware of is that this approach could be misleading if, for example, A and B are rated on the extremes and C,D, and E are moderately rated. For example, $A=1$ and $B=5$ result in a mean of 3, but so do $C=2, D=3, E=4$. Pathological? Perhaps. If OP takes this approach (and I suggest they do) they should check to see this sort of confounding is not present. $\endgroup$
    – Demetri Pananos
    Commented Apr 11, 2021 at 17:22
4
$\begingroup$

The unfortunate fact is that this can not likely be done with an easy statistical test like a chi-square or t test. That you've chosen an ordinal outcome (not a count, as you say) and have experimental subjects rank multiple images will require us to use an ordinal mixed effects model.

Thankfully, the ordinal package in R can do this. However, the model will not give you the exact answer you are looking for. It will answer if there is a change in the probability of choosing A or B over some reference group (C,D, or E). That is to say, it won't tell if A or B are chosen more over C, D, or E, it will tell you how the probability of choosing a 5 changes when you show someone img B over img A.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Maybe a paired comparison design had been better? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 11, 2021 at 1:53
  • $\begingroup$ @kjetilbhalvorsen Depends on what OP is trying to answer, I anticipate they are leaving out some key details. $\endgroup$
    – Demetri Pananos
    Commented Apr 11, 2021 at 1:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.