How to test for pairwise differences between factor levels while looking at an interaction with another factor?

Question

Let's say I have a survey with questions 1 through 10 about different characteristics of animals rated from 1 to 10 and this is repeated for 5 different animals.

Suppose I collect 50 surveys so my data looks something like:

   Person Question  Animal Rating
1      P1       Q1     Dog      7
2      P1       Q1  Rabbit      3
3      P1       Q1     Cat      6
4      P1       Q1   Horse      3
5      P1       Q1 Hamster      5
6      P1       Q2     Dog      8
7      P1       Q2  Rabbit      5
8      P1       Q2     Cat      6
9      P1       Q2   Horse      3
10     P1       Q2 Hamster      7

Where Q1 could be something like: How cute is this animal? Q2: How strong is this animal? And so on…

Is there a way to compare the different types of animals and different questions to see which questions are related to which kinds of animal? This would be in order to find the highs and lows of each kind of animal.

I thought of conducting an ANOVA but this does not allow me to test for pairwise comparisons between each animal within each question.

From my data, I can see that there is a significant interaction between the questions and animals but can’t decipher which animals within each question differ from one another – as the data is always compared against a base factor level (e.g. dog in this case).

The interaction between Dog and Q1 might show as significant when the base is Cat, but might not be significant when Rabbit is the reference level for example. And I don't think it would be valid to conduct many ANOVAs with different reference levels.

Is there a way to get a contrast matrix with the differences between each kind of animal for each question and their respective statistical significance?

Thanks in advance

Answer 1

I don't think you need to overthink this. First, do your anova and test that the interaction effect is significant. Then create the scaled interaction effects, like so:

$$ \sqrt{25} \frac{(\hat{y}_{ik} - \hat{y}_{il}) - (\hat{y}_{jk}-\hat{y}_{jl})}{4s} $$

where $s$ comes from the anova and $\hat{y}_{ik}$ is the average rating of animal $k$ to question $i$. These differences would have a standard $t$ distribution with 1288 degrees of freedom, $(25-1)(15-1)(5-1)$, so basically normal, if there were no interaction effects.

But you have just shown from the global test that there are interaction effects somewhere. So calculate the effects and see where you find major outliers from a normal distribution.

There is perhaps more art than science to this. Strictly speaking, you can't do standard t-tests on all these interactions -- they are not independent, you are doing multiple comparisons, and you already know that some of them must be "significant" since the global test indicated significance. It's more a question of ranking them and looking for the largest values and seeing if what you get makes sense.

If the significant interaction is due to one animal being weird or one question flagging a different quality than the others, this approach should indicate where the problem lies. But if every animal is sort of a one-off, this approach will be confusing. But in that situation, all you can conclude is that every animal is different with respect to the questions posed.

@BruceET's suggestions are also valid. With 15 questions and 5 animals, you can build up to 56 orthogonal contrasts, which you can test independently. These need to be chosen in advance of the analysis. My approach is more heuristic, and you don't get specific p-values out of it.

On the other hand, you might want to view this as a problem in measurement theory. Suppose your 15 questions can be grouped into a smaller number of traits -- say a cute/fierce axis and a trainable/not-trainable axis. It might make more sense to explore a factor structure for the questions and see if the same structure holds for all animals. Then you could compare the animals on the basis of how they load on the factors.