Dissimilarity rating experiment: how to analyze the results

Question

Study:

I simulated some surfaces materials at audio and haptic level, and I asked subjects to evaluate on a 9-point Likert scale the degree of coherence between the two stimuli. For example there are stimuli with metal at auditory level and snow at haptic level, or wood both at auditory and haptic level.

So it is like a dissimilarity rating experiment.

The experiment had only 12 participants, so for each stimulus I have 12 response (no repeated measures involved).

Question:

How should I analyse the following experiment?

Initial Thoughts:

So far the only way I am thinking to use is ANOVA.

Update

Hello again, I used the cmdscale command in R, but I did not get thecperceptual map I would love to see. I as you an help! Maybe the problem is that I misunderstood if my table is well built for the purpose of the analsys.

Let´s summarize my experiment, my goals and the problem:

I simulated some surfaces materials at audio and haptic level, and I asked subjects to evaluate on a 9-point Likert scale the degree of coherence between the two stimuli. For example there are trials where at auditory level a metal surface was simulated and at haptic level the snow surface, or wood both at auditory and haptic level.

The experiment had only 12 participants, so for each trial I have 12 response (no repeated measures involved). In total there were 36 trials, and each trial was evaluated only once by each of the 12 participants. So each subject provided 36 ratings on the 9 point Likert scale, one for each trial.

Basically, these are my data:

 WD   MT   SW   GR   SN   DL

WD 7.00 6.50 4.91 4.83 5.50 5.00

MT 7.33 6.91 2.08 3.16 4.25 3.25

SW 2.91 1.75 7.91 6.25 6.83 5.41

GR 2.91 2.66 6.25 6.41 7.25 6.75

SN 4.00 4.00 5.58 6.00 7.00 6.58

DL 3.91 3.08 5.16 6.25 6.50 6.83

On the rows the haptic stimuli and on the columns the auditory stimuli. In each cell there is the average score for each trial (e.g. the trial GR-MT is 2.66, that is the average score given by participants to the trial where the material "gravel" was provided at haptic level and the material "metal" was provided ar auditory level).

Now I want to analyze the data in the correct ways, and as said MDS is the best analysis instead of ANOVA as I was thinking.

My first goal is to print a perceptual map where to place the pairs of audio-haptic stimuli (e.g. WD-WD, MT-DL, etc.) and see how far are the trials from each other. I used cmdscale in R but I did not get the wanted result. Any suggestion?

My second goal would be to find some p-values like I normally get with ANOVA.

For example I would like to understand if having the coherent couple of stimuli SW-SW (which means "snow" both at audio and haptic level) produces significant differences n the evaluations rather than the couple SW-MT (which means "snow" at audio and "metal" at haptic level)

Again I would like to undestand if there is any statistical difference between all the couples of stimuli corresponding to solid surfaces (like the couples MT-MT, MT-WD, WD-WD, MT-MT) and all the couples where a solid surface and a aggregate surface are presented (like the couples MT-SN, or WD-GR, etc.).

...I want to get as many information as possible from that table. I really thanks anyone who can provide any suggestion or useful information.

Answer 1

It's hard to name an appropriate method without knowing the research question you're trying to answer. With that in mind, multidimensional scaling (MDS) takes measures of global dissimilarity between pairs of stimuli as input data: observers are asked to rate the similarity of two stimuli without being given explicit criteria for that judgement.

MDS tries to place the stimuli within a (low-dimensional) space such that the distance between the stimuli represent their dissimilarity. One is typically interested in the number of dimensions of the resulting space, and tries to infer some meaning about these dimensions. There are many variants of MDS. In R, check out the functions cmdscale() as well as isoMDS() and sammon() from package MASS.

Answer 2

ANOVA sounds reasonable to me — two-way ANOVA, to be specific, treating subjects as the blocking factor.

Answer 3

A few thoughts

1. you might want to rearrange your data frame either in wide (rows are cases and columns are ratings for stimuli pairs) or long format (rows are case by stimuli pair combinations). Then you could run mixed models lme4 or perhaps aov (see here for info on different ANOVAs).

2. You might find useful the tutorial on multidimensional scaling on the Quick-R website. It shows how to prepare the data and produce a plot. However, from what I understand of your data, you may have to transform your ratings. At the moment, higher scores seem to indicate greater similarity, whereas MDS will expect that higher scores mean greater dissimilarity. At a basic level you could use (9 minus rating).