# Dissimilarity rating experiment: how to analyze the results

### Study:

I simulated some surfaces materials at audio and haptic level, and I asked subjects to evaluate on a 9-point Likert scale the degree of coherence between the two stimuli. For example there are stimuli with metal at auditory level and snow at haptic level, or wood both at auditory and haptic level.

So it is like a dissimilarity rating experiment.

The experiment had only 12 participants, so for each stimulus I have 12 response (no repeated measures involved).

### Question:

How should I analyse the following experiment?

### Initial Thoughts:

So far the only way I am thinking to use is ANOVA.

### Update

Hello again, I used the cmdscale command in R, but I did not get thecperceptual map I would love to see. I as you an help! Maybe the problem is that I misunderstood if my table is well built for the purpose of the analsys.

Let´s summarize my experiment, my goals and the problem:

I simulated some surfaces materials at audio and haptic level, and I asked subjects to evaluate on a 9-point Likert scale the degree of coherence between the two stimuli. For example there are trials where at auditory level a metal surface was simulated and at haptic level the snow surface, or wood both at auditory and haptic level.

The experiment had only 12 participants, so for each trial I have 12 response (no repeated measures involved). In total there were 36 trials, and each trial was evaluated only once by each of the 12 participants. So each subject provided 36 ratings on the 9 point Likert scale, one for each trial.

Basically, these are my data:

 WD   MT   SW   GR   SN   DL


WD 7.00 6.50 4.91 4.83 5.50 5.00

MT 7.33 6.91 2.08 3.16 4.25 3.25

SW 2.91 1.75 7.91 6.25 6.83 5.41

GR 2.91 2.66 6.25 6.41 7.25 6.75

SN 4.00 4.00 5.58 6.00 7.00 6.58

DL 3.91 3.08 5.16 6.25 6.50 6.83

On the rows the haptic stimuli and on the columns the auditory stimuli. In each cell there is the average score for each trial (e.g. the trial GR-MT is 2.66, that is the average score given by participants to the trial where the material "gravel" was provided at haptic level and the material "metal" was provided ar auditory level).

Now I want to analyze the data in the correct ways, and as said MDS is the best analysis instead of ANOVA as I was thinking.

My first goal is to print a perceptual map where to place the pairs of audio-haptic stimuli (e.g. WD-WD, MT-DL, etc.) and see how far are the trials from each other. I used cmdscale in R but I did not get the wanted result. Any suggestion?

My second goal would be to find some p-values like I normally get with ANOVA.

For example I would like to understand if having the coherent couple of stimuli SW-SW (which means "snow" both at audio and haptic level) produces significant differences n the evaluations rather than the couple SW-MT (which means "snow" at audio and "metal" at haptic level)

Again I would like to undestand if there is any statistical difference between all the couples of stimuli corresponding to solid surfaces (like the couples MT-MT, MT-WD, WD-WD, MT-MT) and all the couples where a solid surface and a aggregate surface are presented (like the couples MT-SN, or WD-GR, etc.).

...I want to get as many information as possible from that table. I really thanks anyone who can provide any suggestion or useful information.

• What is your research question? Also, what are you calling a stimulus - one type of audio, or the pair? Feb 13, 2011 at 11:45
• ok, thanks a lot, I will have a look to MDS and try to use the R functions.
– L_T
Feb 13, 2011 at 16:15
– user88
Feb 13, 2011 at 16:17

It's hard to name an appropriate method without knowing the research question you're trying to answer. With that in mind, multidimensional scaling (MDS) takes measures of global dissimilarity between pairs of stimuli as input data: observers are asked to rate the similarity of two stimuli without being given explicit criteria for that judgement.

MDS tries to place the stimuli within a (low-dimensional) space such that the distance between the stimuli represent their dissimilarity. One is typically interested in the number of dimensions of the resulting space, and tries to infer some meaning about these dimensions. There are many variants of MDS. In R, check out the functions cmdscale() as well as isoMDS() and sammon() from package MASS.

• +1 MDS is the main kind of analysis that I've seen for this type of study. Feb 13, 2011 at 11:48
• Hi Jeromy, I am convinced that likely the MDS is not the right analysis, since my table does not contain the distances. Indeed if you have a look to the diagonal of the table you can see that the values are not 0 (which should express zero distance between the same stimuli). The values I have in my table are not distances but average scores from a 9 points Likert scale, assessing the degree of coherence between two modalities (audio and haptic) simulating different pairs of materials. So my question now is: which analysis should I perform in your opinion? Thanks in advance
– L_T
Feb 15, 2011 at 12:40

ANOVA sounds reasonable to me — two-way ANOVA, to be specific, treating subjects as the blocking factor.

• ok, thanks a lot. Is there anyone who can suggest a different analysis?
– L_T
Feb 12, 2011 at 18:50
• Could you please state your research question(s) and how many ratings each subject provided. Feb 12, 2011 at 23:48
• Hi, thanks a lot for your reply. Actually there were 36 stimuli, and each stimulus was evaluated only once by each of the 12 participants. So each subject provided 36 ratings on the 9 point Likert scale, one for each stimulus. Any suggestion?
– L_T
Feb 13, 2011 at 1:59

A few thoughts

1. you might want to rearrange your data frame either in wide (rows are cases and columns are ratings for stimuli pairs) or long format (rows are case by stimuli pair combinations). Then you could run mixed models lme4 or perhaps aov (see here for info on different ANOVAs).

2. You might find useful the tutorial on multidimensional scaling on the Quick-R website. It shows how to prepare the data and produce a plot. However, from what I understand of your data, you may have to transform your ratings. At the moment, higher scores seem to indicate greater similarity, whereas MDS will expect that higher scores mean greater dissimilarity. At a basic level you could use (9 minus rating).

• Dear Jeromy, thanks a lot for your feedback. Anova seems correct to me. Anyways I am just having a look to a "conjoint measurement analysis" and apparently it seems to fit my problem. All the best
– L_T
Feb 17, 2011 at 1:18