4
$\begingroup$

Presence (or better sense of presence, i.e. the extent to which one feels that a mediated experience is not actually mediated) is a notoriously difficult variable to pin down and the question of how to measure it is hotly debated in its respective community. One particular debate concerns the the validity of standard self-report questionnaires used to measure a subject's sense of presence.

In a between-subjects experiment, people performed a search task in either a real or a virtual office. They then rated their sense of presence on a set of standard questionnaires. Participants in the virtual condition self-reported a similar sense of presence than participants in the real world condition.

While the authors only conclude that presence questionnaires should pass a "reality test" in order to be useful, I personally think that the results might have been different in a within-subjects experiment. Presence is such a subjective and abstract concept to the layman that I think it unlikely that subject A's feeling of presence under condition X can be meaningfully compared to subject B's feeling of presence under condition Y, but that condition X gives a subject a frame of reference on which to evaluate condition Y, if the subject is exposed to both (Note, IMO this would entail that presence could only be regarded a measure for relative differences between conditions rather than an absolute scale). This is somewhat substantiated by the fact that answers on presence questionnaires have a notoriously high variance.

I have already designed an experiment (which has four different conditions) as a within-subjects experiment using 48 subjects and balancing the order of conditions across subjects.

Then I got to wonder whether I could test my assumption with my data. So I am thinking about analysing my data as within-subjects data (i.e., using a repeated-measures ANOVA) but also taking the very first condition each subject is exposed to and simply treat that subset of the data as if it came from a between-subjects experiment.

However, I am wondering whether this is a valid statistical approach and what statistical test would be appropriate, rather than simply stating, for example, that in the within analysis condition X was different from condition Y while in the between it was not.

For example, would it be valid to treat mean for condition X using within data and mean for condition X using between data (for all four conditions) as factors for an ANOVA to see if the mean for a condition significantly differs for a within and a between subjects experiment?

NB: I could potentially add more subjects which only get exposed to one condition in order to equalise the number of data points available for the within and between subjects analysis but I would prefer to do the analysis on the original subjects only, as (a) the experiment is quite lengthy and cumbersome, and (b) I was actually hoping that keeping the same pool of subjects for both analyses would give me a stronger indication of whether my hypothesis holds, as it eliminates any variability due to some subjects appearing in one set of data but not the other.

$\endgroup$
0

2 Answers 2

1
$\begingroup$

The analysis you want to pursue is not uncommon (unfortunately I don't have a reference right now), but you have to split it. You have to run 2 ANOVAs and hope that the analyses agree.

  1. The overall ANOVA uses the full within subject design.
  2. The between-subjects ANOVA just uses the data from the first session and compares those.

If the seond analysis is not significant but shows the same pattern, this may be, as there is less power.

$\endgroup$
1
  • $\begingroup$ Thanks, at the moment I have two answers with no upvotes and I'm not versed enough in statistics to decide on the correct one. Can you a) find a reference, and b) tell me, were I to run a between-subjects study with equal or greater power and still found no result, while I do in the within-one, would my assumptions above be substantiated by that result? $\endgroup$
    – ThomasH
    Nov 26, 2012 at 0:28
1
$\begingroup$

It's not a very sound argument you have for your hypothesis... from what you said. Having a reference point of being in other conditions might make subjects less variable, and the effects observed across conditions within subjects should be much less variable. You should get more power in the within design. But I don't see any argument here at all for why they would switch direction of their effect. You don't have any foundation for that unless you're arguing that the between design previously run is just a spurious result. Check the power of the study and see if that's likely.

Let's say you have two condition A, and B and you do them repeated measures with 4 subjects like this.

trt1 trt2 order
A    B    1
A    B    1
B    A    2
B    A    2

Now, I'm not sure what you predict but it seems to me that you're saying B>A in the literature but you think A>B with repeated measures. That means that you should replicate the literature in column 1, and then also A>B in column 2. You'd need that in column 2 to get your repeated measures effect (assuming the literature replicates in column 1). But then you really are going to need something more in your explanation than you have now because you'll have replicated what, as near as I can tell, you're just calling a spurious finding because people don't understand presence until they've experienced multiple conditions.

$\endgroup$
2
  • $\begingroup$ You are right, B>A in between and A>B in within makes no sense. I have checked the paper again and I seem to have gotten it wrong. In fact, they found no significant difference between the groups using 20 subjects in total. If you were to run both a within- and a between-study, with the between having equal or greater power than the within, and the between result were A=B while in the within study you do find an effect A>B, can you then conclude that the effect is purely due to participants being exposed to multiple treatments? $\endgroup$
    – ThomasH
    Nov 26, 2012 at 0:25
  • $\begingroup$ It's practically impossible to have more power between S when you're measuring the same kind of thing either way. If you do then THAT's a meaningful effect. It means there's a negative correlation between the conditions. A positive correlation across subjects is expected, and how with within gets more power. But a negative would be surprising. Either way, you seem to be discussing the difference between significant and not significant, which is not itself significant (see Gelman). $\endgroup$
    – John
    Nov 26, 2012 at 2:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.