3
$\begingroup$

In the test in question, we have data [measurements of discomfort] for the same subjects with different virtual reality experiences. We took multiple measurements of discomfort over the course of the experience. We want to measure the significance between difference in discomfort between the two experiences.

In effect, we have a 3D table of data where the dimensions are the different experiences, the subjects that volunteered, and the measurements over time.

Measurements from the same subject at the same time but different experiences are paired and would be appropriate for a paired t-test, but measurements from the same subject at different times are highly correlated and so [if I understand the paired t-test properly] this would be inappropriate to analyze with.

Is there some other version of a t-test that returns a significance value but can deal with data like this? Is the best option just paired t-tests per time slice and "combining" significance values by some statistical method?

Improve this question
$\endgroup$
1
  • $\begingroup$ If you were to fit a random effect model, what are the sources of unmeasured variation and are they between pair or within pair effects. For instance, age is a between pair effect in studies of twins because they're always the same age, but sex is not. $\endgroup$
    – AdamO
    Mar 29, 2016 at 21:35

2 Answers 2

Reset to default
1
$\begingroup$

If I understand your setup, this experiment is as follows:

  • 2 independent variables (VR scenario experienced, time)
    • VR scenario, two levels (A and B)
    • Time (start, 5 minutes in, 10 minutes in, 15 minutes in, end, etc)
  • 1 dependent variable (measure of discomfort) with multiple measurements taken per subject over time (repeated measures).

The first important question is the nature of your measure of discomfort. I assume you'll use something like a Likert-scale (on a scale from 1 to 5, with how much discomfort do you feel, with 1 being mild/no discomfort and 5 being extreme discomfort), so the measure will be parametric and on an interval/ratio scale. This is the most common (and usually the most useful) method, so we'll pretend that's what you had in mind.

You'll also of course have two possible ways to setup the human participants: 1) have every participant experience both conditions (you'll probably want to use counter-balanced ordering of experimental conditions), or 2) participants only experience one VR condition (either Scene A or Scene B, not both). You can do either one, but generally you'll need less participants if you go with option 1, as there will be less variance due to between-person factors (your experience of "carrots vs celery" will be more similar than "your experience of carrots" vs "my experience of celery", after all). You can use option 2 if necessary and this won't really change the test, but is generally avoided unless you have a good reason (like learning effects are just too great, etc).

If I've described your experimental scenario accurately, the most common test used for this is a two-way repeated measures ANOVA. This will allow you determine first if there is any statistically significant difference in any of the conditions (taking care of the issues you'd have by running repeated t-tests), and then the post-hoc tests will allow you to identify just what conditions are different from each other. If you decided to test only one VR scenario, then you'd use a one-way repeated measures ANOVA instead.

You might also reasonably ask a question like, "we also wonder how excited the participants feel", in which case you'd have participants respond to both a measure of discomfort and a measure of excitement. In this case you'd be adding a dependent variable, excitement, and this would change the test you'd need. In such a case what you'd likely want is a two-way repeated measures MANOVA if you kept both VR conditions, or if you dropped to one you'd just want a (one-way) repeated measures MANOVA. The more questions you ask (dependent variables) the more power you lose, so make sure you actually care about all the measures and don't just add them in willy-nilly.

Someone might be tempted to include one more independent variable, but generally I strongly warn you to avoid that temptation unless you have a lot of experience with such a beast, as heaven forbid you end up with a 3-way interaction of variables and need to interpret what is going on in a sensible fashion. It can get really messy and end up muddying the waters rather than clarifying them.

You'll naturally want to make note of all the assumptions of your chosen test, and SPSS will help you test the assumptions as well. These sorts of tests are very common in areas like HCI and cognitive psychology, and are not at all exotic. There are surely other approaches that could be used, but these are the classic approaches which are popularly published in these fields.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Exactly right - it's situation #1 with the Likert scale. Thank you so much! It's a very well-thought-out answer with some other avenues to wander down if I need to. What's SPSS? A quick Google search just tells me it's a IBM software package. Is that right? $\endgroup$
    – Mark Miller
    Mar 30, 2016 at 0:30
  • $\begingroup$ Wait - there's one issue. The time variable, though it is technically independent, seems to cause some problems. The link you provided, statistics.laerd.com/spss-tutorials/…, has an assumptions list. It says "You should have independence of observations, which means that there is no relationship between the observations in each group or between the groups themselves." Because simulator sickness in VR is generally cumulative, measurements from the same subject at different times have a very noticable relationship. $\endgroup$
    – Mark Miller
    Mar 30, 2016 at 0:51
  • $\begingroup$ @MarkMiller In theory there is still independence, in that people's discomfort at t=10 could have gone down from t=5, such as if they acclimated to the environment. It's theoretically independent, compared to, for instance, a high water line measurement that never moves down (can only move up or stay the same) - which would mean measurements weren't independent. It just so happens that VR sickness builds. The measurements are independent, and the the building over time of VR sickness is what you would expect the data to show (and the post hoc tests to confirm). $\endgroup$
    – BrianH
    Mar 30, 2016 at 1:59
  • $\begingroup$ @MarkMiller And SPSS is just software to analyzing statistics - well, really popular and incredibly powerful software. It's basically a spreadsheet to collect data and then countless tools to analyze it - though it can't do much to help you figure out what test you should run or what the results mean. Alternatives are other stat packages, such as Minitab, libraries in R or Python, plugins for Excel, etc - or technically calculating by hand, for those with a lot of time on their hands :) $\endgroup$
    – BrianH
    Mar 30, 2016 at 2:03
0
$\begingroup$

You can use non-parametric tests.
You have the Wilcoxon signed-rank test when there is just two periods of comparison, or the Kruskal-Wallis test which is appropriate when you want to check if the equality of more than two distributions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.