3
$\begingroup$

I have data that involves 2 groups (equal sample size in each) and data for each group over 3 time points (they are actually 3 different monetary reward conditions). I want to investigate within group differences. All time/condition points are important, I don't have a "control" time point.

Any opinions on the following:

  1. If I am comfortable that sphericity is assumed by Mauchly's Test not being violated, would a multivariate Lambda F test statistic be more appropriate or a sphericity assumed estimate (within-condition estimate from SPSS)?

  2. If I choose to go with the understanding that despite a test result saying it is OK, assuming sphericity may still be over-confident, any thoughts on potentially reporting ALL Greenhouse-Geiser estimates regardless of Mauchly's or Lambda? I have read that this may reduce the chance of a Type-1 error without having to assume sphericity or equality of covariance matrices. Perhaps too overcautious? Or, is potentially adjusting df's more invasive than assuming sphericity?

$\endgroup$
  • 1
    $\begingroup$ Are you doing one mixed within-between ANOVA or are you trying to compare between two separate RM-ANOVAs? $\endgroup$ – John Sep 30 '11 at 9:55
  • $\begingroup$ Actually I am doing two separate RM-ANOVAs. Sorry, forget what I said earlier about comparing between groups. I am looking at within group, so if time-condition (3 time points) has an effect for accuracy for one group at a time. Thanks for clearing that up! $\endgroup$ – Short Elizabeth Sep 30 '11 at 11:29
  • $\begingroup$ Please edit your question to reflect that. Also, is one of those time points the control or is there an additional one? $\endgroup$ – John Sep 30 '11 at 13:24
  • $\begingroup$ No control time point, I am looking at how three different reward conditions effect task performance. There is a "neutral" condition, however it is not really a control condition. $\endgroup$ – Short Elizabeth Oct 1 '11 at 2:38
  • $\begingroup$ OK, my question is still... this neutral condition, is it one of the three reward conditions or is it an extra one, for 4 all together? $\endgroup$ – John Oct 1 '11 at 7:17
1
$\begingroup$

so... here is a bit of a dog's breakfast of suggestions

There are more ways to approach this than the options you give yourself. One of them might be to take your three reward levels, one being neutral, and turn them into two reward effects. So, if C is neutral reward, and A and B are test levels makes up an A effect (A-C) and a B effect (B-C) and then compare them to each other. Because there are only two levels sphericity is not an issue. And then you're actually comparing your two effects. Do not make the mistake of testing A-C, and finding it significant and B-C, and finding it not, and then concluding there's a difference between A-C, and B-C. The difference between those two may not be significant in itself.

Mauchly's test, like all such tests, isn't terribly useful. It will fail all of the time with enought power, even if the sphericity violation isn't too bad and you can pass it all of the time with very low power. It definitely shouldn't be used like a hypothesis test in the Neyman-Pearson sense. No test of assumptions should be. Meeting your assumptions is a qualitative decision and Mauchly can help you with that but it's not used as a hard decision rule. Along those same lines, always using GG corrections can reduce the amount of Type I error that you make, as you inquired. However, it can also increase the amount of Type II.

And assuming sphericity isn't invasive at all... Describing it that way makes it sounds like you have a little bit too much reverence for your data and believe there's some church of decisions here. If you want to be conservative in your tests, GG everything and Bonferroni correct. But, if you do that, recognize that you're possibly making Type II errors and note that in your write up. If you don't want to do any of that then don't, but make sure that then you draw weaker conclusions about your tests, especially multiple ones and use them to point the way for future researchers to look.

If you want to go multivariate knock yourself out. It helps with the sphericity issue, if there is one. It doesn't fix multiple testing issues. But you should pick one beforehand and stick with it, not run all kinds of different analyses and see which makes your results look better. That's a whole different level of multiple comparisons. Posting your actual Mauchly test numbers and GG corrections on here might result in you getting some expert advice on how large a violation they are. It's unlikely they're big given that you only have 3-levels.

Speaking of 3-levels, there are no GG corrections for when you have two levels. There is no test of sphericity then. If you decide to make three comparisons, A-B, A-C, B-C, then none of these if GG corrected.

A final option you haven't mentioned is to calculate confidence intervals for each of your 3 comparisons and speak about them rationally. You could alpha adjust them if you wish, or even put two levels of bars on a graph. Then you just describe what are likely and unlikely to be the true values of the effect. So, if A-C does not cross 0 then 0 is an unlikely value. Only the values within the CI are likely.

As you can see, the reason you haven't gotten hard answers to your questions is that, despite your having formulated them well, there are only hard answers to small aspects of what you want to know. That can get firmer when someone has real numbers and hypotheses to deal with. To get more specific help in your description, multiple comparison issues, and some expert advice on how to treat your data, then make a new query with your analyses posted, numbers and what they mean, what your hypotheses are, and what you hope to find out or discuss. That will be more likely to land you some hard advice that's useful.

$\endgroup$
0
$\begingroup$

Assessing sphericity assumption

  • Like with most assumption tests, significance tests should not be taken too seriously. With small sample sizes the assumption test will be underpowered. Furthermore, if you are pragmatic, you are really more interested in the degree of violation of your assumption, rather than a binary decision rule.
  • If you are interested in the effect of time, then sphericity is almost always violated (correlations tend be larger between time points that are closer together in time). If you are interested in the effect of experimental manipulations, then sphericity may be a more plausible assumption. I guess it depends on the experimental manipulation. My intuition is that the greater differences between group means would be associated with smaller covariances, because of individual differences in change as well as measurement on each occasion. However, I don't have any specific studies on hand to justify this.
  • In addition to examining Mauchly's test, it can be interesting to actually look at the correlation or covariance structure of your repeated measures conditions in order to get a sense of the extent of any violation.
  • Furthermore, if you want get more sophisticated, you can adopt a modelling framework that specifically tries to model the error structure. However, it sounds like you are mainly interested in simply assessing group mean differences.

Univariate corrected or multivariate tests

  • If after reflecting on your assumptions, you conclude that sphericity is a reasonable assumption, then I'd use the sphericity assumed.
  • If you conclude that sphericity is not a reasonable assumption, then choose one of the univariate corrected or multivariate test approaches. I have never read a compelling case of why a researcher should prefer one over the other. Just don't choose your technique based on which has the better p-value.

If you want to read more, check out

  • Algina, J. and Keselman, H. (1997). Detecting repeated measures effects with univariate and multivariate statistics. Psychological Methods, 2(2):208. This article discusses relative statistical power of univariate corrected and multivariate approaches.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.