When to interpret multivariate tests when performing repeated-measures ANCOVA?

Question

Yesterday, I ran a repeated-measures ANCOVA. The purpose was to determine the usability of two computer systems, call them 1 and 2. Each subject completed three (conceptually different) tasks, call them A B and C, on each system. (The covariate was whether the subject had prior experience with the current system, system 1). The dependent variable was time taken to complete each task.

A GLM in SPSS prints out both the standard and multivariate results. The standard, within-subjects results indicated a main effect of system and an interaction of system*task. Now why would I need to use the multivariate results? They were significant as well, and I've heard multivariate tests have more power. But this seems to provide too much wiggle room for researchers-- if the within subjects test isn't significant, they can just look to see if the multivariate results are significant.

Also, since there was an interaction system*task, I ran simple main effects on each task. Turned out, A and B were significant in one direction, C was significant in the other. This seems to suggest to me that doing a MANOVA is a bad idea here. But how would I even know that if I was only looking at the multivariate results?

In short, I guess I just don't understand when to use MANOVA vs ANOVA if I have more than one DV.

Answer 1

This is not a complete answer as ANOVA with repeated-measures is a complex topic. You can look at your design from a multivariate point of view if you regard your data not as representing realisations of one DV in different conditions, but of (ultimately) different DVs which are to be analysed simultaneously. (Note that the multiple DVs will not correspond to the 2*3 cells themselves but will represent all linearly independent differences between the levels of a factor for the test of its main effect. For the "system" factor, there is just 1 difference variable, hence the multivariate test for this factor should be equivalent to the univariate approach.)

One consideration when choosing between univariate und multivariate tests are the test's assumptions. The univariate approach assumes (among others) sphericity of the error variance-covariance matrix for each of the three tests (factors A, B, interaction A*B). This may be doubtful in many cases - which is why the epsilon-correction schemes were developed. But these are not universally regarded as the best solution. (Note that the sphericity assumption here automatically holds for the "system" factor as it only has 2 levels.)

I suggest chapters 11-14 in Maxwell & Delaney (2004). Designing Experiments and Analyzing Data. NJ: Lawrence Erlbaum Associates. It is a long read but worth it - the chapters provide an in-depth explanation of the univariate and multivariate approach to one-way and multi-way repeated measures ANOVA.

Answer 2

Your question brings up a substantial problem with SPSS... too much automatic output. You are absolutely correct in your implication that the automatic generation of multiple tests is problematic. Some people take the fact that their computer program does this as evidence that it is good procedure. It isn't. Trust your statistical instincts on that point.