Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm trying to understand the statistical analysis I saw in a clinical study. They measured performance of 3 groups of subjects with a series of performance measures (A, B, C, ..., N). The objective was to find out if there was a difference between the performance of the 3 groups, and if yes, then in which ways they differed.

Their analysis consisted of doing a MANOVA like this:

set.seed(100)
group <- rep(c(0,1), each=40)
A <- rnorm(80, 5, .5) + .1 * group
B <- rnorm(80, 9, .3) + .2 * group + .5 * A
C <- rnorm(80, 12, .3) + .2 * group + .7 * B
d.1 <- data.frame(A = A, B = B, C = C, group = group)

fit.manova <- manova(cbind(A, B, C) ~ group, d.1)
summary(fit.manova, test="Pillai")

          Df  Pillai approx F num Df den Df    Pr(>F)    
group      1 0.19669   6.2027      3     76 0.0007949 ***
Residuals 78

When they showed they see significant differences, they continued to perform a series of ANOVA tests for each DV, i.e.:

summary(lm(A ~ group, d.1))
summary(lm(B ~ group, d.1))
summary(lm(C ~ group, d.1))

to find out which performance measures where different between the groups.

What I'd like to know is:

  1. Is this approach (MANOVA followed by series of ANOVA) justified? Are there strict assumptions before we could take this path?
  2. If yes, should there be some kind of correction for the second step, i.e. series of ANOVAs for individual DVs (multiple-comparisons)? What kid of correction?
  3. What is the recommended approach for problems like this with multiple DVs?

EDIT 1: changed the text to include an example code.

EDIT 2: updated the example. DVs are now correlated.

EDIT 3: this is actually a very common situation in studies involving objective measures. Devices typically just spit out an array of measures, even if you don't specifically ask for them. For example, you have two groups, a control and patient group, and do a laboratory gait analysis. The gait analysis systems gives you 50 different gait measures. Your research question might be: do the two groups have similar gait? If not, in which ways do they differ?

share|improve this question
    
Any linear model maximizes correlation between a linear combination of predictors and a linear combination of predictands. So, MANOVA with DVs X1 X2 X3 and IV Y is an exact mirror of multiple regression of Y on X1 X2 X3. The quantity you call Pillai's trace in the former is R-squared in the latter. –  ttnphns Nov 24 '11 at 17:15
    
So if I understand correctly, are you suggesting that an alternative way would be a multiple regression method to estimate group by a linear combination of DVs? Then I could follow it by a model selection scheme to see which subset of DVs could be used to build parsimonious linear model to predict group? But how about the original method? Do you think it is sound? –  AlefSin Nov 24 '11 at 17:27

2 Answers 2

1) Is this approach (MANOVA followed by series of ANOVA) justified? Are there strict assumptions before we could take this path?

In the example data you provide, there is no correlation between A, B, and C. Hence, a MANOVA seems beside the point. Unless you are interested in the relationship between A, B, and C, or have some reason to think that the three will be somehow correlated, just skip to the ANOVA.

2) If yes, should there be some kind of correction for the second step, i.e. series of ANOVAs for individual DVs (multiple-comparisons)? What kind of correction?

No. Likely not.

3) What is the recommended approach for problems like this with multiple DVs?

Well, if you know a relationship between them, or know that they will all be influenced by, say, subject, you have two possible options. If you know the relationship between them, try something like Structural Equation Modeling. If there is some reason to suspect that each metric will be influenced in the same way by subject, then you need to control for this.

Might I recommend you see the following paper, as it addresses most of your questions: H. J. Keselman, Carl J. Huberty, Lisa M. Lix, Stephen Olejnik, Robert A. Cribbie, Barbara Donahue, Rhonda K. Kowalchuk, Laureen L. Lowman, Martha D. Petoskey, Joanne C. Keselman and Joel R. Levin. 1998. Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses. REVIEW OF EDUCATIONAL RESEARCH. 68; 350-386 DOI: 10.3102/00346543068003350

share|improve this answer
    
You may also want to see MANOVA Method for Analyzing Repeated Measures Designs: An Extensive Primer by O'Brien and Kaiser 1985 Psychological Bulletin. –  jebyrnes Dec 1 '11 at 1:40
    
jebyrnes: thanks for your answer. Indeed my example is lacking as usually DVs are going to have some unknown correlations. I'm not sure about your second answer though. Why no correction for multiple comparisons is not necessary? To me a series of individual ANOVAs after a MANOVA sound so close to a kind of post-hoc test. –  AlefSin Dec 1 '11 at 8:55
    
I updated the example so that the measures are correlated. –  AlefSin Dec 1 '11 at 9:04
    
Then, yes, I'd try a structural equation model! See the lavaan package for R - lavaan.org –  jebyrnes Dec 1 '11 at 15:14
    
I don't think SEM can help as number of DVs can be large and we really don't have a sound hypothesis to model the structure of the dependencies. Also, in this limited setup we don't really care about that. The question is really focused on the inference, type I error rate and such. –  AlefSin Dec 1 '11 at 16:47

The answer is in how you have simulated your data. It defines the stochastic process that you are assuming, and is enlightening on how you ought to draw inference.

This

set.seed(100)
group <- rep(c(0,1), each=40)
A <- rnorm(80, 5, .5) + .1 * group
B <- rnorm(80, 9, .3) + .2 * group + .5 * A
C <- rnorm(80, 12, .3) + .2 * group + .7 * B
d.1 <- data.frame(A = A, B = B, C = C, group = group)

sets up a multivariate normal distribution.

If you know the dependency structure and it is a directed acyclic graph--like your example where A depends on group, B depends on A and group, C depends on B and group--just do a series of linear regressions and do inference on the coefficient of the group term.

If the dependency structure is more complicated, you should have a look at structural equations models.

Either way, the superior method is to write out your likelihood function (which is the product of multivariate normals) and figure out how to estimate the parameters of your link function (in this case, linear).

share|improve this answer
    
I don't think SEM or similar approaches would work as the correlation structure of DVs is really unknown and is not really of interest. For example in my gait example, there are tens of outcome measures so things can get really complex. –  AlefSin Dec 1 '11 at 16:43
    
So you have no theory driven model for how these gaits relate to each other? Do you have some sort of objective function that you can use to hang these performance measures together? –  jalospinoso Dec 1 '11 at 17:33
    
That's true, correlations between these gait measures are not really well investigated and could change for different populations. That's probably why these guys have used MANOVA to compress any differences between the two sets of measures of the two groups into a statistics. –  AlefSin Dec 1 '11 at 22:18
    
Yes, I think this is going to be a hindrance for you in deciding what performance looks like for each of these groups. In the absence of some theory driven model, using MANOVA followed by ANOVA is a reasonable approach, so long as the residuals are reasonably normal. –  jalospinoso Dec 2 '11 at 8:35
    
That's great for question 1. Do you have any opinion about the question 2? And finally what would YOU do if you had the same problem (i.e. Q3). –  AlefSin Dec 2 '11 at 8:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.