Different ANOVAs by subgroup

Question

I am analysing data which I generated using a simulation model. Due to the stochasticity of the model, the output is also stochastic. I have several dependent variables (let's say X and Y) and vary six different variables (A, B, C, D, E, F) in a full factorial design. Each variable has between 2 and 5 levels.

Three of the variables (A, B, C) are used to simulate different "framework conditions". I am not directly interested in the effect of these variables on my dependent variable, but I'd like to know if there is an effect of the other three variables (D, E, F) on my dependent variable given a set of values for A, B and C. I am not directly interested in the effects of A, B or C.

So far, I tried to use factorial ANOVA, but integrating all the variables A, B, C, D, E and F highly violates the assumptions of (normally distributed residuals, homoscedasticity) as tested by a Shapiro-Wilk- and Levene's-Test. This can easily explained by the fact that the extent of variation and the distribution of the subgroups heavily depends on the demand profile (e.g. small standard deviation for the subgroups if factor C has a specific value).

If I group the dataset by the factors A, B, C and apply individual factorial ANOVAs for the remaining variables (D, E and F), assumptions are not violated.

My question is:

1. Can I run different ANOVAs for the mentioned subgroups and meaningfully interpret the results?

2. Do I have to account for multiple testing?

Answer 1

Should you account for multiple testing? I am sure opinions will differ, and it will depend on the goals for the analysis, and the number of separate anova's (you didn't tell.) It is probably better to go for one common model. For that, formulate the anova as a regression model, and choose some technique for handling heteroskedasticity in regression models. There are many possibilities, and you did not tell us much context:

• transformations

• bootstrapping, Alternatives to one-way ANOVA for heteroskedastic data, ANOVA for non-normal, heteroskedastic response

• joint modeling of mean and variance, a possibility is the R package gamlss Simulate linear regression with heteroscedasticity

• ? maybe robust standard errors? How to get ANOVA table with robust standard errors?

To get some more specific advice, maybe you could augment your Q with some plots and model output?

• There are certainly other possibilities. Ordinal regression is adviced by an answer to What to do with non-normality and heterogeneous variances in two-way ANOVA when transformations do not work?

• Bayes methods, code up the model with heteroskedasticity in stan or jags.