# Should I use ANCOVA or multiple regression with dummy variables?

I am writing a manuscript using an experimental design which predicts interactions between 1 continuous variable and multiple dichotomous variables, all predicting a continuous variable. As is traditional in experimental design, I have used ANCOVAs to analyze the data, but am concerned about the inability to specifically test the interaction between the covariate and the dichotomous variables.

Also, while I am expecting only a small amount of variance to be explained, there is only one significant finding among the multiple predictors and interactions. I suspect the non-significant findings may be due to the small amount of explained variance in the DV, which when spread across multiple predictors in the model is insufficient to distinguish between them.

A colleague has suggested that the best option to deal with both concerns is to do four 2 or 3 stage hierarchical regressions with dummy variables, i.e. each regression would comprise: stage 1: one dummy variable stage 2: add the continuous variable and the interaction between the two. After running these analyses, any significant predictors and interactions could be combined into a single hierarchical regression.

I am unable to find a precedent for this unusual procedure but interestingly, it reveals a number of significant results, albeit explaining a very small amount of variance.

Is this a reasonable procedure to follow, given the requirements of my variables?

-
Hi Camille, welcome to the site. Your question looks interesting, but you're probably more likely to get better answers if your provide some specific details of what you're looking at. See meta.stats.stackexchange.com/questions/1479/… –  naught101 Dec 10 '12 at 10:03
@naught101. Thanks for the direction. I thought it would be easier to answer if I wrote in abstract ideas rather than specific variables but here goes. I'm a statistician so just wrote –  Camille Dec 10 '12 at 11:04

ANCOVA and multiple regression are mathematically identical. In matrix algebra terms, both are $Y + XB + e$. If you don't have much variance explained in ANCOVA, you won't in regression.

The main problem I can see with your colleague's approach is that of inflating type I error by running multiple tests.

Statistical significance is, generally, of less importance than many people think it is. From a scientific/substantive point of view, if the effect sizes are small they are usually not interesting (although there are exceptions.... If you could reduce the number of, say, plane crashes by a small amount, airlines would probably be interested).

-
I have the same concern regarding my colleagues approach. I agree about statistical significance but unfortunately psychology is a field where the p value rules!! I should also say that I have no problem with reporting a lack of significant findings or with very small effect sizes as negligible results are informative in themselves regarding the lack of importance in the real world. –  Camille Dec 10 '12 at 11:28
Oh, I know about psychology! I make my own small efforts to change things, but .... yes, p value is still (all too often) viewed as the most important (or even only important) thing. –  Peter Flom Dec 10 '12 at 11:34
Clearly this is not ground-breaking stuff so small effect sizes are indicative that context doesn't mean much here. However I still have the question of how to test the interactions and whether the regressions are a better technique. –  Camille Dec 10 '12 at 12:10
As I said, ANCOVA and regression are mathematically equivalent. Interactions may be easier to add in the regression model, but that would be because of notation and software implementation. –  Peter Flom Dec 10 '12 at 13:06
+1. The issue is not ANCOVA v regression (they are the same) but how you build a model. The colleague's proposed approach sounds like a variant on stepwise regression, and will have all the associated problems as Peter Flom points out. See stats.stackexchange.com/questions/13686/… . –  Peter Ellis Dec 11 '12 at 0:14
show 1 more comment