5
$\begingroup$

I am writing a manuscript using an experimental design which predicts interactions between 1 continuous variable and multiple dichotomous variables, all predicting a continuous variable. As is traditional in experimental design, I have used ANCOVAs to analyze the data, but am concerned about the inability to specifically test the interaction between the covariate and the dichotomous variables.

Also, while I am expecting only a small amount of variance to be explained, there is only one significant finding among the multiple predictors and interactions. I suspect the non-significant findings may be due to the small amount of explained variance in the DV, which when spread across multiple predictors in the model is insufficient to distinguish between them.

A colleague has suggested that the best option to deal with both concerns is to do four 2 or 3 stage hierarchical regressions with dummy variables, i.e. each regression would comprise: stage 1: one dummy variable stage 2: add the continuous variable and the interaction between the two. After running these analyses, any significant predictors and interactions could be combined into a single hierarchical regression.

I am unable to find a precedent for this unusual procedure but interestingly, it reveals a number of significant results, albeit explaining a very small amount of variance.

Is this a reasonable procedure to follow, given the requirements of my variables?

Improve this question
$\endgroup$
2
  • $\begingroup$ Hi Camille, welcome to the site. Your question looks interesting, but you're probably more likely to get better answers if your provide some specific details of what you're looking at. See meta.stats.stackexchange.com/questions/1479/… $\endgroup$
    – naught101
    Commented Dec 10, 2012 at 10:03
  • $\begingroup$ @naught101. Thanks for the direction. I thought it would be easier to answer if I wrote in abstract ideas rather than specific variables but here goes. I'm a statistician so just wrote $\endgroup$
    – Camille
    Commented Dec 10, 2012 at 11:04

1 Answer 1

Reset to default
7
$\begingroup$

ANCOVA and multiple regression are mathematically identical. In matrix algebra terms, both are $Y + XB + e$. If you don't have much variance explained in ANCOVA, you won't in regression.

The main problem I can see with your colleague's approach is that of inflating type I error by running multiple tests.

Statistical significance is, generally, of less importance than many people think it is. From a scientific/substantive point of view, if the effect sizes are small they are usually not interesting (although there are exceptions.... If you could reduce the number of, say, plane crashes by a small amount, airlines would probably be interested).

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I have the same concern regarding my colleagues approach. I agree about statistical significance but unfortunately psychology is a field where the p value rules!! I should also say that I have no problem with reporting a lack of significant findings or with very small effect sizes as negligible results are informative in themselves regarding the lack of importance in the real world. $\endgroup$
    – Camille
    Commented Dec 10, 2012 at 11:28
  • 1
    $\begingroup$ Oh, I know about psychology! I make my own small efforts to change things, but .... yes, p value is still (all too often) viewed as the most important (or even only important) thing. $\endgroup$
    – Peter Flom
    Commented Dec 10, 2012 at 11:34
  • $\begingroup$ Clearly this is not ground-breaking stuff so small effect sizes are indicative that context doesn't mean much here. However I still have the question of how to test the interactions and whether the regressions are a better technique. $\endgroup$
    – Camille
    Commented Dec 10, 2012 at 12:10
  • $\begingroup$ As I said, ANCOVA and regression are mathematically equivalent. Interactions may be easier to add in the regression model, but that would be because of notation and software implementation. $\endgroup$
    – Peter Flom
    Commented Dec 10, 2012 at 13:06
  • 2
    $\begingroup$ +1. The issue is not ANCOVA v regression (they are the same) but how you build a model. The colleague's proposed approach sounds like a variant on stepwise regression, and will have all the associated problems as Peter Flom points out. See stats.stackexchange.com/questions/13686/… . $\endgroup$
    – Peter Ellis
    Commented Dec 11, 2012 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.