7
$\begingroup$

Using ANCOVA when groups differ on the covariate is controversial, although Tabachnick and Fidell write that this is a plausible function of ANCOVA in quasi-experimental (or observational) studies. As they state:

The second use of ANCOVA commonly occurs in nonexperimental situations when subjects cannot be randomly assigned to treatments. ANCOVA is used as a statistical matching procedure, although interpretation is fraught with difficulty [...]. ANCOVA is used primarily to adjust goup means to what they would be if all subjects scored identically on the CV(s). Differences between subjects on CVs are removed so that, presumably, the only differences that remain are related to the effects of the grouping IV(s). (Differences could also, of course, be due to attributes that have not been used as CVs.) This second application of ANCOVA is primarily for descriptive model building: the CV enhances prediction of the DV, but there is no implication of causality. If the research question to be answered involves causality, ANCOVA is no substitute for running an experiment.

Moreover, in this question the same issue was addressed, and the use of ANCOVA for intact groups was encouraged.

My question is: in these situations, in which the assumption of independence of the covariate from the treatment variable is violated, what are the assumptions? For example, must the covariate be correlated with the dependent variable inside the groups? Or are the assumptions simply the same as for ANOVA?

$\endgroup$
7
$\begingroup$

In an ANCOVA, you typically model

$$E(Y|T,X)=\gamma T+X \beta$$

where $Y$ is your outcome variable, $T$ is your treatment indicator ($T=0$ to indicate control, and $T=1$ to indicate treatment), and $X$ is a covariate (or a vector of covariates). Then $\gamma$ is the average treatment effect (ATE) conditional on $X$.

Now let $Y=TY^T+(1-T)Y^C$, where $Y^T$ is the outcome in treamtent group and $Y^C$ is the outcome in control group. The primary assumption, which is exploited by ANCOVA, is that the outcome variables $Y^T$ and $Y^C$ are independent from $T$ conditional on $X$. This is also called 'unconfoundedness' written as:

$$P(T|Y^T,Y^C,X)=P(T|X)$$

Otherwise outcome variables and treatment assignment are confounded and (conditional) mean differences on $Y^T$ and $Y^C$ may be caused by other factors than the manipulation (i.e., even given $X$). If $T$ and $Y^C$ and $Y^T$ are unconfounded conditional on $X$, the ATE estimate $\gamma$ from ANCOVA will be unbiased given that also all other model assumptions are met.

You may ask when it is clear whether there is unconfoundedness: this can never be assessed with absolute certainty and it represents the key weakness of adjustment for bias in observational studies. It is recommended (see ref. below) that you include all covariates that are even in tendency (p<.10) statistically associated (correlated) with either $T$, $Y^C$ or $Y^T$. This suggests that it is not problematic, rather desirable, that $X$ and $T$ are correlated when using ANCOVA (your first question).

In fact, the correlation of covariate(s) with dependent variable 'within the groups' (i.e., $X$ with $Y^C$ or $Y^T$) is an indication that the unconfoundedness assumption holds or is more plausible (your second question). But correlation with $T$ likewise indicates this. However: an 'ideal' $X$ covariate is associated to, both, treatment indicator and outcome variables. Since ANOVA does not include $X$ (your third question), it would assume unconfoundedness unconditional $X$, i.e., $$P(T|Y^T,Y^C)=P(T)$$which is a very strong assumption and dependence of $X$ and $T$ would point to its potential violation. It is therefore not recommended in your hypothetical situation and should be preserved to fully randomized experiments, in which any $X$ by definition is independent of treatment and criterion variables.

It is important to note that meeting all of the other model assumptions of ANCOVA is required to find unbiased ATE estimates (e.g., using least squares estimators). Chiefly, this suggests that there is no interaction between $T$ and $X$. This is sometimes referred to as effect homogeneity (as opposed to hetorogenous effects, if there is an interaction). Therefore, the model should at least include the interactions as well, which is not standard in ANCOVA models. Furthermore, you assume linearity (inspect residuals to check this assumption) and you also assume that the Y-model is correct (i.e., that you included all relevant $X$ to model $Y$).

Sometimes, propensity score methods and nonparametric matching methods are superior to ANCOVA because they do not feature the linearity assumption and can include interactions 'on the go'. Moreover, so-called double-robust methods combine Y-modeling with propensity score methods. They guarantee unbiased effect estimates even if the model for $Y$ is incorrect (assuming the propensity score model is correct). Still all of these methods make the unconfoundedness assumption.

For an excellent treatment of ANOCVA adjustment for selection bias (and also other methods) see:

Schafer, J. L., & Kang, J. (2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods, 13(4), 279–313. doi:10.1037/a0014268

$\endgroup$
  • 1
    $\begingroup$ Excellent answer, @tomka. Thanks. Regarding the interaction term, I insert it into the model, but when it is not significant (the slopes are homogeneous) I ususally repeat the analysis without the interaction term. Is this practice correct? $\endgroup$ – this.is.not.a.nick Dec 27 '13 at 15:16
  • 1
    $\begingroup$ @this.is.not.a.nick Thank you -- I think that it is under current debate whether statistical significance alone should guide model building. P-values are rather strict cut off points and sometimes leaving a non-significant covariate or interaction in the model can still have a bias adjusting effect, if in the population there is a relation between $Y$ and $X$ or $TX$. That is, small sample size (power) might have lead to non-significance. I would only use a model selection procedure (based on p-values or also AIC or BIC), if you would otherwise have an over-parameterized model. $\endgroup$ – tomka Dec 27 '13 at 15:50
  • $\begingroup$ In more plein words: a few non-significant IEs are not so problematic, I believe. I would exclude non-significant IEs based on a standardized procedure (like forward or backward model building) if otherwise you would have many non-significant parameters in the model. More specific advice depends on the particular situation, like sample size, number of covariates etc. $\endgroup$ – tomka Dec 27 '13 at 15:55
  • 1
    $\begingroup$ Probably leaving it in the model or omitting it will only incrementally affect the ATE estimate. The confidence interval bounds might be a bit more conservative (wider) when leaving the IE in. You could check this and communicate it to the reviewers of your work. $\endgroup$ – tomka Dec 27 '13 at 16:07
  • 2
    $\begingroup$ FYI, There are several good R packages such as $Matchit$ (see website gking.harvard.edu/matchit) that implements nonparametric propensity score matching. $\endgroup$ – forecaster Dec 27 '13 at 21:38
4
$\begingroup$

I think a good starting point with this issue is to think logically about the meaning of a covariate adjustment in such situations. If the expected value of the CV is conditional on the group how is there any way to remove variation associated only with the CV? Surely a CV adjustment removes group effects as well! What then do group differences actually mean? In fact, as far as I am aware, the only truely interpretable method of ANCOVA is one where th CV and treatments are wholly unrelated. In such situations "control" would seem the wrong metaphor as the ANCOVA is more of an error-reducing technique to increase power to detect group differences.

I think this issue always needs logical consideration in terms of interpretation. There is no way of knowing quite what the adjustment made to the outcome is when looking for further group differences. Indeed, does it even make sense to consider groups as if they were the same on the CV? If the CV and the groups are so closely aliased does that not suggest that the CV may represent some fundamental element of group?

More can be read about this in Miller & Chapman (2001) "Misunderstanding Analysis of Covariance". Although groups differing on the CV may not be a strict assumption of ANCOVA I'm of the opinion that there are limited legitimate ways of interpreting results if the condition is not met. ANCOVA is a tehnique for designed experiments with randomised treatment assignment. Use beyond this should always be treated with caution.

I should perhaps add that I don't think that you can never use ANCOVA with non-randomised groups, but if you do you just need to be cautious. Generally speaking the only conditions that would need satisfying would be independence of group and CV (which you can test by running the ANOVA with the CV as the outcome variable), homogeneity of the regression slopes (which you can test by including an interaction term), and linearity, which can be checked using residuals. If your aim is to "control" for a concomitant variable then all assumptions need satisfying for your group differences to be interpretable. If, however, an assumption such as the homogeneity of slopes is violated then the model can always be re-framed as a multiple regression inclusive of the interaction term. The focus of the analysis would then be more exploratory and predictive then the classical ANCOVA, but ultimately allows you to see the CV not as a nuisance but as an interesting relationship to be explored.

$\endgroup$
  • 1
    $\begingroup$ I agree with you, but in my case I think my covariate is a disturbing variable that is not a feature of the treatment group. Indeed, I have another variable that differs between the two groups, but in this case it is not logical to consider it as a covariate, since the variable is a predictable personality trait of the group. $\endgroup$ – this.is.not.a.nick Dec 27 '13 at 15:12
  • 2
    $\begingroup$ @this.is.not.a.nick The complication is that classical causal inference does not make a difference between confoundng variable and 'feature of the treatment group'. That is, treatment may have more than one effect (say on Y and on Z) or there may a causal chain (treatment affects Y, which affect Z). It is true that controlling for Z then does not make any sense. There is more recent work on causal graphs, which includes these causal mechanisms in the equations. A good introduction is Morgan & Winship (2007). Conterfactuals and Causal Inference. $\endgroup$ – tomka Dec 27 '13 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.