Impact of Covariates on ANOVA Results

Question

I am conducting an analysis using ANOVA with covariates and encountering significant results. However, when I remove the covariates, the results are no longer significant. Here are the details of my analysis:

model_with_covariates <- lm(size_continuous ~ 
    activation_categorical + age + handedness + sex, 
    data = current_tract_and_measure)
summary_anova <- anova(model_with_covariates)
        
model_without_covariates <- lm(size_continuous ~ 
    activation_categorical, data = current_tract_and_measure)
summary_anova <- anova(model_without_covariates)

I am performing 171 ANOVAs for different shape measures and different white matter tracts. Therefore, I am correcting all my p-values for multiple comparisons using FDR (False Discovery Rate). Here are the results after FDR correction:

With Covariates: FDR-corrected p-value = 0.001 for two different ANOVAs. Without Covariates: FDR-corrected p-value = 0.1110490 for the same ANOVAs that were significant before. My covariates (age, handedness, and sex) are associated with both the laterality and size of the regions. I want to account for these associations in my analysis.

Questions:

1. Should I always include covariates in my analysis if they are known to be associated with the outcome variable?
2. How should I interpret the differences in results with and without covariates?

Answer 1

Q1. James Davis' The Logic of Causal Order (Sage pubs.) has helpful guidelines about when to control for variables -- if trying to explain cause and effect, as I think you are. I'll paraphrase here.

1. Control for an X that freezes before Y starts.

2. Control for an X linked to some earlier step in a well-known sequence.

3. Control for an X that never changes while Y does change.

4. Control for an X that is generative (like socioeconomic status) as opposed to non-generative (like choice of toothpaste).

Q2. Since you have raised the question, others probably will too. Thus it's best to report both sets of results. But you have said "I want to account for these associations in my analysis." You must have reasons with which you'll make the case that one set is more informative and valid, but you should give readers the chance to make up their own minds as to which set tells the more compelling story.

Also please see this related post.

Answer 2

I'm not entirely sure you are talking about the same thing in your body and your question... nevertheless, to answer your questions:

1. Yes, you are (usually) trying to explain the effect of some variables on your outcome whilst controlling for known factors. If you do not include variables which have an effect on your outcome their effect might be transferred to some other variables which are not really related or have a transitory relationship with known factors, which are the ones having an effect on your outcome. By excluding known effects you are more likely to find spurious relationships.
2. Very carefully. In general the more variables you have in your model the less degrees of freedom, meaning less power and more uncertainty, hence why you can expect less certainty the more complex your model is.