Do I have a justified reason to exclude a non-significant covariate from my ANCOVA? How interesting is unequal variance?

Question

I am comparing glutamate concentrations across groups (2 groups). Age was significantly correlated with glutamate while IQ was not.
If there is a significant difference in variance between groups. Is this meaningful and what can I conclude from this? Most of the information about unequal variance reports that it needs to be cleaned up. However, can I make any useful inferences about the fact that my groups have different variances of glutamate?
Does it make any difference statistically if you use a Type II or Type III sum of squares. From reading it looks like a Type II is preferred if you don't have any significant interactions and Type III if you do. What if you have some significant interactions for other metabolites and no significant interactions for my main metabolite of glutamate(I am looking at other things besides glutamate). I assume it must be standardized and you should go with either or for the entire analysis. What's the best way forward in this regard?
Thank you for any help you can provide. I have been agonizing about these problems and just want to represent the data in the most appropriate way.

Answer 1

First, let me assure you that - as mentioned by @amoeba - you are on the right path to land in "research hell", that is the place where researchers (should) go when they let their p-value to decide what to include or not in the analysis.

Reason 1.

You have to decide a priori whether you consider Levene's test a good test for heteroscedasticity or not, and what to do in case the test's significant/not significant. Also, while you and I may disagree, if you use thresholds to make decisions (and this is the still the standard in classic statistics, unfortunately) whether something is less or more significant has little importance.

Reason 2 and 3.

"Both groups are well matched on age and IQ", this is good. One of ANCOVA most important assumption is independence of the covariate and treatment effect. "This association is not significant for IQ and glutamate", I am not completely sure this is a problem, I actually believe it isn't. The second most important assumption for ANCOVA, in fact, is that the co-variates have the same relationship with the dependent variable regardless of the independent variables. This is usually called Homogeneity of regression slopes.

Question 1.

The problem here is not that you are going to be eaten alive; likely, you won't (I have seen worse analysis being published...). The problem is that you are thinking about publishing results that are the results of an analysis you are not completely sure of after you have violated the most important rule in experimental data analysis which is not to make changes based on the final outcome.

Question 2.

In theory, yes: heteroscedasticity could be the sign of something else going on in your data. Or not. We don't (can't) know. However, given your small sample size I would argue against continuing with your analysis without fixing the problem of unequal variance. There are techniques you may want to consider, they may or may not work. But, if they don't work, you are left with only one possibility: do not conduct your analysis (since you have already done that: do not report the results).

Question 3.

Yes, it does. Again, you should decide a priori. Type II is preferable if you are interested in main effects while Type III is preferable when you anticipate interactions. I tend to prefer Type III but a debate over which one is better still exists.

Conclusion.

My suggestion is not to publish these results and to try to collect more data. Some of the assumptions you broke become less important with bigger sample (60+). That said, you should never look at the final results before being sure you have done everything correctly because, in theory, that is a point of no return.

Answer 2

First, ask yourself why IQ is always included in such models. There is probably some reason. It might be that IQ is a mediator (see below)

Second, from what you say, it sounds like IQ is a type of mediator of the relationship between glutamate concentration and whatever your group variable is. Matching will not deal with mediating relationships. The correct way to establish mediation is not completely agreed on (even the terminology is not completely settled), but my view is that statistical significance has little role in it. The key thing is not whether the relationships are significant or not (with N = 33, significance is hard) but changes in the parameter estimates.

Third, the fact that "the model becomes messy" is no reason to exclude a variable. Not all relationships are simple. To exclude a mediator can give a very wrong picture of a relationship.

Answer 3

I'm a bit worried about the y-axis label on your plot, "C glutamate SD less than 20 - extremes," which has two potentially important implications.

For one, it might be taken to suggest that there has already been some removal of "outlier" determinations, which is tricky business. This should usually only be done if you know that measurements were in error (as opposed to messy). A possible interpretation of your axis label is that there were multiple analytical determinations of glutamate for each individual, each determination involving more than 1 technical replicate, and that some determinations were excluded if the technical replicates disagreed with an SD more than 20. If the "- extremes" means that you further removed the extreme values of the determinations for an individual, then that's an additional issue to consider.

Second, if you have analysis SDs on the order of 20 with mean values on the order of 9, and glutamate concentrations certainly cannot go below 0, then you probably should not be analyzing your glutamate analyses on a linear scale, at least in terms of combining analytical results to obtain a glutamate value for each individual. My guess is that the analytical errors in glutamate determinations are more or less proportional to the values measured, so for the glutamate-analysis part of this work you would be better off working on a log scale so that magnitudes of analytical errors are independent of the measured values, on that scale. On a log scale some of your "outliers" might not be so far off, and your results might be more reliable (and potentially even in support of your hypothesis).

The "weak" relation between IQ and glutamate that you cite (Pearson Correlation = .203, sig=.213, N=33) is not necessarily so weak. Trying to rule out a relation between two variables is different from trying to demonstrate a significant relation between them. That correlation coefficient isn't atypical of many biological relationships, and the lack of "significance" might simply represent the small number of cases, so that's not a reason to exclude IQ.

Part of the problem here is an under-powered experimental design, as you seem to understand. If controlling for age and IQ is typically expected in this type of study, then there needed to be enough cases to accommodate that. Each additional covariate uses up a degree of freedom in your analysis, potentially making it harder to detect significance if the covariate bears only a weak relation to the outcome variable. It is not unusual to find "significance" with a small number of predictors, which then disappears as extra predictors are added.

If I am correct about the nature of your glutamate determinations, you will need to re-evaluate these relationships in any event.