4
$\begingroup$

I am comparing glutamate concentrations across groups (2 groups). I planned on controlling for age and IQ, simply because this is what is always done in cognitive neuroscience. When doing this, age was a very useful co-variate but IQ makes the model very very messy.

I am thinking of excluding it for the following reasons...

Reason 1: When IQ is added as a co-variate, it makes the variance in the sample unequal (Levene's test). When it is out, Levene's test is less significant (although still significant). I have tried to transform the data and this makes no difference.

Reason 2: Both groups are well matched on age and IQ. There are no significant differences. However, even though they are well matched - age is a significant co-variate to glutamate. This association is not significant for IQ and glutamate.

Reason 3: There is an extremely week linear relationship between IQ and glutamate concentrations (see below). I have only included the output for the control group, but it is similar for the other group. It is also not correlated with glutamate (Pearson Correlation = .203, sig=.213, N=33).
enter image description here

My main problem is that I get different results depending on what I do. With IQ in the model, there are no significant differences between groups. With IQ out, there are significant differences. For greater simplicity, I am thinking of excluding IQ as a co-variate. I don't have a very large sample size (N=33). How justified am I in doing this? Will I be eaten alive by reviewers even if I have a justified reason for excluding it from the model?

Question 2: If there is a significant difference in variance between groups. Is this meaningful and what can I conclude from this? Most of the information about unequal variance reports that it needs to be cleaned up. However, can I make any useful inferences about the fact that my groups have different variances of glutamate?

Question 3: Does it make any difference statistically if you use a Type II or Type III sum of squares. From reading it looks like a Type II is preferred if you don't have any significant interactions and Type III if you do. What if you have some significant interactions for other metabolites and no significant interactions for my main metabolite of glutamate(I am looking at other things besides glutamate). I assume it must be standardized and you should go with either or for the entire analysis. What's the best way forward in this regard?

Thank you for any help you can provide. I have been agonizing about these problems and just want to represent the data in the most appropriate way.

$\endgroup$
  • $\begingroup$ What exactly do you mean when you say that the model becomes "very very messy"? $\endgroup$ – amoeba says Reinstate Monica Aug 21 '15 at 13:09
  • 3
    $\begingroup$ Also, be aware of multiple testing. If including IQ as a covariate had made your group difference significant you would probably not have hesitated to include it. In your case it makes it insignificant; so you want to exclude it. This is a dangerous path that can lead to what people nowadays call "p-hacking". $\endgroup$ – amoeba says Reinstate Monica Aug 21 '15 at 13:13
  • $\begingroup$ You are completely right. I was mostly concerned about getting a very different result (significant and not significant) based on the inclusion of IQ and also the significantly different variance. I have seen some suggestion that when variance between groups is significantly different, it can sometimes help to simplify the model. IQ makes logical sense in the model and was included in our original thesis, therefore, it clearly will clearly remain in our conclusions. Thank you for your answer. It was really helpful. $\endgroup$ – brainimaging_new_to_unix Aug 27 '15 at 15:47
3
$\begingroup$

First, let me assure you that - as mentioned by @amoeba - you are on the right path to land in "research hell", that is the place where researchers (should) go when they let their p-value to decide what to include or not in the analysis.

Reason 1.

You have to decide a priori whether you consider Levene's test a good test for heteroscedasticity or not, and what to do in case the test's significant/not significant. Also, while you and I may disagree, if you use thresholds to make decisions (and this is the still the standard in classic statistics, unfortunately) whether something is less or more significant has little importance.

Reason 2 and 3.

"Both groups are well matched on age and IQ", this is good. One of ANCOVA most important assumption is independence of the covariate and treatment effect. "This association is not significant for IQ and glutamate", I am not completely sure this is a problem, I actually believe it isn't. The second most important assumption for ANCOVA, in fact, is that the co-variates have the same relationship with the dependent variable regardless of the independent variables. This is usually called Homogeneity of regression slopes.

Question 1.

The problem here is not that you are going to be eaten alive; likely, you won't (I have seen worse analysis being published...). The problem is that you are thinking about publishing results that are the results of an analysis you are not completely sure of after you have violated the most important rule in experimental data analysis which is not to make changes based on the final outcome.

Question 2.

In theory, yes: heteroscedasticity could be the sign of something else going on in your data. Or not. We don't (can't) know. However, given your small sample size I would argue against continuing with your analysis without fixing the problem of unequal variance. There are techniques you may want to consider, they may or may not work. But, if they don't work, you are left with only one possibility: do not conduct your analysis (since you have already done that: do not report the results).

Question 3.

Yes, it does. Again, you should decide a priori. Type II is preferable if you are interested in main effects while Type III is preferable when you anticipate interactions. I tend to prefer Type III but a debate over which one is better still exists.

Conclusion.

My suggestion is not to publish these results and to try to collect more data. Some of the assumptions you broke become less important with bigger sample (60+). That said, you should never look at the final results before being sure you have done everything correctly because, in theory, that is a point of no return.

$\endgroup$
  • $\begingroup$ Thank you for your answer and currently looking into increasing our sample size to more clearly answer our research question. To clarify, I was not trying to change the outcome of the statistical test, but was concerned about the different results obtained with/ without IQ included in the model (significant vs insignificant results and the changes in the Levene's test). Most of the statistical resources that I used always tested for differences in variance, but were not clear about what to do when these assumptions were violated. Thanks for your very helpful answer in this regard. $\endgroup$ – brainimaging_new_to_unix Aug 27 '15 at 16:04
  • $\begingroup$ However, advice very much taken, I will stick with the initial hypothesis and not be concerned about how slight variations in the model impact the outcome. $\endgroup$ – brainimaging_new_to_unix Aug 27 '15 at 16:07
4
$\begingroup$

First, ask yourself why IQ is always included in such models. There is probably some reason. It might be that IQ is a mediator (see below)

Second, from what you say, it sounds like IQ is a type of mediator of the relationship between glutamate concentration and whatever your group variable is. Matching will not deal with mediating relationships. The correct way to establish mediation is not completely agreed on (even the terminology is not completely settled), but my view is that statistical significance has little role in it. The key thing is not whether the relationships are significant or not (with N = 33, significance is hard) but changes in the parameter estimates.

Third, the fact that "the model becomes messy" is no reason to exclude a variable. Not all relationships are simple. To exclude a mediator can give a very wrong picture of a relationship.

$\endgroup$
  • $\begingroup$ Firstly, thank you very much for you answer. Thinking about your answer, I agree. IQ is a standard co-variate and this is for good reason. So, significant or not, it stays. $\endgroup$ – brainimaging_new_to_unix Aug 27 '15 at 15:40
2
$\begingroup$

I'm a bit worried about the y-axis label on your plot, "C glutamate SD less than 20 - extremes," which has two potentially important implications.

For one, it might be taken to suggest that there has already been some removal of "outlier" determinations, which is tricky business. This should usually only be done if you know that measurements were in error (as opposed to messy). A possible interpretation of your axis label is that there were multiple analytical determinations of glutamate for each individual, each determination involving more than 1 technical replicate, and that some determinations were excluded if the technical replicates disagreed with an SD more than 20. If the "- extremes" means that you further removed the extreme values of the determinations for an individual, then that's an additional issue to consider.

Second, if you have analysis SDs on the order of 20 with mean values on the order of 9, and glutamate concentrations certainly cannot go below 0, then you probably should not be analyzing your glutamate analyses on a linear scale, at least in terms of combining analytical results to obtain a glutamate value for each individual. My guess is that the analytical errors in glutamate determinations are more or less proportional to the values measured, so for the glutamate-analysis part of this work you would be better off working on a log scale so that magnitudes of analytical errors are independent of the measured values, on that scale. On a log scale some of your "outliers" might not be so far off, and your results might be more reliable (and potentially even in support of your hypothesis).

The "weak" relation between IQ and glutamate that you cite (Pearson Correlation = .203, sig=.213, N=33) is not necessarily so weak. Trying to rule out a relation between two variables is different from trying to demonstrate a significant relation between them. That correlation coefficient isn't atypical of many biological relationships, and the lack of "significance" might simply represent the small number of cases, so that's not a reason to exclude IQ.

Part of the problem here is an under-powered experimental design, as you seem to understand. If controlling for age and IQ is typically expected in this type of study, then there needed to be enough cases to accommodate that. Each additional covariate uses up a degree of freedom in your analysis, potentially making it harder to detect significance if the covariate bears only a weak relation to the outcome variable. It is not unusual to find "significance" with a small number of predictors, which then disappears as extra predictors are added.

If I am correct about the nature of your glutamate determinations, you will need to re-evaluate these relationships in any event.

$\endgroup$
  • $\begingroup$ Thank you for your comment. Regarding your first point, we did exclude 1 participant from the sample that had values more than 3 SD from the mean. This participant was well beyond the values of our sample. The standard deviation of 20 is referring to something else (not related to my question). In fact, the SD was between 1 and 2 (rather than 20). $\endgroup$ – brainimaging_new_to_unix Aug 27 '15 at 16:12
  • $\begingroup$ We are currently looking into increasing the sample same size and have decided that it is important to keep IQ in the model. Thank you very much for your incredibly helpful answer. Sorry it took me so long to respond as I have been thinking about all the responses I received. $\endgroup$ – brainimaging_new_to_unix Aug 27 '15 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.