In my analyses I had a clear hypothesis about a relationship Y ~ X1. In order to show that this is specific, I ran a control analyses with Y ~ X2. I showed that the relationship with X1 was significant, and the one with X2 not (we did this for several specific predictions). The reason that I should not correct for multiple comparisons (according to my colleague) is that the chance of having a Type I error is the same for the predicted effect as for the control effect. So the fact that I did not find an association with the control variables is another way for correcting. Is this a correct inference?
2
-
$\begingroup$ That doesn't sound right. There are ways to plan stepwise testing that preserve the type I error but it doesn't sound like this was done in a preplaned stepwise manner. It seems that you mainly worried about the type I error after the two tests conflicted. Am I right about this description? $\endgroup$– Michael R. ChernickOct 9, 2017 at 3:08
-
$\begingroup$ No, we had set out those hypotheses from the beginning. But the discussion came up when a friend said that I should still correct for multiple comparisons. However, my PI says that this is why we decided to include control variables. $\endgroup$– HILOct 9, 2017 at 3:50
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
This isn't the right way to do things. There is no reason to look at the relationship between Y and X2 in order to say something about the relationship between Y and X1. You haven't given a lot of detail on exactly what you did, but this can't be right in any situation that I can see.
Perhaps you are conflating significance testing with some aspect of validity? That is, if your goal was to show that Y is a valid measure of whatever you are trying to measure, one method would be to show that Y is related to things it ought to be related to and not related to things it ought not be related to.
answered Oct 9, 2017 at 11:48
$\begingroup$
$\endgroup$
X2 not significant and X1 significant does say nothing very strong at all about X1 being a better predictor than X2. This type of comparing significance is not the right way of going at this. It is not even worth discussing type I error control in this context. Depending on your question and context analyses adjusting for mA not significant and B significant does say nothing very strong at all about B being a better predictor than A. It is not even worth discussing type I error control in this context. May analyses adjusting for multiple covariates are an option (depends on exactly why you wish to do what you proposed).
In general, one would normally try to control the type I error rate across different analyses for which one would intend to "claim significance".
