I need some advice from statistics experts. I often encounter in biomedical research field that people do one statistics test upon another on the same dataset. Let's say we have two groups of patients, some 'sick' and the other 'healthy control', and we are 'exploring' for explanations and therefore have performed statistics tests on many features, corrected for multiple testing and found a few that were significantly different between the groups. Let's say one of these significant features was the presence of a certain bacteria. Then a second hypothesis based upon this is formed which is that patients in the different groups have different levels of antibodies to this bacteria and this is then tested. I would say that it is not correct to form new hypotheses based on a dataset and then test them in the same dataset. What is your take on this? Is there a way to get away with the above problem?
2 Answers
I think your intuition is solid here. The second test you mention sounds very suspicious in this context, and my view is that it would not be appropriate to test this without a further adjustment for multiple comparisons (which would be extremely complicated and possibly prohibitive). As a general rule, when we "adjust for multiple comparisons" we are essentially adjusting the null distribution of a statistical test to condition on all the testing coming before/concurrently with that test. That is certainly going to be required in this case, and it will not be easy.
There is good reason to believe that the presence or absence of antibodies to the bacteria would affect the association between the bacteria variable and the sickness outcome in the first test. (Indeed, there is a plausible causal relationship between these variables, which could be quite strong.) Consequently, conditional on the result of the first test, the null distribution of the second test would not be the same as if the first test had not been performed to get there. In practice, the conditional null distribution for the second test would be complicated, because it is conditional on the outcome of an optimisation result in the first test involving multiple variables that are related to the variables in the second test.
In such circumstances, it is difficult to "adjust" the second test to take account of the first test, and it would require some heavy theoretical development. I wouldn't rule out the possibility that some clever statistician could come up with a testing sequence that properly adjusts for this, but it would be a difficult theoretical exercise which would probably constitute a publishable paper in its own right. Unless you can find an existing sequential test that has been developed for this purpose (and I'm not aware of one) you would either need to undertake some heavy theoretical work to develop one, or else write off this comparison and test it with new data instead.
Testing lots of hypotheses is not a problem in a preliminary study, and forming new hypotheses from those tests is good. However, if the newly formed hypotheses are chosen on the basis that the preliminary data suggest that there is something there, then you really cannot 'test' the new hypothesis using those same data, just as you suspect. Hypotheses selected on the basis of preliminary data should be tested using new data.
The problem that comes from testing a hypothesis with the data that first suggested the hypothesis is that the data are always going to support the hypothesis! After all, that's why the hypothesis was chosen in the first place. The false discovery rate and false positive rate for the procedure will be far higher than the nominal level.
On the specifics that you mention, the situation might be subtly different if the antibody levels were not measured in the original data collection. Measuring the antibody levels afterwards goes some way towards being the new data necessary for testing the hypotheses that were selected from the original study. However, it seems to me that the antibody levels in patients stratified into groups on the basis of the presence and absence of bacteria might tell you about the relationship between infection and antibodies, but it cannot tell you much about the original condition of grouping, sickness.
