I recently read an article that included a checklist aimed at improving the reporting of results in psychology. Among other things, they made the following suggestions:
- If observations are eliminated, authors must also report what the statistical results are if those observations are included.
- If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate.
- Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.
From a philosophy of science point of view, their suggestions seem reasonable: Volatility caused by informed, but ultimately arbitrary decisions should be avoided. But running analyses again with different specifications tests the hypothesis that results will be the same, even if it is done in a naive "looks pretty much the same" kind of way. Say I designed an experiment, decided how to control the Type I error rate and have tested my a priori hypotheses. Afterwards I wonder how much it might have distorted the results that I polytomized a continuous variable and I want to see what would have happened had I included the variable in the continuous or a dichotomized version.
(The example was the first I could think of and is just a place holder for situations where there might have been another way to do it. The (in)appropriateness of polytomizing a continuous variable is of no interest at this point.)
I have been browsing the articles on experiment- and family-wise error rates, but they seem to have a different focus. I find it difficult to articulate the question, but I think what I want to ask is: If I re-ran a model with different specifications and obtained the same results, should that strenghten my confidence in the results (they don't hinge on that one decision) or weaken my confidence (because I tested the model multiple times and now the experiment-wise error rate is through the roof)?
My instinct says that checking a model stability is a different kind of testing and reaching the same conclusion through different specifications should strenghten my confidence. But everyday logic is not always the same as statistical logic, so I wonder if my reasoning is wrong.
Matt Krause suggested re-phrasing the question as:
"Are multiple comparisons corrections necessary when testing model stability?"
I'm mostly interested in situations where you didn't or couldn't control the error rate beforehand to have some leeway for possible model stability tests.
Full source information: Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. http://doi.org/10.1177/0956797611417632