I recently read an article that included a checklist aimed at improving the reporting of results in psychology. Among other things, they made the following suggestions:

  • If observations are eliminated, authors must also report what the statistical results are if those observations are included.
  • If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate.
  • Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.

From a philosophy of science point of view, their suggestions seem reasonable: Volatility caused by informed, but ultimately arbitrary decisions should be avoided. But running analyses again with different specifications tests the hypothesis that results will be the same, even if it is done in a naive "looks pretty much the same" kind of way. Say I designed an experiment, decided how to control the Type I error rate and have tested my a priori hypotheses. Afterwards I wonder how much it might have distorted the results that I polytomized a continuous variable and I want to see what would have happened had I included the variable in the continuous or a dichotomized version.

(The example was the first I could think of and is just a place holder for situations where there might have been another way to do it. The (in)appropriateness of polytomizing a continuous variable is of no interest at this point.)

I have been browsing the articles on experiment- and family-wise error rates, but they seem to have a different focus. I find it difficult to articulate the question, but I think what I want to ask is: If I re-ran a model with different specifications and obtained the same results, should that strenghten my confidence in the results (they don't hinge on that one decision) or weaken my confidence (because I tested the model multiple times and now the experiment-wise error rate is through the roof)?

My instinct says that checking a model stability is a different kind of testing and reaching the same conclusion through different specifications should strenghten my confidence. But everyday logic is not always the same as statistical logic, so I wonder if my reasoning is wrong.


Matt Krause suggested re-phrasing the question as: "Are multiple comparisons corrections necessary when testing model stability?"
I'm mostly interested in situations where you didn't or couldn't control the error rate beforehand to have some leeway for possible model stability tests.

Full source information: Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. http://doi.org/10.1177/0956797611417632

  • $\begingroup$ (+1) for an excellent question. Would it be fair to summarize your question as asking, "Are multiple comparisons corrections necessary when testing model stability?" (My gut instinct is that the answer is a combination of 'This is an issue with NHST' and 'No', but I'm not sure I have an amazing argument to share for either of those...) $\endgroup$ Oct 16, 2015 at 15:49
  • $\begingroup$ Also, welcome to Cross Validated! $\endgroup$ Oct 16, 2015 at 15:49
  • $\begingroup$ @Matt Krause Yes, that's a fair summary. I'm mostly interested in situations where you decide to test the stability after you have your model or where it is impractical to make Type I error control of the original model stricter than necessary to leave some "wiggle-room" for comparisons you didn't think of before. $\endgroup$
    – Lea
    Oct 16, 2015 at 16:02

2 Answers 2


After thinking about this some more, I shall try to answer my own question:

  • Null hypothesis significance testing (NHST) aims at inference. You want to draw conclusions about a population.
  • Testing model stability is about understanding what's happening in your data set, especially understanding the limitations of your data set. It's descriptive rather than inference statistics.

Both are done with the same methods of NHST (how else would you compare it?), but with a different rationale. Therefore, I would "exempt" tests of model stability from experiment-wise Type I error control, because you don't use them to draw conclusions about a population.
In fact, you use model stability testing as a different kind of error control, namely it alerts you to situations where you wanted to draw a conclusion but maybe shouldn't.
If someone would argue that it still inflates experiment-wise Type I error rates, I would call it an artifact of NHST that is not in the spirit of science.

PS. I've been mulling over this question for days and it took asking the question "out loud" to find an answer for myself. It reminds me of Heinrich von Kleist's 1805 essay On the Gradual Production of Thoughts Whilst Speaking (Über die allmähliche Verfertigung der Gedanken beim Reden).


The comments to this thread, including the great Kleist reference, led me down a slightly different path. One of Simonsohn, et al's (and others, going back at least to Neyman) critiques of current practice is the obsession with alpha and Type I errors -- aka "p-hacking." One antidote to this ongoing discussion is a recent Andrew Gelman post citing a Carl Morris article underscoring the importance of Type II errors and statistical power in driving confidence in one's findings.


In addition, there is Regina King's recent article in Science that does a good job of decomposing the controversy.


However, the original question concerned the stability of model results, not p-hacking or confidence. In my opinion, "stability" is an under-researched area in statistics (I would be interested in other's thoughts about that). I wanted to use this answer to develop some operational and applied heuristics for quantifying this otherwise vague notion.

Some of the quickest and easiest diagnostics wrt model instability is to decompose any collinearity in the model's structure since, as is well known, collinearity can lead to inflated standard errors which can contribute to unstable results. One of the myths about collinearity is that it's a product of small sample sizes, as is common with psychological research. Another myth is that it can be diagnosed with pairwise correlations. Neither is true. Collinearity is just as likely to occur in models with huge amounts of data where every relationship is statistically significant as it is in small datasets. Some of the best diagnostic evidence for underlying collinearity are partial or semi-partial correlations, not pairwise measures. Other diagnostics include "wrong" signed parameters, VIFs and eigenvalue decompositions that go back to Belsey, Kuh and Wallace's 80s book Regression Diagnostics.

A completely different and much more CPU intensive approach would be to leverage Monte Carlo simulation of your data -- Bayesian methods also work here. Based on the random draws, a range of parameters and outcomes across differing data "landscapes" can be generated which would provide empirical evidence towards understanding the stability of key model results. This could be done based on a coefficient of variation of the resulting simulated metrics, suggesting differential model strengths and weaknesses as a function of the magnitude of the variability. "Tornado"-type visualizations of this variability would highlight the range of significance in outcomes across all of the draws. This would help delineate those areas or combinations of model inputs and data that drive instability.

This may seem a bit vague as a solution to some but it is only intended as a heuristic. All of this work should only serve to deepen one's understanding of the model.

I'm very interested in hearing the suggestions of others.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.