8
$\begingroup$

Suppose we have a standard regression model and want to identify whether we have violated the assumptions of the model. Traditionally, we might utilize a significance test to determine whether (for example) the residuals are normally distributed (where the null model states that the residuals belong to a normal distribution).

I think there's consensus that using hypothesis tests to evaluate model assumptions isn't the best idea. These tests are sensitive to sample size (failing to detect major departures when N is small, and detecting even minor discrepancies when N is large). Personally, I think the biggest problem with these tests is they evaluate whether violated assumptions can be detected, not whether they are problematic.

When I evaluate assumptions, I exclusively use visuals. Unfortunately, visuals are more subjective (and they don't really tell you whether the observed violation is problematic). So, here's my question:

Are there any objective ways to evaluate statistical assumptions that do not rely on significance tests?

And, do these objective means of evaluating assumption evaluate whether the assumption violation matters, rather than whether it can be detected?

I'd love to see references to articles that I can study, if you have it. Thanks!

Edit

Some of the comments/answers mention that it depends on what one's research goals are. Yes, I absolutely agree and appreciate you all pointing this out. I suppose what I'm asking for is some literature that says, "If your goal is A, index B is appropriate. If your goal is C, criteria D is appropriate."

Improve this question
$\endgroup$
7
  • $\begingroup$ Your bolded sentence seems to be missing a word (or perhaps two). One can of course discern the gist of what you intended but specific word choice might change an answer. $\endgroup$
    – Glen_b
    May 21 at 5:17
  • $\begingroup$ Good catch. I'll correct that. $\endgroup$
    – dfife
    May 25 at 13:52
  • 2
    $\begingroup$ I would offer a counterargument to your "visuals are more subjective." First, that requires some justification. If the answer is "people often misinterpret the graphics" or "people often disagree," then we might think to question the qualifications of such people rather than infer the graphics are to blame. Otherwise, appropriate graphics offer far more information about the nature of the data. When correctly interpreted, those graphics can be more useful and more objective than the results of some putatively objective test (that itself relies on yet more assumptions...). $\endgroup$
    – whuber ♦
    May 25 at 14:15
  • 2
    $\begingroup$ For references, start with Tukey's EDA. Although that book says absolutely nothing about testing assumptions--it doesn't even talk about probability models or related assumptions--it shows how graphics can be deployed in quantitative, insightful ways to assess deeply hidden characteristics of data. Those techniques, in conjunction with modern simulation techniques (to avoid being fooled by apparent patterns), offer more than any battery of assumptions tests could. $\endgroup$
    – whuber ♦
    May 25 at 14:18
  • 1
    $\begingroup$ @whuber--You're exactly anticipating the direction of the paper I'm writing. We're trying to argue that visuals are far superior to "tests," and offering suggestions on how to be more informed. This question is really to help us frame our solution. $\endgroup$
    – dfife
    May 25 at 14:21

2 Answers 2

Reset to default
9
$\begingroup$

This is a good question, as it acknowledges that the issue is not whether model assumptions are fulfilled or not (they never are), but rather whether violations of the model assumption matter (in terms of misleading conclusions).

Unfortunately I tend to answer this question with "no". A problem is this: The statistical problem of interest is well defined within the framework of the assumed model, but if model assumptions are violated, there is no unique objective way to define what it actually means to say that conclusions are misled.

Here is an illustration. Let's say you are running a test about the mean of a normal distribution, but the underlying distribution is in fact skew. Now your sample size may be fairly large and there may not be any indication that second moments do "explode", so the Central Limit Theorem may justify normal theory to hold approximately (violation of normality could therefore be seen as not problematic). However, in most skew distributions mean, mode, and median are different, whereas in the normal distribution they are the same, meaning that even though the CLT authorises inference about the mean, it isn't clear whether in your specific application you should rather be interested in median or mode in order to summarise your distribution. Under the normal assumption this doesn't matter, but if the underlying distribution is in fact skew, it usually does, and the answer whether the normal assumption theory is fine or not depends on this.

Here is a paper I had my hand in on investigating the quality of formal misspecification tests for assumptions in testing, with connected considerations. Overall we're not as negative about this approach as some others, even though we agree that in a good number of situations it is not very good. But sometimes it's fine. Unfortunately this depends on a number of specifics of the situation.

M. I. Shamsudheen & C. Hennig: Should we test the model assumptions before running a model-based test? https://arxiv.org/abs/1908.02218

Two maybe interesting papers that we cite:

Zimmerman DW (2011) A simple and effective decision rule for choosing a significance test to protect against non-normality. British Journal of Mathematical and Statistical Psychology 64:388-409 (Here an objective rule different from a formal assumptions test is proposed to choose between a t-test and a nonparametric test; namely to use the nonparametric test in case they have very different results.)

Spanos A (2018) Mis-specification testing in retrospect. Journal of Economic Surveys 32:541–577 (good and thoughtful survey paper even though I don't agree with everything)

I believe that developing decision rules between model-based and alternative procedures that have better performance than standard misspecification tests is a very promising research area (even though there will always have to be some "subjective" decision making, see above), but as far as I know there isn't much.

PS: Regarding the Zimmerman paper, I have seen a similar suggestion for linear regression somewhere, namely to run a least squares and a robust regression, and to use the least squares one if both are reasonably in line (of course one can define this "objectively"), and otherwise to use the robust fit. I don't remember exactly where that was anymore, somewhere in the robustness literature, and as far as I remember, it was defined but not much (or even nothing) was done to compare its quality to alternative approaches including running either least squares or robust fit all the time.

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Since you mentioned Spanos (2018) and misspecification testing, here is a related thread. $\endgroup$
    – Richard Hardy
    May 20 at 18:07
  • $\begingroup$ That looks like the exact sort of thing I'm looking for. Thanks! $\endgroup$
    – dfife
    May 20 at 18:22
  • $\begingroup$ @RichardHardy Thanks for this, hadn't seen it before. There is a comment of Spanos prompted by a presentation of mine here, with a long reply of mine. $\endgroup$
    – Christian Hennig
    May 20 at 19:22
  • $\begingroup$ The general stance of this I like. But -- as a point of detail -- it's easy to find skewed distributions in which mean, median and mode are identical, and they aren't pathological either. Binomial distributions with integer means provide examples. $\endgroup$
    – Nick Cox
    May 25 at 14:00
  • 1
    $\begingroup$ I don't regard subjectivity as a bugbear. There are judgments at all levels in statistical science. Some that I understand I might regard as good judgment and others bad judgment. I hope that's banal; and what matters is how far what someone else did is transparent so that we have scope to discuss it if we care. (These matters came up in your paper with Andrew Gelman.) $\endgroup$
    – Nick Cox
    May 25 at 14:05
8
$\begingroup$

This is not a complete answer, and it's not exactly "objective", but a useful tool is what is known as posterior predictive checks in a Bayesian context (but probably by other names elsewhere). In this, you simulate data from the distribution implied by your model, and compare it to your actual data, usually by plotting them side-by-side in whatever fashion makes sense for your problem. Mismatches then suggest assumptions that don't hold - e.g. your simulated data is symmetrical where the real data is skewed, or the real data has fatter tails.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ This is certainly a reasonable approach, but what OP is asking for is something objective, which is problematic for the reasons Christian writes. $\endgroup$
    – Bryan Krause
    May 20 at 18:21
  • 1
    $\begingroup$ I totally agree! $\endgroup$
    – Eoin
    May 20 at 18:21
  • 3
    $\begingroup$ But this is still helpful. I'm doing a literature review on the topic in a paper, so this sort of thing is excellent. $\endgroup$
    – dfife
    May 20 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.