3
$\begingroup$

Does any definitive work exist, e.g. a review, a book chapter, or a book, on the advantages and disadvantages of existing approaches? Is there a consensus on which approach is the right one?

I have come across three approaches so far:

  1. We test by a statistical test whether a given assumption of a statistical test is met. The problem with this is that the error of checking the assumption actually adds to the error of the test we originally wanted to perform. So it's misleading to report later the result of the latter test as, e.g. $p<0.05$ because, in fact, the test procedure's error rate is more than $5\%$. Is this accumulation of errors calculable?

  2. Later I was taught that the previous approach is heresy. Instead, we tested by Q-Q plots the probably most common assumption: normality. But it is also based on the sample on which we will later apply the test we want to perform. Is this relevant? Plus, it seems like all that’s happening is that we replace the statistical error with a subjective error because the figure needs to be assessed. This subjective error is certainly not calculable. It might be decreased by randomly generating as many data points as there are in our sample from a normal distribution parametrized based on the sample mean and SD, and comparing the Q-Q plots.

  3. We don't use parametric tests. Well, we use them only if we know a priori that their assumptions are met. When we don't know that for sure, we don't check anything, we use non-parametric tests.

$\endgroup$
4
  • 1
    $\begingroup$ Are you asking whether model diagnostics are post hoc and therefore invalid? For your second question, residuals in linear regression are correlated and nonnormal, so we already know that their normal probability plot is always "incorrect" but it still shows us how close they are to normal. $\endgroup$
    – Daeyoung
    Commented Mar 15, 2022 at 20:47
  • 1
    $\begingroup$ Well, model diagnostics can be an example of the second approach. When one handles the assumptions by graphical means. Another example can be checking the normality of the relevant variables in order to perform a Welch's t-test. An example of the first approach can be checking the equality of the variances of the relevant variables by performing an F-test in order to perform a Two-sample t-test. An example of the third approach can be performing a Welch's t-test instead of a Two-sample t-test so that we can avoid checking the equality of the variances of the relevant variables. $\endgroup$
    – rokamama
    Commented Mar 15, 2022 at 21:35
  • $\begingroup$ It's not clear to me what you really mean by "definitive" and "consensus" (consensus between whom exactly? How can you be sure when you have it and when you don't? Clearly there's not a consensus between the three different sets of people in your anecdote - nor an identification of what people were asserting each; is there some subset of them you care about having a consensus within?) Why is consensus important, rather than correctness? $\endgroup$
    – Glen_b
    Commented Mar 15, 2022 at 23:29
  • $\begingroup$ Maybe definitive was a strong word. I meant something of recognized excellence in discussing the topic. Well, there can be bad habits locally. I was curious about whether there is consensus in a wider statistical community. Correctness is important. I could have asked for the ultimate truth concerning the subject, but then there would have been comments saying it's a bit too much to ask for. $\endgroup$
    – rokamama
    Commented Mar 16, 2022 at 0:38

2 Answers 2

3
$\begingroup$

In my opinion semiparametric regression models provide the best default approach. They are invariant to Y-transformations and contain common nonparametric tests as special cases, but allow for much more than that. An overview is here.

As you rightly noted, assessing assumptions (whether by p-values or graphs) can make unknown changes to tests' operating characteristics.

$\endgroup$
4
  • $\begingroup$ What are Y-transformation? I am guessing they are transformations of the dependent variable but I am unsure. $\endgroup$
    – Galen
    Commented Mar 15, 2022 at 22:57
  • $\begingroup$ Correct; the transformation that you give Y or should have given Y (dependent variable). $\endgroup$ Commented Mar 16, 2022 at 2:07
  • $\begingroup$ Okay. I am having difficulty understanding one more thing about your answer then. How are these semiparametric regression models invariant to non-monotonic transformations? $\endgroup$
    – Galen
    Commented Mar 16, 2022 at 2:25
  • $\begingroup$ I should have clarified - they are invariant to order-preserving transformations, i.e., monotonic transformations. The principal type of transformations used in practice, i.e., log, $\sqrt{}$, reciprocal. $\endgroup$ Commented Mar 16, 2022 at 11:31
3
$\begingroup$

Your question is a laundry list of suggestions for doing one-sample tests of location, not an explicit question. So this is a laundry list of comments, not an explicit answer.

(1) Is deprecated, but perhaps more because tests of normality can be unreliable, than because of specific worries about error probabilities.

(2) Is often recommended, mainly because many statisticians feel it works better in practice than (1). Specific discussions of what looking at Q-Q plots might do to error probabilities are rare.

(3) Is a comment, not a strategy. Nonparametric tests do not assume normality, but they do have assumptions that need to be considered. Unless data are roughly symmetrical, it is not always clear what they are testing. (Location of population median?)

Something analogous is recommended for the choice of a two-sample test for a difference in normal population means. Unless there is a priori knowledge that group populations have equal variances, it is recommended to use the Welch two-sample t test instead of the pooled two-sample test. The loss in power. using the Welch test (which does not assume equal variances), is usually small compared to the risk of an incorrect decision, using the pooled test.

$\endgroup$
4
  • $\begingroup$ To supplement your answer on (3), often one of a few types of stochastic dominance are being tested. Sometimes we are interested in this, and sometimes not. $\endgroup$
    – Galen
    Commented Mar 15, 2022 at 22:11
  • $\begingroup$ @DifferentialCovariance: Stochastic dominance may apply more to two-sample tests than to one-sample tests, but is certainly worth mentioning here. $\endgroup$
    – BruceET
    Commented Mar 15, 2022 at 22:33
  • $\begingroup$ I'm sorry for posting a poor question. It was supposed to be a reference request primarily. And I wrote the list to sketch approaches I'm familiar with. I didn't think those were clear cut strategies. $\endgroup$
    – rokamama
    Commented Mar 16, 2022 at 0:51
  • $\begingroup$ At least, your Question seems to have covered points of interest, and now you have @FrankHerrell's answer with a link. IMHO, you should click the check mark to 'Accept' it. $\endgroup$
    – BruceET
    Commented Mar 16, 2022 at 1:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.