So I have heard it said that it is not a good idea to choose one statistical test based on the outcome of another. This seems strange to me though. For example, people often choose to use a non parametric test when some other test suggests that the residuals are not normally distributed. This approach seems pretty widely accepted but does not seem to agree with the first sentence in this paragraph. I was just hoping to get clarification on this issue.

  • 3
    $\begingroup$ Just because residuals aren't Gaussian doesn't mean you need non-parametric tests. You can usually discern the type of model to use (yes model, not test) from the nature of the data (count, 0 1 data, continuous, mean-variance relationship, linear or non-linear relationship, etc) and fit models accordingly to meet the features of the data having previously decide what the hypothesis to be tested was. Once you feel the fit meets the assumptions of the model fitted then you can evaluate the p-value and other statistics, $\endgroup$ Jun 14 '13 at 4:01

Given that $p$ is the probability of observing data this extreme or more extreme if $H_0$ is true, then what is the interpretation of $p$ where the $p$ is arrived at through a process where there was a contingent decision made in the selection of the test that produced that $p$? The answer is unknowable (or at least very nearly unknowable). By making the decision to run the test or not on the basis of some other probabilistic process you've made the interpretation of your outcome even more convoluted. $p$ values are maximally interpretable when the sample size and analysis plan was fully selected in advance. In other situations, the interpretations get difficult, that is why it is 'not a good idea'. That being said, it is a widely accepted practice... after all, why even bother to run a test if you find out that the test you had planned to run was invalid? The answer to that question is far less certain. This all boils down to the simple fact that null hypothesis significance testing (the primary use case of $p$) has some problems that are difficult to surmount.

  • $\begingroup$ I was unable to find any articles discussing this phenomenon on Google, possibly because I used the wrong search terms. Would somebody be able to point me in the direction of an article that discusses the problem of tests based on tests? $\endgroup$
    – Rob Hall
    Oct 8 '13 at 16:25
  • 1
    $\begingroup$ @RobHall: This is a specific instance of "The importance of hypothetical issues for imaginary data". Cf. Wagenmakers, 2007, p. 784. Wagenmakers specifically draws in the issue of transformations in the second column stating "in order to calculate a p value, you need to know what you would have done had the data turned out differently... this includes what you would have done if data had clearly been nonnormally distributed ... , p values can only be computed once the sampling plan is fully known and specified in advance". $\endgroup$ Oct 9 '13 at 14:13

For example, people often choose to use a non parametric test when some other test suggests that the residuals are not normally distributed. This approach seems pretty widely accepted but does not seem to agree with the first sentence in this paragraph. I was just hoping to get clarification on this issue.

Yes, a lot of people do this kind of thing, and change their second test to one that can deal with heteroskedasticity when they reject equality of variance, and so on.

Just because something is common, doesn't mean it's necessarily wise.

Indeed, in some places (I won't name the worst-offending disciplines) a lot of this formal hypothesis testing contingent on other formal hypothesis testing is actually taught.

The problem with doing it is that your procedures don't have their nominal properties, sometimes not even close. (On the other hand, assuming things like that without any consideration at all for potentially extreme violation could be even worse.)

Several papers suggest that for the heteroskedastic case, you're better off simply acting as if the variances aren't equal than to test for it and only do something about it on rejection.

In the normality case it's less clear. In large samples at least, in many cases normality isn't all that crucial (but ironically, with large samples, your test of normality is much more likely to reject), as long as the non-normality isn't too wild. One exception is for prediction intervals, where you really do need your distributional assumption to be close to right.

In part, one problem is that hypothesis tests answer a different question than the one that needs to be answered. You don't really need to know 'is the data truly normal' (almost always, it won't be exactly normal a priori). The question is rather 'how badly will the extent of non-normality impact my inference'.

The second issue is usually either just about independent of sample size or actually gets better with increasing sample size - yet hypothesis tests will almost always reject at large sample sizes.

There are many situations where there are robust or even distribution free procedures which are very close to fully efficient even at the normal (and potentially far more efficient at some fairly modest departures from it) - in many cases it would seem silly not to take the same prudent approach.

  • $\begingroup$ Nice (+1) Could you give a reference to the articles you mention about the heteroskedastic case? $\endgroup$
    – gui11aume
    Jun 14 '13 at 9:09
  • 2
    $\begingroup$ I don't wish to point any out, but I run across them online all the time, so it's not hard to figure out which ones tend to emphasize it (they tend to be the same ones that historically over-emphasize hypothesis testing). Indeed the disciplines of the people generating questions here where posters think they have to use formal tests would usually be the same ones. It's not just one or two disciplines - I see many - but some do seem to do it especially often. For it to be reasonably common, I can only assume there's been particularly well-known texts in those areas that insisted on it. $\endgroup$
    – Glen_b
    Jun 14 '13 at 9:48
  • 1
    $\begingroup$ @gui11aume Here's a reference ... it's not one of the ones I was looking for, but it does make the point I was getting at (that preliminary testing can make things worse). $\endgroup$
    – Glen_b
    Jun 14 '13 at 10:05
  • 2
    $\begingroup$ Andrew Gelman had a related post recently about heterogeneity between groups that is related (at least about why such a process is problematic). $\endgroup$
    – Andy W
    Jun 14 '13 at 12:39
  • 1
    $\begingroup$ A question related to these discussions from a while back: stats.stackexchange.com/questions/305/… $\endgroup$ Jun 22 '13 at 23:57

The main issues have been well explained by others, but are confounded with underlying or associated

  1. Over-reverence for P-values, at most one kind of evidence in statistics.

  2. Reluctance to see that statistical reports are inevitably based on a combination of choices, some firmly evidence-based, others based on a mix of previous analyses, intuition, guesswork, judgment, theory, so forth.

Suppose I and my cautious friend Test Everything both chose a log transformation for a response, but I jump to that conclusion based on a mix of physical reasoning and previous experience with data, while Test Everything chooses log scale based on Box-Cox testing and estimation of a parameter.

Now we both use the same multiple regression. Do our P-values have different interpretations? On one interpretation, Test Everything's P-values are conditional on her previous inferences. I used inferences too, but mostly they were informal, based on a long series of previous graphs, calculations, etc. in previous projects. How is that to be reported?

Naturally, the regression results are exactly the same for Test Everything and myself.

The same mix of sensible advice and dubious philosophy applies to choice of predictors and functional form. Economists, for example, are widely taught to respect previous theoretical discussions and to be wary of data snooping, with good reason in each case. But in the weakest instances the theory concerned is just a tentative suggestion made previously in the literature, very likely after some empirical analysis. But literature references sanctify, while learning from the data in hand is suspect, for many authors.

  • $\begingroup$ Very clear (+1). $\endgroup$
    – gui11aume
    Jun 14 '13 at 9:10
  • 1
    $\begingroup$ +1. There is a long-run difference in the performance of your analyses vs. Test Everything's analyses, however. Every time this analysis is run, you will use the same strategy, based on what was written in the literature (which doesn't fluctuate experiment by experiment). OTOH, the data are a random sample, & the output from Box-Cox testing will fluctuate study by study. $\endgroup$ Jun 20 '13 at 21:55
  • $\begingroup$ That's droll, but my experience does change too, long-run. $\endgroup$
    – Nick Cox
    Jun 20 '13 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.