So I have heard it said that it is not a good idea to choose one statistical test based on the outcome of another. This seems strange to me though. For example, people often choose to use a non parametric test when some other test suggests that the residuals are not normally distributed. This approach seems pretty widely accepted but does not seem to agree with the first sentence in this paragraph. I was just hoping to get clarification on this issue.
Given that $p$ is the probability of observing data this extreme or more extreme if $H_0$ is true, then what is the interpretation of $p$ where the $p$ is arrived at through a process where there was a contingent decision made in the selection of the test that produced that $p$? The answer is unknowable (or at least very nearly unknowable). By making the decision to run the test or not on the basis of some other probabilistic process you've made the interpretation of your outcome even more convoluted. $p$ values are maximally interpretable when the sample size and analysis plan was fully selected in advance. In other situations, the interpretations get difficult, that is why it is 'not a good idea'. That being said, it is a widely accepted practice... after all, why even bother to run a test if you find out that the test you had planned to run was invalid? The answer to that question is far less certain. This all boils down to the simple fact that null hypothesis significance testing (the primary use case of $p$) has some problems that are difficult to surmount.
For example, people often choose to use a non parametric test when some other test suggests that the residuals are not normally distributed. This approach seems pretty widely accepted but does not seem to agree with the first sentence in this paragraph. I was just hoping to get clarification on this issue.
Yes, a lot of people do this kind of thing, and change their second test to one that can deal with heteroskedasticity when they reject equality of variance, and so on.
Just because something is common, doesn't mean it's necessarily wise.
Indeed, in some places (I won't name the worst-offending disciplines) a lot of this formal hypothesis testing contingent on other formal hypothesis testing is actually taught.
The problem with doing it is that your procedures don't have their nominal properties, sometimes not even close. (On the other hand, assuming things like that without any consideration at all for potentially extreme violation could be even worse.)
Several papers suggest that for the heteroskedastic case, you're better off simply acting as if the variances aren't equal than to test for it and only do something about it on rejection.
In the normality case it's less clear. In large samples at least, in many cases normality isn't all that crucial (but ironically, with large samples, your test of normality is much more likely to reject), as long as the non-normality isn't too wild. One exception is for prediction intervals, where you really do need your distributional assumption to be close to right.
In part, one problem is that hypothesis tests answer a different question than the one that needs to be answered. You don't really need to know 'is the data truly normal' (almost always, it won't be exactly normal a priori). The question is rather 'how badly will the extent of non-normality impact my inference'.
The second issue is usually either just about independent of sample size or actually gets better with increasing sample size - yet hypothesis tests will almost always reject at large sample sizes.
There are many situations where there are robust or even distribution free procedures which are very close to fully efficient even at the normal (and potentially far more efficient at some fairly modest departures from it) - in many cases it would seem silly not to take the same prudent approach.
The main issues have been well explained by others, but are confounded with underlying or associated
Over-reverence for P-values, at most one kind of evidence in statistics.
Reluctance to see that statistical reports are inevitably based on a combination of choices, some firmly evidence-based, others based on a mix of previous analyses, intuition, guesswork, judgment, theory, so forth.
Suppose I and my cautious friend Test Everything both chose a log transformation for a response, but I jump to that conclusion based on a mix of physical reasoning and previous experience with data, while Test Everything chooses log scale based on Box-Cox testing and estimation of a parameter.
Now we both use the same multiple regression. Do our P-values have different interpretations? On one interpretation, Test Everything's P-values are conditional on her previous inferences. I used inferences too, but mostly they were informal, based on a long series of previous graphs, calculations, etc. in previous projects. How is that to be reported?
Naturally, the regression results are exactly the same for Test Everything and myself.
The same mix of sensible advice and dubious philosophy applies to choice of predictors and functional form. Economists, for example, are widely taught to respect previous theoretical discussions and to be wary of data snooping, with good reason in each case. But in the weakest instances the theory concerned is just a tentative suggestion made previously in the literature, very likely after some empirical analysis. But literature references sanctify, while learning from the data in hand is suspect, for many authors.