Effects of model selection and misspecification testing on inference: Probabilistic Reduction approach (Aris Spanos)

Question

This question is about pre-test bias, inference after model selection and data snooping within the Probabilistic Reduction (PR) methodology by Aris Spanos (which is related to the Error Statistics philosophy by Deborah Mayo; see e.g. her blog).

---

I have been reading papers by Aris Spanos (2000, 2010, 2016, 2017, 1989) on the PR methodology in econometrics. The methodology may be briefly summarized as follows. There are two points of departure, Theory and the True data generating process (DGP), and the two meet to produce an Econometric model:

• Theory $\rightarrow$ Theoretical model $\rightarrow$ Estimable model $\rightarrow$ Statistical analysis $\rightarrow$ Econometric model.
• True DGP $\rightarrow$ Observed data $\rightarrow$ Statistical model $\rightarrow$ Statistical analysis $\rightarrow$ Econometric model.

The sequences above are mostly self-explanatory, except for the Statistical analysis part. Statistical analysis amounts to the sequence {Specification, Estimation, Respecification} iterated until all of the Statistical model's assumptions are satisfied so that the model arrived to is "statistically adequate". The final step in Statistical analysis is Identification by which the Theoretical model is related to the estimated Statistical model, and the Econometric model is born.

Once the Econometric model is in place, one may engage in testing some theoretical claims and doing inference.

Note that the Estimable model must be embedded in the "statistically adequate" Statistical model (i.e. the latter must nest the former) to facilitate testing and inference. Spanos stresses that inference is only valid if all of the assumptions of the Statistical model are met, i.e. we have a "statistically adequate" model; otherwise inference is unreliable.*

Question: But what about the infamous pre-test bias and problems with post-selection inference and data snooping?

In the PR methodology, the Statistical model aims at describing the DGP. Notably, the Statistical model is formulated independently of the Theoretical model and based solely on the observed the data. It is built so as to reflect the chance regularities found in the data. Therefore, it is highly unlikely that the first Statistical model tried on the data will satisfy all of the model's underlying assumptions; hence, multiple steps of respecification and estimation based on the observed data will be carried out. As such, the "statistically adequate" Statistical model will be built by exploiting quite some information in the data. And then this model will be used for inference. My knee-jerk reaction: pre-test bias, post-selection inference.

Spanos (2000)

Spanos addresses my concerns in 2000 (which is almost entirely dedicated to the topic) suggesting to have a single general model and only consider submodels of it, which allows keeping track of actual significance level in sequential and multiple testing when selecting out some regressors (e.g. end of Section 4.5). This is in contrast to appending the general model by new regressors that Spanos criticizes.

He also argues in Section 6.2 that diagnostic testing does not cause pre-test bias because after a failed diagnostic test the modeller is not supposed to automatically select the implicit or explicit alternative of the test as the new Statistical model but rather has to examine this model using misspecification testing first. This leaves me wondering whether this does not cause at least a mild form of pre-test bias, because inference will be conditional on having passed the misspecification tests.

Section 6.3 of 2000 endorses data snooping as a valuable tool for building "statistically adequate" models without discussing its impact on inference (presumably since the impact of specification testing and model respecification has been discussed before).

Spanos (2010)

Spanos also addresses these concerns in 2010, Sections 6.1 and 6.3. In Section 6.1, he says that

[F]or many statistical models, including the simple Normal and the Normal/linear regression models, [misspecification] testing can be based solely on a maximal ancillary statistic <...> which is independent of a complete sufficient statistic <...> used solely for primary inferences.

In my understanding this means that essentially the questions asked to the data in misspecification testing are so different from those asked when conducting inference that answers to the former ones do not affect answers to the latters ones, thus no double-use of data and no pre-test bias. Is it that simple?

He concludes Section 6.3 by saying

The pre-test bias charge is ill-conceived because it misrepresents model validation as a choice between two models come what may

and the preceding discussion in Section 6 tries to show that somehow model specification testing and model respecification following the failed tests is not the same as model selection and does not induce the pre-test bias. I have a hard time following the argument...

---

Perhaps the answer to my question lies in the distinction between inference on some theoretical claims that were specified before seeing the data vs. inference on claims about statistical properties of the data that where specified based on the observed data? I.e. since the Statistical model is constructed without any regards to the Theory, it does not (and cannot) abuse model selection so as to suit the Theory better. Thus, inference on Theory is not affected in a systematic way (e.g. it is neither biased towards rejecting nor towards accepting some theoretical claims). Meanwhile, building a model based on the data and then testing the very model to do inference on statistical properties of the data specified after having seen the data is of course wrong, because the hypotheses being tested are inspired by the observed chance regularities in the data and hence this is the classical pernicious double-use of the data.

Question reiterated: Are the infamous pre-test bias and problems with post-selection inference and data snooping not really problems in the PR methodology, and why? (I do not get the argument...)

* _{Interestingly, Spanos remarks that the famous phrase "All models are wrong but some are useful" applies to wrongness on the substantive side (we cannot explain complex real-world phenomena with our simple models, but we can still benefit from what we learn from these models) but not on the statistical side (we must make sure the statistical models meet their assumptions; otherwise inference from these models will be invalid).}

References:

• Spanos, A. (1989). On rereading Haavelmo: A retrospective view of econometric modeling. Econometric Theory, 5(3), 405-429.
• Spanos, A. (2000). Revisiting Data Mining: ‘hunting’ with or without a license. Journal of Economic Methodology, 7(2), 231-264.
• Spanos, A. (2010). Akaike-type criteria and the reliability of inference: Model selection versus statistical model specification. Journal of Econometrics, 158(2), 204-220.
• Spanos, A. (2016). Transforming structural econometrics: substantive vs. statistical premises of inference. Review of Political Economy, 28(3), 426-437.
• Spanos, A. (2017). Mis-specification testing in restrospect. Journal of Economic Surveys, forthcoming.

Answer 1

There are quite some similarities between Aris Spanos' framework and David Hendry's econometric methodology; no wonder as Spanos was a student of Hendry. Here is my brief summary of what Hendry had to say when confronted by Edward Leamer and Dale Poirier on the problem of pretesting and post-selection inference (Hendry et al., 1990).

Summary

Hendry does not see a problem with pretesting and post-selection inference in his methodology. He views it as the model discovery stage which is "outside the confines of classical hypothesis testing theory" (p. 213). The conventional theory of estimation and inference is suited for a given model with unknown parameters, not for an unknown model (p. 201). There is no theory for design of models (p. 224). Hendry intentionally and willingly conditions inference on the model (p. 222) (!!!).

It is not important how one arrives at a model as this has nothing to say about the model's validity. The route to the final model does affects the model's compellingness, however. Extensive specification search makes the model less compelling, but not less (or more) valid.

Quotes

Here are some quotes from the paper. P. 207-210:

Poirier: David, you stated something before which I think suggests behavior very much in tune with the Likelihood Principle. As Pagan [38, p. 7] also points out, your attitude seems to be how the final model is derived is largely irrelevant in concluding what evidence there is in the data about the unknown parameters. That is something that a likelihood proponent would adhere to. The path getting there, however, is something that becomes very important for the frequentist...
Hendry: The path is obviously irrelevant for the validity of the model (see, for example, my comments above about the principle of buoyancy).
Poirier: Well, for purposes of drawing inferences about the parameters...
Hendry: No, I haven't said that. We must be clear about what the route independence proposition applies to. The validity of the model as an intrinsic description of the world is independent of the discovery path. The inferences you draw from the model might still be route dependent. This is the issue that Ed called "compellingness." If I thought of the model in my bath, you might not think that's very compelling. You might not accept any inferences from that model. But whether or not that model characterizes reality to the degree that is claimed is independent of how the model was found. That is the statement I'm making.
Poirier: There is a mixing here of when to condition on the data and when not. I think you are saying that it is okay to condition on it for evaluating the model, but not for drawing inferences concerning the parameters.
<...>
Leamer: My understanding is that you refuse to submit to the discipline of either one of those approaches. You're clearly not asking what is the prior distribution that underlies the procedure that you are recommending. Nor do I see you laying out the sampling properties of these very complex processes that you are working with. This makes it very difficult for me to know whether what you're recommending is appropriate or not, because I don't see that there is a framework by which we can evaluate it.

More on p. 213-214:

Hendry: In the context of evaluation the role of testing is clear cut. Someone produces a model. I make a prediction on the basis of their claims about the model, and construct a test that would be accepted as valid, at an agreed significance level. Then I check if the outcome falls within the critical region. That is critical evaluation of the model. In the context of discovery, we are outside the confines of classical hypothesis testing theory. We do not know what the properties of our procedures are. But the intrinsic validity of the model is independent of the route, so validity cannot depend on the order of testing, how many tests were done, etc. The ability to find good models or the credence that others might place on the model may depend on the procedure, but the latter doesn't worry me greatly. If you come up with good models, those models will be robust over time and will serve the functions you claim they serve, and the fact that you thought of them in your bath or did fifty tests or five hundred regressions or discovered them in the very first trial, seems to me irrelevant. But in the context of evaluation or justification it is very important to reveal whether or not the four hundredth test on the model yielded the first rejection.

(Emphasis is mine.)

P. 220-221 (this is quite on the point):

Hendry: My treatment of the pretesting issue per se is that in the context of discovery the tests are not tests, they are selection criteria or indices of adequacy of design. They show if the bridge you are building will withstand a particular gust of wind or a certain volume of traffic, whether the steel in it was properly made, etc. These are ways of self-evaluation, so you can decide for yourself whether you have matched the criteria that are relevant for congruency. So you are always going to look at some index of white noise or innovation, some index of exogeneity, some index of invariance and constancy, some index of theory con-sistency, and some index of encompassing. PCGIVE (see Hendry [19]), for example, provides many of those that I think are necessary, although they are not sufficient. When one has designed the model to characterize the data, I call it congruent.
The pretesting issue would be if one wanted at that stage to make inferences which were not simply that "the model is well designed." That is all that can be claimed when you quote these criteria: "Here is my design criteria and I meet them. This bridge is designed to take a ten-ton truck. Here's a ten-ton truck going over it and it stood up." That's the sense in which the indices of model adequacy are being offered.
Outside of that context, including diagnostic testing in a new data set or against new rival models or using new tests, then you must be careful of the pretesting issue. Not for the parameter standard errors, but for the fact that if under the null of a valid model, you conducted 100 tests at the 5% level, then there's a fair probability you'll get some rejections. If you want to interpret them correctly, the overall test size in the evaluation domain is an important factor to think about. It is fairly easily controlled. You can let it get smaller as the sample size gets larger, and smaller for each individual test as the number of tests gets larger. It is rare that you find a situation in which the model does well in many ways, but badly in a rather obvious dimension, but it could happen.

P. 222-224 (this is quite on the point):

Poirier: One frequentist result on pretest estimators is that in usual situations they're inadmissable. Now, as a good frequentist, why doesn't that bother you?
Hendry: Because at the end of the day I want to condition on the model. Given route independence, if the model congruently characterizes reality, then the statistics I quote with it are the correct basis for forecast variances, etc.
<...>
It is not usually worth spending a lot of time worrying about the particular properties of estimators when you are in the context of discovery, because the revision process takes us outside the formal domain of statistics.
<...>
But I see the model selection problem as being the crucial one, which cannot be formulated as "we already know that $y=X\beta+u$, and just need the best estimate of $\beta$". That latter is a different statistical problem, and it is one to which pretesting is relevant. But it is not directly relevant when we're analyzing data.
Poirier: So, do you think classical statistics has misled people by emphasizing admissability criteria and sampling distributions of procedures? Is it asking the wrong questions?
Hendry: It's asking different questions. It's asking questions concerning if you know $y=X\beta+u$, and you are going to get different samples of data from this process, how should you estimate j? That is a mathematical/statistical question that falls into my second category where we can study the properties of procedures, whether they are Bayes procedures, classical procedures, or likelihood procedures. We can study them, but they cannot solve what is wrong in econometrics. They are necessary tools, but do not answer the practical question of how do you find a model that characterizes the data which is a question in my third category.
<...>
We do not yet have any theory, either Bayesian or sampling for design of models. It's not in your work and I haven't seen it anywhere else.

(Emphasis is mine.)

References:

• Hendry, D. F., Leamer, E. E., & Poirier, D. J. (1990). The ET dialogue: a conversation on econometric methodology. Econometric Theory, 6(2), 171-261.

Answer 2

There is some research on the effect of pre-testing on subsequent inference. This has a long tradition starting from Bancroft's work in 1944. The baseline is that it can hurt, but it doesn't always. No black or white there. We did a survey paper on this with some new results.

M. I. Shamsudheen & C. Hennig: Should we test the model assumptions before running a model-based test? https://arxiv.org/abs/1908.02218

Spanos is not exactly keen on this, see here (I respond in the comments): https://errorstatistics.com/2021/02/25/aris-spanos-modeling-vs-inference-in-frequentist-statistics-guest-post/