5
$\begingroup$

This question is about pre-test bias, inference after model selection and data snooping within the Probabilistic Reduction (PR) methodology by Aris Spanos (which is related to the Error Statistics philosophy by Deborah Mayo; see e.g. her blog).


I have been reading papers by Aris Spanos (2000, 2010, 2016, 2017, 1989) on the PR methodology in econometrics. The methodology may be briefly summarized as follows. There are two points of departure, Theory and the True data generating process (DGP), and the two meet to produce an Econometric model:

  • Theory $\rightarrow$ Theoretical model $\rightarrow$ Estimable model $\rightarrow$ Statistical analysis $\rightarrow$ Econometric model.
  • True DGP $\rightarrow$ Observed data $\rightarrow$ Statistical model $\rightarrow$ Statistical analysis $\rightarrow$ Econometric model.

The sequences above are mostly self-explanatory, except for the Statistical analysis part. Statistical analysis amounts to the sequence {Specification, Estimation, Respecification} iterated until all of the Statistical model's assumptions are satisfied so that the model arrived to is "statistically adequate". The final step in Statistical analysis is Identification by which the Theoretical model is related to the estimated Statistical model, and the Econometric model is born.

Once the Econometric model is in place, one may engage in testing some theoretical claims and doing inference.

Note that the Estimable model must be embedded in the "statistically adequate" Statistical model (i.e. the latter must nest the former) to facilitate testing and inference. Spanos stresses that inference is only valid if all of the assumptions of the Statistical model are met, i.e. we have a "statistically adequate" model; otherwise inference is unreliable.*

Question: But what about the infamous pre-test bias and problems with post-selection inference and data snooping?

In the PR methodology, the Statistical model aims at describing the DGP. Notably, the Statistical model is formulated independently of the Theoretical model and based solely on the observed the data. It is built so as to reflect the chance regularities found in the data. Therefore, it is highly unlikely that the first Statistical model tried on the data will satisfy all of the model's underlying assumptions; hence, multiple steps of respecification and estimation based on the observed data will be carried out. As such, the "statistically adequate" Statistical model will be built by exploiting quite some information in the data. And then this model will be used for inference. My knee-jerk reaction: pre-test bias, post-selection inference.

Spanos (2000)

Spanos addresses my concerns in 2000 (which is almost entirely dedicated to the topic) suggesting to have a single general model and only consider submodels of it, which allows keeping track of actual significance level in sequential and multiple testing when selecting out some regressors (e.g. end of Section 4.5). This is in contrast to appending the general model by new regressors that Spanos criticizes.

He also argues in Section 6.2 that diagnostic testing does not cause pre-test bias because after a failed diagnostic test the modeller is not supposed to automatically select the implicit or explicit alternative of the test as the new Statistical model but rather has to examine this model using misspecification testing first. This leaves me wondering whether this does not cause at least a mild form of pre-test bias, because inference will be conditional on having passed the misspecification tests.

Section 6.3 of 2000 endorses data snooping as a valuable tool for building "statistically adequate" models without discussing its impact on inference (presumably since the impact of specification testing and model respecification has been discussed before).

Spanos (2010)

Spanos also addresses these concerns in 2010, Sections 6.1 and 6.3. In Section 6.1, he says that

[F]or many statistical models, including the simple Normal and the Normal/linear regression models, [misspecification] testing can be based solely on a maximal ancillary statistic <...> which is independent of a complete sufficient statistic <...> used solely for primary inferences.

In my understanding this means that essentially the questions asked to the data in misspecification testing are so different from those asked when conducting inference that answers to the former ones do not affect answers to the latters ones, thus no double-use of data and no pre-test bias. Is it that simple?

He concludes Section 6.3 by saying

The pre-test bias charge is ill-conceived because it misrepresents model validation as a choice between two models come what may

and the preceding discussion in Section 6 tries to show that somehow model specification testing and model respecification following the failed tests is not the same as model selection and does not induce the pre-test bias. I have a hard time following the argument...


Perhaps the answer to my question lies in the distinction between inference on some theoretical claims that were specified before seeing the data vs. inference on claims about statistical properties of the data that where specified based on the observed data? I.e. since the Statistical model is constructed without any regards to the Theory, it does not (and cannot) abuse model selection so as to suit the Theory better. Thus, inference on Theory is not affected in a systematic way (e.g. it is neither biased towards rejecting nor towards accepting some theoretical claims). Meanwhile, building a model based on the data and then testing the very model to do inference on statistical properties of the data specified after having seen the data is of course wrong, because the hypotheses being tested are inspired by the observed chance regularities in the data and hence this is the classical pernicious double-use of the data.

Question reiterated: Are the infamous pre-test bias and problems with post-selection inference and data snooping not really problems in the PR methodology, and why? (I do not get the argument...)

* Interestingly, Spanos remarks that the famous phrase "All models are wrong but some are useful" applies to wrongness on the substantive side (we cannot explain complex real-world phenomena with our simple models, but we can still benefit from what we learn from these models) but not on the statistical side (we must make sure the statistical models meet their assumptions; otherwise inference from these models will be invalid).

References:

$\endgroup$
  • $\begingroup$ I think employing some very basic examples in an answer could be helpful. E.g. show how for a very simple model, model selection, pre-testing and data snooping (one by one) are not invalidating inference within the PR framework. (But I cannot provide starting examples myself as I cannot understand how the PR is supposed to work in light of model selection, pre-testing and data snooping.) $\endgroup$ – Richard Hardy Sep 21 '17 at 14:11
  • $\begingroup$ Maybe I'm just being cynical, but this stuff reads mostly like expounding philosophical principles and then jumping to conclusions (for which there is no empirical or mathematical proof) by analogies and hand-waving. Some of it is not wrong (e.g. it is surely worse to specifically try to p-hack your way to a better p-value than to test normality, but that does not mean normality testing is harmless), but otherwise this has me a bit skeptical. $\endgroup$ – Björn Jan 15 at 18:51
  • $\begingroup$ @Björn, I was more optimistic when I started studying the topic but might converge towards your viewpoint simply due to lack of success in cracking it. For now I think this might be a paradigm clash and cannot simply be explained within one paradigm. I already see some connection with Fisherian vs. other ways of understanding hypothesis testing (after having recently read Christensen (2005) referenced to in this thread). $\endgroup$ – Richard Hardy Jan 18 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.