What is the intuition behind pre-testing in empirical work? in my class, the professor talks about pre-testing, and he said:
"In empirical applications there is usually uncertainty about model specification, and the same data are typically used both to select a model and to estimate the parameters of the selected model.Usually, however, one reports results as if estimation had not being preceded by model selection, that is, as if the model had been given a priori. This is clearly wrong"
But I didn't get what this pre-testing is for? And what the consequence would be if we don't do pre-testing?
We already talked about using  F and T test to determine whether to add any regressor into the model, and some other test to test the homoskedasticity or serial correlation hypothesis. I didn't find tutorial by university of "pre-testing" online but according to my understanding of Giles and Giles(1993) and MAGNUS(1998), pre-testing is exactly using all kind of test mentioned above to see what kind of model I should specify. Am I understand correctly?
I paste the link of the two papers here:
http://web.uvic.ca/~dgiles/blog/Giles%20and%20Giles.pdf
https://www.janmagnus.nl/papers/JRM050.pdf
Thank you!
 A: PRETEST ESTIMATORS (I.E., MODEL SELECTION)
Sticking to the words of your professor, the problem is that by using the same data to both select and estimate the model, we underestimate the uncertainty of our estimates.
For simplicity, let us focus on the simple linear regression case, where models are written as $Y = XB + \epsilon$. Model selection then consists of choosing which variables must enter the matrix $X$. Do we choose one variable? Do we choose three? Do we add interactions?
Say that the computational burden is not a problem, and that we fit all these different models. Then, we can rely on some criterion (e.g., Cp, AIC, BIC, FIC, CV) to select one model. This is a pretest, or model selection, estimator. The following are the basic steps of this approach:

*

*pre-select a set of models (in our case, different specifications of the matrix $X$);

*estimate each of the models from step 1;

*calculate and compare the information criterion (e.g., Cp or AIC) of each estimated model;

*select the "best" model.

Standard inference methods are here too optimistic, in the sense of underestimating the variability of our estimates. The idea is that they are designed for a given model, i.e., assuming that the model is the true one. In the simple exercise sketched above, we face two sources of uncertainty: the estimation uncertainty, which is the "standard" problem we are taught in introductory courses (sampling variability of our estimators), and the model uncertainty, where the question concerns the choice of the true model (which could be different from the model giving the lowest test error). Standard inference does not take into account the model uncertainty. This explains your professor's statement, and here fits the comment from @Henry about testing for the fairness of a roulette wheel.
Let me add a simple example, so to address your concern about the $F$ statistic. Assume that in step 1 we select two models only, which write as:

*

*$\mathcal{M_1}: \mathbb{E}[Y] = X_1 \beta_1 + X_2 \beta_2$

*$\mathcal{M_2}: \mathbb{E}[Y] = X_1 \beta_1$
In step 2, we estimate both models. Then, we rely on some criterion to select either $\mathcal{M_1}$ or $\mathcal{M_2}$. Then, in step 3 and 4, we could rely on the $F$ statistic, which tests for the null hypothesis $\mathcal{H_0}: \beta_2 = 0$. According to the value of this statistic, we select one of the two considered models, say $\mathcal{M_1}$. So far nothing wrong. The issue is that we cannot simply report OLS point estimates and standard errors of the chosen model, as this would not take into account the uncertainty about the process of the model selection (actuall, point estimates should be fine). Data gave us evidence to choosing $\mathcal{M_1}$ over $\mathcal{M_2}$, but this does not mean that $\mathcal{M_1}$ is the true model. Our estimation of the variability of the results should take into account the whole process, from step 1 to 4, while common practice is to ignore that steps 1 to 3 exist, and report the estimates as if the "best" model from step 4 is the only considered model.
BONUS: MODEL AVERAGING AS POSSIBLE ALTERNATIVE
A possible solution would be model averaging, that is, we do not choose a single model, but we consider all of them, thus explicitly accounting for the model uncertainty. The most common approach is called Bayesian Model Averaging (BMA), where at the end we average models' estimates according to their probability of being the true model.
For more references, I suggest reading Moral-Benito (2015, JES) and Steel (2020, JEL). Both papers review the most common techniques of model averaging, but more importantly for your case they both have nice introductions where the problem is well exposed, together with main advantages and alternatives. Magnus publishes a lot on this topic (I think he developed the WALS approach to model averaging), and more recently he further investigated the topic jointly with De Luca and Peracchi.
