The role of validation in estimation and hypothesis testing

Question

Validation, with or without statistical/machine learning procedures, is used often, if not universally, in prediction. In estimation or hypothesis testing that does not seem to be the case, yet I don't understand why. If the purpose of estimation and hypothesis testing is generalization, as is normally the case, shouldn't we safeguard against overfitting as we similarly do when the objective is prediction?

I am currently conducting a cross-sectional study in which I try to estimate relative effect sizes between two groups. Is there any reason that I should or shouldn't concern myself with model training and validation?

Answer 1

Good question. In most situations there is an estimand of primary interest, e.g. treatment effect or group difference. We seek an unbiased (usually) estimate that has high precision, so a Bayesian credible interval or frequentist confidence interval will often be of major interest. The width of these intervals, for the most part, take into account the data's information content (sample size, variation, etc.). So in many cases that's enough. Still, validation of the overall model can help quantify the volatility/reliability of the whole model, and especially whether the non-target variables are being overfitted so that you might worry about over-adjustment of covariates.

The same type of optimism bootstrap validation that you might use for a predictive model may just as well be used here.

If the adjustment variables were not pre-specified but determined by data dredging, you really need to use a bootstrap procedure to penalize the target variable's confidence interval. Otherwise it is been shown that all the variables in the model will have confidence intervals that are too narrow.

Answer 2

Johan, your excellent question has two aspects:

1. positive -- Why do people actually do that?
2. normative -- What should one do?

Frank Harrell has answered the normative question; I would like to weigh in on the positive aspect.

I suspect most users of statistics have accustomed themselves to an artificial distinction between prediction and estimation, which is supposed to make the latter somehow less exacting. To paraphrase Samuel Johnson's quip about being condemned to hang, "Depend upon it, sir, when a statistician knows a prediction is due in a fortnight, it concentrates the mind wonderfully." Like being hanged, the 'moment of prediction' seems very real indeed. Prediction has an obvious and unavoidable connection with the real world and with real costs, such as might be described by a custom-built loss function. Even for a prediction task that has not been fully worked-out in decision-theoretic terms, the statistician usually has some intuitive sense that a real cost will be borne (by someone) if the prediction is made with a large error.

By contrast, estimation may seem to be happening on an entirely different plane -- in some etherial realm of ideas and abstractions. Except when the estimation task has been explicitly cast in decision-theoretic terms, that 'mind-focusing' connection with reality is simply not as obvious. Seeing this connection in the case of estimation requires connecting the dots to what people (including you) will believe after they read your analysis, what they will do acting under those beliefs, and what harms will ensue if those beliefs are wrong.

But as demonstrated e.g. in Christian Robert's The Bayesian Choice 2nd ed (2007), it is possible to put prediction and estimation on equal footing by regarding them both as decisions -- whether about what value to choose as 'the prediction' or what value to choose as 'the estimate'. Either way, the decision-theoretic formulation explicitly connects these choices with their consequences.

In brief: the decision-theoretic perspective unifies the tasks of 'prediction' and 'estimation', demonstrating that there is no essential difference and thereby underscoring Frank Harrell's normative answer. The positive answer comes in the form: "because in estimation tasks, people are accustomed to employing standard, off-the-shelf loss functions (e.g., quadratic loss) that allow them to sidestep a realistic assessment and specification of a loss function properly adapted to their particular problem."