Is it (always) better to build a model prior to viewing the data?

Question

When it comes to data exploration, aside from checking for outliers (human error), correlated covariates, and missing values, is there a downside to viewing relationships between a response variable and candidate covariates before building a statistical model?

I've heard it's better to construct models (based on well researched background info. or expert knowledge) before viewing any relationships, as such models have a tendency to be biased. Is this a reasonable statement to make? I don't remember where I heard this, but I'm a bit confused by it as I've been trained since college to build models based on patterns I observed while exploring the data.

I know it's important to be very deliberate when it comes to choosing covariates (not to fit everything at once and let AIC decide what stays), but wouldn't it be useful to know if there are non-linear patterns in the response instead of, say...starting with a glm and going through the whole processes of checking for patterns in the residuals, etc..?

Answer 1

If you build a model before seeing the data you are more likely to get results (effect sizes, p-values, etc.) that correctly relate the variables in the model to the process you are studying. That is, your model should generalise fairly well to future observations. However, this model may be next to useless in practice if the variables you fitted don't explain much of the variation.

If instead you build the model after seeing the data, i.e. in a data-dependent strategy, you are more likely to discover things you didn't expect but you will also overestimate their effect size.

For example, before seeing the data you decide to fit "sex" as covariate when in fact sex has no effect at all. If you decide to follow up results with smallish p-value, you have, say, ~1/20 (i.e. p < 0.05) chance of obtaining a seemingly interesting result that is in fact a false positive.

But if you allow yourself to fit covariates after seeing the data you are guaranteed to find something with small p-value. If it's not sex, try something like year of birth, age, whatever. If "age" does have an effect you will be able to pick it up with this strategy, but you are likely to overestimate its effect because if the effect is small you would have not noticed it in this dataset.

If you are familiar with R or some other language try this experiment: simulate a treatment vs control experiment with small effect size for "treatment", test using for example a t-test. Repeat many times, like 10000, and keep only results with p < 0.01. The difference between means of these selected experiments will be, on average, larger than the ground truth from your simulation.

In my opinion, building after seeing the data is acceptable, even essential for research to progress, provided you are aware of these drawbacks. Also, ask yourself whether you have a plausible explanation for a variable that you have considered only after seeing the data. When you present results state whether you built the model before or after seeing results. It would be very misleading to report that your hypothesis was that "age" has an effect, then you looked at the data and, yes!, the effect is there. In fact you came up with that potentially interesting hypothesis only after seeing the data.

Answer 2

The problem is that of potential bias in p-values and confidence intervals as well as overfitting or poor predictive performance without making adjustments for the process of choosing the model. Optimal variable selection is an active area of research and of course it is important to accurately capture the potential nonlinear relationships that may have generated your data. A good resource for all of this is Frank Harrell’s Regression Modeling Strategies, some of which is summarized here. You’ll likely find some other good answers and other strategies if you search for variable selection on this site.

Answer 3

There is a division, or gradation, between exploratory and confirmatory statistics. Both have value. Mixing the two is dangerous.

At one end, we have a lot of data mining. Some of this comes very close to "Here is a whole lot of data. Find something!"

At the other end, we have very well formulated models based on strong theories about what is going on. Some of this comes directly from science (usually the "hard" sciences). Other times, you are trying to confirm what someone else found (this doesn't happen enough, for a variety of reasons).

There are ways to avoid making confirmatory statements about exploratory results. If you have sufficient data, you can divide it into "train", "test", and "validate" sets. If you have less data, then you can be sure to state clearly that the results are exploratory -- I would avoid stating p values and CIs here and I would put lots of graphs. You can also do various kinds of sensitivity testing. And, there is the name of this site: Cross validation.

Answer 4

I actually consider building an initial model to be part of the process of validating the data (it doesn't have to be all that good, but does need to be credible), but given how rarely one gets a modelable dataset out of the box, only a fool proceeds directly to modeling without checking the integrity of the data.