Minimizing bias in explanatory modeling, why? (Galit Shmueli's "To Explain or to Predict")

Question

This question references Galit Shmueli's paper "To Explain or to Predict".

Specifically, in section 1.5, "Explaining and Prediction are Different", Professor Shmueli writes:

In explanatory modeling the focus is on minimizing bias to obtain the most accurate representation of the underlying theory.

This has puzzled me each time I've read the paper. In what sense does minimizing the bias in estimates give the most accurate representation of the underlying theory?

I also watched professor Shmueli's talk here, delivered at JMP Discovery Summit 2017, and she states:

...things that are like shrinkage models, ensembles, you will never see those. Because those models, by design, introduce bias in order to reduce the overall bias/variance. That's why they won't be there, it doesn't make any theoretical sense to do that. Why would you make your model biased on purpose?

This doesn't really shed light on my question, simply restating the claim that I don't understand.

If the theory has many parameters, and we have scant data to estimate them, the estimation error will be dominated by variance. Why would it be inappropriate to use a biased estimation procedure like ridge regression (resulting in biased estimates of lower variance) in this situation?

Answer 1

This is indeed a great question, which requires a tour into the world of the use of statistical models in econometric and social science research (from what I have seen, applied statisticians and data miners who do descriptive or predictive work typically don't deal with bias of this form). The term "bias" that I used in the article is what econometricians and social scientists treat as a serious danger to inferring causality from empirical studies. It refers to the difference between your statistical model and the causal theoretical model that underlies it. A related term is "model specification", a topic taught heavily in econometrics due to the importance of "correctly specifying your regression model" (with respect to the theory) when your goal is causal explanation. See the Wikipedia article on Specification for a brief description. A major misspecification issue is under-specification, called "Omitted Variable Bias" (OVB), where you omit an explanatory variable from the regression that should have been there (according to theory) - this is a variable that correlates with the dependent variable and with at least one of the explanatory variables. See this neat description) that explains what are the implications of this type of bias. From a theory point of view, OVB harms your ability to infer causality from the model.

In the appendix of my paper To Explain or To Predict? there's an example showing how an underspecified ("wrong") model can sometimes have higher predictive power. But now hopefully you can see why that contradicts with the goal of a "good causal explanatory model".

Answer 2

In what sense does minimizing the bias in estimates give the most accurate representation of the underlying theory?

In the usual sense intended in econometrics. In typical economic models some parameters are involved, the original role of econometrics was to quantify them. So in economics/econometrics models the parameters are the core of the theory. Them carried out the causal meaning that economists looking for (or it should be so). Exactly for this reason econometrics manuals are mostly focused on concept like endogeneity and, then, bias. Even for this reason, at least until a few year ago, estimator like LASSO and RIDGE (that induce bias) was not considered at all in several econometrics books.

In prediction the theory is not the core, then nor causal questions are. Only the reliability of predicted values is the core and overfitting is the main related problem. Therefore the focus is not on the parameters, then not on bias/endogeneity.

Unfortunately in past years econometricians made some confusion about the key role of causality. This fact seems me related to the problem of conflation between causation and prediction.

In the article To explain or to predict? is underscored that the wrong model (biased) can remain useful for prediction. In some cases it can be also better than the right one (correctly specified). This fact was remarked in the reply of the Prof herself. In my view the main contribution of the article is that it put light on the fact that, if we understand the difference and avoid the conflation between causation and prediction, we can also understand that some concept and tools are useful for one scope but not much for the other.

In several generalistic econometric manuals that address also forecasting problems, the role of overfitting, in terms of in vs out of sample performance, is not discussed at all or, at best, not adequately. Overfitting do not have the same respectability of endogeneity in these texts, while it should be if we understand that overfitting deal with prediction and endogeneity deal with causation. I checked al lot for this distinction and it is far from clear in several econometrics books. Some obscurities about causality are related. Only recently something start to go better … but not enough yet. I wrote something about these problem in this site. For example:

Endogeneity in forecasting

Regression and causality in econometrics

Are inconsistent estimators ever preferable?

endogenous regressor and correlation

I hope that them can help someone

Moreover

If the theory has many parameters, and we have scant data to estimate them, the estimation error will be dominated by variance. Why would it be inappropriate to use a biased estimation procedure like ridge regression (resulting in biased estimates of lower variance) in this situation?

Interesting point. Parsimony is good for both, prediction and causality. In basic linear model can seem also more important for prediction then causality. The reply of Prof (see appendix in the article) seems to go toward this direction; underspecification good for prediction. This discussion is strongly related (Paradox in model selection (AIC, BIC, to explain or to predict?)). However I suggest to consider the example in the article as very relevant ma, at the same time, as didactic example; his technical implications should not be exaggerated … econometrics/statistics modeling is a wide and complex area.

In my opinion the opportunity to have a good theory that imply model with many parameters is debatable; parsimony is good in causal models also. In some cases more for causation then prediction. As relevant example, the so called big data give us possibility that seems me more relevant for prediction than causality. Infact big data, many predictors, are good if we can skip any theoretical scrutiny about them and only correlations matters. This position is good for pure prediction but is hardly justifiable in causal models. The tools that you claim (RIDGE, LASSO, ecc) are good for big data, then for prediction more than causation.

warning 1: here the differences between causation and prediction are extremized, several overlapping can be invoked. The same article warning about this fact.

Warning 2: many parameters case open the door to the non-parametric model. This is not the standard in economic theory, or at least not yet. Maybe in this area the overlap between prediction and causation are more difficult disentangle. I have to study more about that.