In what sense does minimizing the bias in estimates give the most
accurate representation of the underlying theory?
In the usual sense intended in econometrics. In typical economic models some parameters are involved, the original role of econometrics was to quantify them. So in economics/econometrics models the parameters are the core of the theory. Them carried out the causal meaning that economists looking for (or it should be so).
Exactly for this reason econometrics manuals are mostly focused on concept like endogeneity and, then, bias. Even for this reason, at least until a few year ago, estimator like LASSO and RIDGE (that induce bias) was not considered at all in several econometrics books.
In prediction the theory is not the core, then nor causal questions are. Only the reliability of predicted values is the core and overfitting is the main related problem. Therefore the focus is not on the parameters, then not on bias/endogeneity.
Unfortunately in past years econometricians made some confusion about the key role of causality. This fact seems me related to the problem of conflation between causation and prediction.
In the article To explain or to predict? is underscored that the wrong model (biased) can remain useful for prediction. In some cases it can be also better than the right one (correctly specified). This fact was remarked in the reply of the Prof herself. In my view the main contribution of the article is that it put light on the fact that, if we understand the difference and avoid the conflation between causation and prediction, we can also understand that some concept and tools are useful for one scope but not much for the other.
In several generalistic econometric manuals that address also forecasting problems, the role of overfitting, in terms of in vs out of sample performance, is not discussed at all or, at best, not adequately. Overfitting do not have the same respectability of endogeneity in these texts, while it should be if we understand that overfitting deal with prediction and endogeneity deal with causation. I checked al lot for this distinction and it is far from clear in several econometrics books. Some obscurities about causality are related. Only recently something start to go better … but not enough yet.
I wrote something about these problem in this site. For example:
Endogeneity in forecasting
Regression and causality in econometrics
Are inconsistent estimators ever preferable?
endogenous regressor and correlation
I hope that them can help someone
If the theory has many parameters, and we have scant data to estimate
them, the estimation error will be dominated by variance. Why would it
be inappropriate to use a biased estimation procedure like ridge
regression (resulting in biased estimates of lower variance) in this
Interesting point. Parsimony is good for both, prediction and causality. In basic linear model can seem also more important for prediction then causality. The reply of Prof (see appendix in the article) seems to go toward this direction; underspecification good for prediction. This discussion is strongly related (Paradox in model selection (AIC, BIC, to explain or to predict?)). However I suggest to consider the example in the article as very relevant ma, at the same time, as didactic example; his technical implications should not be exaggerated … econometrics/statistics modeling is a wide and complex area.
In my opinion the opportunity to have a good theory that imply model with many parameters is debatable; parsimony is good in causal models also. In some cases more for causation then prediction. As relevant example, the so called big data give us possibility that seems me more relevant for prediction than causality. Infact big data, many predictors, are good if we can skip any theoretical scrutiny about them and only correlations matters. This position is good for pure prediction but is hardly justifiable in causal models. The tools that you claim (RIDGE, LASSO, ecc) are good for big data, then for prediction more than causation.
warning 1: here the differences between causation and prediction are extremized, several overlapping can be invoked. The same article warning about this fact.
Warning 2: many parameters case open the door to the non-parametric model. This is not the standard in economic theory, or at least not yet. Maybe in this area the overlap between prediction and causation are more difficult disentangle. I have to study more about that.