Should I use unpenalized logistic regression, lasso or ridge for explanatory modelling?

Question

When using logistic regression for predictive modelling, the choice between 'standard' logistic regression vs ridge vs LASSO versions of logistic regression seems relatively straightforward - just pick the method that gives the best predictive performance on test data (unless there are additional considerations about whether to perform variable selection, which LASSO does automatically but ridge does not, or other considerations as described here Why can't ridge regression provide better interpretability than LASSO?).

But what about when the aim is not predictive performance, but explanatory modelling - i.e. determining which predictors drive variation in the response, and how strong the effect of each predictor is? Is there any reason to think that one of the three methods is better, or should I just go with the one that gives the best predictions (or alternatively use elastic net regression to find the optimal blend of ridge and LASSO in terms of predictive performance)?

This article http://projecteuclid.org/download/pdfview_1/euclid.ss/1294167961 describes how explanatory modelling aims to minimise bias, whereas predictive modelling aims to minimise some combination of bias and variance. This makes me think that the best model in terms of predictive performance may not be the best in terms of explaining which predictors drive variation in the response.

Answer 1

From your post, and from your reference to the statement “explanatory modelling aims to minimize bias,” I suspect you have the impression that the key differentiator between explanatory and predictive approaches is the choice of statistical procedure.

Granted, in conducting explanatory analysis you will probably be best served by avoiding variable-selection algorithms, since they are all blind to the difference between, colloquially, correlation and causation. That probably means choosing “vanilla” regression over LASSO and ridge, not to mention neural networks, random forests, SVM, CART, etc. To paraphrase Davis (below), algorithms have no way to tell whether one variable preceded another; whether it is more objectively or more subjectively measured; or whether it is typically more generative (like socioeconomic status) or less (like one's choice of breakfast cereal).

But sound, effective, replicable explanatory modeling differs from predictive modeling in other ways. The former needs to be informed by several activities designed to uncover as much as possible about the variables that matter to the outcome – and that give one leverage over that outcome – as well as about the functional forms of those relationships. These activities might include --

1. A deep literature review.

2. Consultation with knowledgeable experts and colleagues.

3. Consultation with less-knowledgeable people. Fresh perspectives from non-experts often yield ideas useful to the analyst.

4. (In many cases) more intensive, deliberate, and resourceful data collection than would be required for purely predictive analysis. You will not be satisfied merely to connect Y with some proxy, some indicator, of a cause; you will want to capture the cause itself as closely as you can.

Pinning down cause-and-effect relationships is usually much more difficult than succeeding at prediction. There are some very useful hands-on guides to causal analysis (e.g., by James A. Davis and by Joshua D. Angrist & Jorn-Steffen Pischke), and they should be prized, because sources like these are far less common than those that skip causal considerations in favor of telling how to conduct a given statistical procedure, or how to write the applicable code. Not that there aren’t some tremendous sources in this category, too.

(Secondarily, when you talk about choosing a predictive model that “gives the best predictive performance on test data,” I hope you mean on multiple iterations, i.e., across many, many instances of building a model using training data and then testing it on fresh data.)