I see two somewhat unrelated questions in this question.
- Is it possible to draw reliable inference about individual coefficients in predictive models, especially if we have a large number of predictors and use some form of variable selection and/or regularization?
- Can the coefficients in a predictive model be interpreted causally?
My short answer to the first question is yes, that's possible, but it's not straightforward to do correctly, and it's the subject of intense current research.
To the second question I find that the safe answer is no, the coefficients in a predictive model don't generally have a causal interpretation. This point should be made very clear to collaborators/clients, who may not have a strong training in causal models.
Emmanuel Candes gave the 2017 Wald Lectures at the Joint Statistical Meetings entitled What's happening in Selective Inference?, which offers a great starting point for learning what the challenges are, and what the status is.
A main challenge, especially when the number of predictors is large, is how to compute and report uncertainty correctly, when the model/predictors have been selected by the data.
Candes explains at length his contributions (mostly with Rina Barber) on the knockoff filter, which is a very nice idea for controlling the false discovery rate of selected predictors.
Another question is how to reliably compute confidence intervals for the coefficients. Candes touches upon this in his talk, but see the paper
Exact post-selection inference, with application to the lasso by Lee at al. for more details, and see also the paper Valid post-selection inference by Berk et al.
The R package Selective Inference implements these ideas. Another relevant R package to consider is hdi, see also the paper High-Dimensional Inference: Confidence Intervals, p-Values and R-Software hdi by Dezeure et al.
Note that there is a non-trivial discussion in selective inference about what the target parameter actually is! Is it the (theoretical) coefficient in the model with the selected predictors, or is it the coefficient in the model with all predictors included? Read the paper by Berk et al. for some discussion on this difference.
I would typically investigate uncertainty of reported coefficients and selected predictors via simulations/bootstrapping (remembering to include the full variable selection procedure within the bootstrapping), but it may actually require some work to make sure that e.g. bootstrapped confidence intervals are appropriate, see Bootstrapping Lasso Estimators by Chatterjee and Lahiri.
I should say that the challenges discussed above are fundamentally frequentistic of nature. See e.g. Gelman's post Bayesian inference completely solves the multiple comparisons problem for a more Bayesian perspective.
Regression models have been used in econometrics and epidemiology, to mention some areas, to estimate causal effects from observational data. This, I find, has historically not always been done with a crystal clear discussion of what actually constitutes a causal effect. Causality has been thought justified by appealing to "no unmeasured confounders" and other similar properties of the setup, in an attempt to argue that the regressors included are precisely those that are needed to justify a causal interpretation of estimated coefficients. But often without a clear conceptual or mathematical framework for defining causality and causal effects.
The history of how Simpson's paradox has been treated in the statistical literature illustrates the problems as described by Pearl in his paper Understanding Simpson’s Paradox.
What is crystal clear to me is that causality is a concept beyond a probabilistic model, and this can be formalized using frameworks such as counterfactuals, structural equation models or graphical models (DAGs). These are not unrelated frameworks, but offer slightly different concepts and languages to introduce the fundamental parameters of interest: the causal effects.
In some situations it might be possible to interpret coefficients from a predictive (regression) model as causal effects, but I would say it's unlikely to be the case if the model is optimized for purely predictive performance from a vast number of potential predictors using observational data.
The forthcoming Causal Inference Book by Hernan and Robins is a great place to learn about the causal models. Part II of the book deals specifically with the use of models for causal inference.
Causal effects can sometimes be estimated using predictive models, but it may require some ingenuity. Inverse probability weighting relies on a predictive model of probability weights, as Hernan and Robins describe. The recent paper Causal inference by using invariant prediction: identification and confidence intervals by
Peters, Bühlmann and Meinshausen relies on the causal model being invariant under different (unspecified) interventions, whereas non-causal associations are not.
In any case, I would strongly advice against careless interpretations of (regression) coefficients as causal effects. If causal effects are of interest, this should be taken seriously, and appropriate methods should be employed to estimate the effects of interest.