Modern applications of Bayesian Model Selection

Question

I'm trying to understand the merits of this field so I'll try to break down my question.

1. Research: Is Bayesian model selection considered a popular topic of research these days?
2. Variable selection: My understanding is that its main area of application is (generalized) linear models. But since we have regularization, PCA, VIF, and so many other techniques for variable and model selection in linear models, how is Bayesian model selection supposed to be any better?
3. Machine learning: On the other hand, we have advanced ensemble learning models as well as Deep Learning. Can Bayesian model selection be applied to a highly non-linear model such as a neural net (maybe in the context of generative models) or to a non-analytical model such as a decision tree? If not, can it be somehow matched up against them?
4. Causal inference: There's maybe something that I'm completely missing, for example the use of Bayesian model selection to obtain more interpretable models better adapted to causal inference rather than prediction (I heard that about BIC somewhere).

Answer 1

Bayesian Model Selection consists in using Bayesian statistics to compare competing hypotheses or models based on previously acquired data. The core idea is to be able to compute the Bayes Factor. Let's assume that we have obtained data $\mathcal{D}$ and have two competing hypotheses/models to explain the data: $\mathcal{M}_1$ and $\mathcal{M}_2$. In this setting, the Bayes Factor is

$$ \frac{p(\mathcal{D}|\mathcal{M}_1)}{p(\mathcal{D}|\mathcal{M}_2)} $$

which is the ratio of the respective evidences in favor of $\mathcal{M}_1$ and $\mathcal{M}_2$. Depending on the value of this ratio, we can determine if the evidence for $\mathcal{M}_1$ compared to $\mathcal{M}_2$ is negative, not significant, or decisive, and thus conclude as to which model is the best one to fit the data $\mathcal{D}$. A nice reference on how it differs from frequentist hypothesis testing is the following one :

Keysers, C., Gazzola, V., & Wagenmakers, E. J. (2020). Using Bayes factor hypothesis testing in neuroscience to establish evidence of absence. Nature neuroscience, 23(7), 788-799.

However, in practice the evidence $p(\mathcal{D}|\mathcal{M})$ is often intractable for complex models, as it requires to integrate marginals for each parameter:

$$ p(\mathcal{D}|\mathcal{M}) = \int_{\theta} p(\mathcal{D}|\theta,\mathcal{M}) p(\theta|\mathcal{M}) $$

A practical and time-efficient approximation of the model evidence is given by the Bayesian Information Criterion:

$$ BIC \approx -2 \log p(\mathcal{D}|\mathcal{M}) $$

You can have a look at this paper we wrote to see how the BIC is derived : https://www.frontiersin.org/articles/10.3389/fncom.2020.558477/full

1. Research. The very notion of popular topic of research is complicated, since it is always difficult to have a precise and exhaustive view of what the scientific community is working on, and since statistical tools should be used based on their validity and usefulness, and not on the possible hype that surrounds them. This being said, the theoretical aspects of Bayesian Model selection indeed saw interesting developments these last years, see for instance:

• https://en.wikipedia.org/wiki/Deviance_information_criterion
• https://en.wikipedia.org/wiki/Watanabe%E2%80%93Akaike_information_criterion

Besides, Bayesian Model selection is widely used for performing model comparison and hypothesis testing, and has been applied to study (most of my examples will come from neuroscience, which is my field of predilection) how populations of neurons encode uncertainty, the volume of synaptic vesicles, the distribution of synapses' sizes, group studies, astronomy and exoplanets. Other fields related to Bayesian Model selection are Bayesian Model Averaging (in which your prediction is the weighted sum of the predictions of your models, weighted by the respective evidences for each model), or Optimal Experiment Design (i.e. how to design your experiment so as to increase the ability to discriminate your competing models).

2. Variable selection. All these tools have actually different objectives:

• The goal of model selection is to find which model (among the family of possible models we have at hand) provides the best fit to the data. Models are compared based on their ability to explain and fit the data, while being penalized for their complexity (usually, for their number of free parameters) to avoid overfitting.
• Regularization only corresponds to the latter objective : its goal is to prune the model, and to penalize the likelihood to prevent overfitting. You can have a look at this answer : Does it make sense to regularize the loss function for binary/multi-class classification?
• PCA is not really about model selection. It is a more practical process in which you perform a change of basis, and may decide to only consider the elements in the new basis that significantly contribute to the data. I summarized here its main goals : What is the purpose of dimensionality reduction?

So it is not really possible to argue that Bayesian Model selection is "better", since all these techniques have different objectives and usefulness. However, they can be seen as different methods to implement Occam's Razor (i.e. the simpler, the better).

3. Machine learning. A quick search on Google Scholar shows that Bayesian Model selection has already been applied to machine learning:

• https://jmlr.org/papers/v20/19-236.html
• https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2007WR006155
• https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiP_Oe75t_sAhWWDGMBHZbRCucQFjADegQIAxAC&url=https%3A%2F%2Fprojecteuclid.org%2Fdownload%2Fpdfview_1%2Feuclid.ss%2F1089808278&usg=AOvVaw12Azf3M6W2gMbt4TXRhSPh

However, compared to "classical" (i.e. normative, or biophysical) models, neural networks have some specificities that might explain why the use of the BIC to analyze them is not so widespread:

• They are usually trained on very large data sets;
• They are usually purely phenomenological : they are designed to fit and predict data, but not to explain them. On how prediction and understanding in science are related, I highly recommend this brilliant article : https://aeon.co/essays/will-brains-or-algorithms-rule-the-kingdom-of-science
• However, regularization can indeed be involved while training a neural network, see : Does it make sense to regularize the loss function for binary/multi-class classification? You will note that performing $L_0$ regularization is similar to penalizing the number of parameters in your model, as does the BIC.

4. Causal inference. Providing explanation rather than prediction is not determined by model selection, it is determined by how you define your models.

Models differ in their nature, and can be classified as phenomenological, normative, or biophysical. On the one hand, purely phenomenological models are useful for relating the output of a system to its input, and can provide a computationally efficient way to make prediction. However, as they are solely based on the empirical relation between the input and the output of the system, and not on its inner biological principles, they lack interpretability. On the other hand, normative and biophysical models can be computationally challenging to fit on data, but are more realistic. In a normative approach, the output of a system is computed from an objective function which models its high-level functions and principles. As opposed to this top-down approach, biophysical models aim at precisely describing the low-level biological components of the system. An interesting property of these biophysical models is that their parameters correspond to real physical quantities (reference).

Depending on whether you want to use phenomenological, normative, or physical models, you can always use Bayesian Model selection to discriminate them.