# Ockham's Razor in Bayesian Modelling

This question might be a little philosophical / generate discussion. I hope, there may still be some useful answers.

I am currently thinking about how Ockham's Razor relates to Bayesian statistical testing. I found, that often Ockham's razor is implemented by priors on models of different complexity (this paper attributes the idea to Jeffreys).

However, in Bayesian statistics a common method for model selection is to have a discrete random variable "switch" between each of the models, and then use the resulting posterior of that discrete variable to get the probabilities. This is equivalent to comparing the (often intractable) integrals over the parameter spaces of the models (weighted by the priors on the parameters).

So when doing Bayesian model selection, models are automatically penalized for their complexity. This is simply the case, because for more complex models we are integrating over a much larger and more complex parameter space, and most elements of this parameter space don't fit the observations.

This is the same Ideas, that can be found in the above paper. In this case, Ockham's Razor is NOT implemented by the priors, but rather results automatically and manifests itself in the posterior. But is this enough to account for different model complexity?

For example, if I have two sets of Bernoulli trials with repeated samples, e.g. $$Y_{1,i}=(1,1,0,1,1,...)$$ and $$Y_{2,i}=(0,0,1,0,0,...)$$ it would be possible to test if $$Y_1$$ and $$Y_2$$ are generated by the same probability (H1) or by different probability (H2). Given enough samples from $$Y_1$$ and $$Y_2$$ it should be possible to construct a situation where $$P(H1|D)=P(H2|D)$$ (assuming equal prior probabilities of H1 and H2). However, given that H1 has only one parameter, whereas H2 has two, should this really be taken as evidence, that both are equally likely?

• Bayes factors are consistent so this is an argument about taking the complexity rightly into account... – Xi'an Dec 14 '18 at 19:30

So when doing Bayesian model selection, models are automatically penalized for their complexity. This is simply the case, because for more complex models we are integrating over a much larger and more complex parameter space, and most elements of this parameter space don't fit the observations.

... Ockham's Razor is NOT implemented by the priors, but rather results automatically and manifests itself in the posterior. But is this enough to account for different model complexity?

You are getting close, but this is not quite right. At least two of the examples in Jefferys and Berger (1991) (the linked paper) involve cases where both the simple and complex models consist of a single specification of the sampling distribution and have no free parameters (i.e., they are both simple hypotheses). In both the coin-toss example and the plagiarism example, there is a simple model with a point-mass distribution for the data, and a more complex model that gives a more diffuse sampling distribution for the data. In these cases the complex model does not have a larger parameter space than the simpler model. The authors show that even in this situation, so long as the observed data does not falsify the simple model, even when both models have no free parameters, the complex model emerges with lower posterior probability due to the greater diffusion of the sampling distribution.

In general, if two model classes are considered equally likely a priori, then lower posterior probability for one of those model classes (e.g., the complex one) can only emerge because the likelihood tends to be lower under that model class (in a sense made more rigorous below). This is not a matter of the size of the model parameter space per se, but really a matter of the average size of the likelihood over the parameter space. It is somewhat artificial to separate the size of the parameter space under the model class from the likelihood function over that space, since the latter only exists over the former space. Nevertheless, if you have a "simple" model and a "complex" model, and an observation that is reasonably consistent with the former, that will tend to occur in cases where the likelihood is high for the former and low for the latter.

Analysis of Ockham's Razor in Bayesian analysis: Comparison of two classes of models in Bayesian analysis can be done by considering them as specifications of parameter spaces in an overarching class of models, and assigning a prior that is a mixture distribution of conditional-priors over the two classes. To understand this analysis properly it is useful to distinguish between the prior probabilities assigned to the two model classes, versus the prior distributions over the individual models within those classes. Both of these things are aspects of the overall prior. To formalise this distinction, suppose you have an overarching model with parameter space $$\Theta = \Theta_0 \cup \Theta_A$$ and your model classes can be specified by the parameter spaces:

\begin{aligned} \text{Simple model }(H_0) & & \theta &\in \Theta_0, \\[6pt] \text{Complex model }(H_A) & & \theta &\in \Theta_A. \\[6pt] \end{aligned}

(Without loss of generality we can take these spaces to be disjoint. If the complex model encompasses the simpler model, or overlaps with it, we consider the points of intersection to be part of the simpler model only.) Any prior over the whole parameter space can be decomposed via the law-of-total-probability into:

$$\pi(\theta) = \delta \pi_0(\theta) + (1-\delta) \pi_A(\theta),$$

where $$\delta = \mathbb{P}(H_0)$$ is the prior probability of the simple model, and $$\pi_0(\theta) = p(\theta|H_0)$$ and $$\pi_A(\theta) = p(\theta|H_A)$$ are the respective prior distributions for the parameter under the simple and complex models. This formulates the overall prior as a mixture of conditional-priors over the two model classes. Under this specification the posterior probability that the data come from the simple model is:

\begin{aligned} \mathbb{P}(H_0|\mathbf{x}) &= \frac{\int_{\Theta_0} L_\mathbf{x}(\theta) \pi(\theta) d\theta}{\int_{\Theta} L_\mathbf{x}(\theta) \pi(\theta) d\theta} \\[6pt] &= \frac{\delta \int L_\mathbf{x}(\theta) \pi_0(\theta) d\theta}{\delta \int L_\mathbf{x}(\theta) \pi_0(\theta) d\theta + (1-\delta) \int L_\mathbf{x}(\theta) \pi_A(\theta) d\theta} \\[6pt] &= \frac{\delta \mathbb{E}(L_\mathbf{x}(\theta) |H_0)}{\delta \mathbb{E}(L_\mathbf{x}(\theta) |H_0) + (1-\delta) \mathbb{E}(L_\mathbf{x}(\theta) |H_A)}. \\[6pt] \end{aligned}

If we set $$\delta = \tfrac{1}{2}$$ so as not to favour either model class a priori, we have:

\begin{aligned} \mathbb{P}(H_0|\mathbf{x}) &= \frac{\mathbb{E}(L_\mathbf{x}(\theta) |H_0)}{\mathbb{E}(L_\mathbf{x}(\theta) |H_0) + \mathbb{E}(L_\mathbf{x}(\theta) |H_A)}. \\[6pt] \end{aligned}

(The corresponding posterior probability for the complex model is one minus this amount, which replaces the numerator with the second term in the denominator.) We can easily see from this result that the posterior probability of the two model classes is determined by the relative sizes of the expected likelihood value under those classes. The size of the parameter space under a model class only affects this insofar as it affects the expected likelihood under the model class. Thus, if we expand the space of a model class, the effect on the posterior depends on how we extend the likelihood function (via the sampling density) over this larger space.

Mathematically, we can see that the posterior probability of the model classes in this case depends only on the expected likelihood under the classes. This is just a reiteration of the fact that model comparisons in Bayesian analysis are fully determined by the prior probabilities of the models, and Bayes factor (which is just the ratio of the expected likelihoods). In practice, it is commonly (but not always) the case that more complex models use a larger parameter space, and if we get data that is reasonably consistent with the simple model, the likelihood function for the complex model is small over most of this larger space. This creates a natural "penalisation" of complex models in cases where the data is reasonably consistent with a simpler model.

$$^\dagger$$ We will proceed without loss of generality here, but it is worth noting that often the simpler model will be a subclass of the complex model, which fixes some of the parameters to zero, thereby yielding an effective parameter-space of lower dimension.