Bayesian inference—interpretation of P(X)

Question

Let's assume we have a simple neural network model for which we want to use Bayesian inference.

• $X$ - is the data we have seen so far
• $W$ - is the weights space of our neural network.

In order to better understand Bayesian inference, i would like to understand the intuitive meaning of each component. Some of the component have a clear intuitive meaning, but i am not sure regarding $P(X)$.

The Bayes formula is: $$ P(W|X) = \frac{P(X|W) \times P(W)}{P(X)} $$ The components are:

• $P(W|X)$ - the posterior - This represents the weights distribution of our NN model.
• $P(W)$ - the prior: This represent the prior probability for each weight. We can assume normal distribution for simplicity
• $P(X|W)$: the likelihood. This represents our model prediction i.e. what is the probability of the data give the current weights of the model
• $P(X)$: the data probability - ?

My question is what is the intuitive meaning of $P(X)$? What does it represent?

Let's take a simple example:

We have dataset of pairs $(x,y)$ which is drawn from $y = \sin(x) + \mathrm{noise}$. We choose normal distribution as our prior $P(W)$. We have a neural network that learns this function. In this case, what is the meaning of each component (especially $P(X)$)?

Answer 1

In a more general format, $P(X)=\int_{\theta\in\Theta}f(X|\theta)P(\theta)\mathrm{d}\theta$.

Observe that $\int_\theta{}P(\theta)=1,$ so it is a proper probability distribution. That does not have to be true for $f(X|\theta)$ over the set of $\theta\in\Theta$.

That is why most people ignore the denominator as being anything except an annoying scalar. It makes the product in the numerator sum to unity. That view is that it is just a utilitarian necessity.

However, it does have an intuitive meaning. It is the subjective expectation of the likelihood of the observed data. It is the sum or generalized sum of all possible explanations for the observed data under your subjective prior beliefs.

It doesn't have much implied value because of two rules in Bayesian thinking.

First, your prior has to be fixed. Two people with differing priors will get two different values, but you don't get to go experimenting with priors once your experiment has started. Your beliefs are your beliefs. A good prior reflects that. A bad prior does not.

Second, in Bayesian thinking, the sample space is ignored, with the exception of the observed sample itself. Other than for things like model checking to prevent things like having negative calories in a model or other such nonsense, the sample space gets short shrift in day-to-day practice.

Nonetheless, consider the following model with a binomial likelihood. The parameter space is discrete, $\theta\in\{2/5,3/5,4/5\}$ and a sample space of $k=0\dots{5}$.

If your prior mass function was $\Pr(\theta=2/5)=1/3$, $\Pr(\theta=3/5)=1/2$, and $\Pr(\theta=4/5)=1/6$ then your denominator would be in the first chart over each possible realization.

If your friend's prior mass function was $\Pr(\theta=2/5)=8/10$, $\Pr(\theta=3/5)=1/10$, and $\Pr(\theta=4/5)=1/10$ then your denominator would be in the second chart.

Nonetheless, it sort of begs a "so what." With a fixed prior and a known observation, it is the weight of the evidence, but we don't get to live in parallel universes or with parallel bodies in one universe.

It is the expected value of the likelihood of the observed data under your subjective beliefs. It doesn't have any inferential value. It is a scalar.

It is important. It has intuitive meaning, but it doesn't have much value other than the utilitarian purpose to cause the posterior to sum to unity.

Its most practical value is when it doesn't work. When the integral diverges, as it easily can with improper priors, then you should either be honest about your priors or use a Frequentist method.

A Bayesian model minimizes the average loss created by being unlucky and getting a bad sample. However, the prior acts much like a context for solving a problem in. Without an understanding of the prior probability of getting the actual observation, how could you possibly minimize a loss?

Consider a person raised on a desert island by parents that didn't tell the child about horses, racing, money or gambling. The parents die and the child is rescued as an adult. They look for a job and see one for a bookie.

How can a person, using only data, minimize a loss without proper data on what a horse is, what a race is, or the value of money?

Frequentist methods, in most general cases, minimize the maximum amount of risk you would be exposed to. Without a context to solve a problem in, which is really what a prior distribution is, that is a wonderful alternative.

The denominator can tell you when it is time to stop using a Bayesian method.