5
$\begingroup$

Let's assume we have a simple neural network model for which we want to use Bayesian inference.

  • $X$ - is the data we have seen so far
  • $W$ - is the weights space of our neural network.

In order to better understand Bayesian inference, i would like to understand the intuitive meaning of each component. Some of the component have a clear intuitive meaning, but i am not sure regarding $P(X)$.

The Bayes formula is: $$ P(W|X) = \frac{P(X|W) \times P(W)}{P(X)} $$ The components are:

  • $P(W|X)$ - the posterior - This represents the weights distribution of our NN model.
  • $P(W)$ - the prior: This represent the prior probability for each weight. We can assume normal distribution for simplicity
  • $P(X|W)$: the likelihood. This represents our model prediction i.e. what is the probability of the data give the current weights of the model
  • $P(X)$: the data probability - ?

My question is what is the intuitive meaning of $P(X)$? What does it represent?

Let's take a simple example:

We have dataset of pairs $(x,y)$ which is drawn from $y = \sin(x) + \mathrm{noise}$. We choose normal distribution as our prior $P(W)$. We have a neural network that learns this function. In this case, what is the meaning of each component (especially $P(X)$)?

$\endgroup$
8
  • 1
    $\begingroup$ This question is pretty much the same as this one, which also asks about the meaning of the denominator term (called the evidence or marginal likelihood). $\endgroup$ Mar 26 '21 at 14:57
  • 1
    $\begingroup$ Does this answer your question? What is the role of model likelihood? $\endgroup$ Mar 26 '21 at 14:57
  • $\begingroup$ @AryaMcCarthy not really. I know the definition of the marginal likelihood and understand its purpose as normalization factor and for model comparison, but i still don't understand the intuitive interpretation of it. In the case that i have only one model what does it mean P(X) which does not depend on the model and on its weights? What does it mean "the probability of observing the data"? Does it represent the reality probability i.e. the perfect prediction? $\endgroup$
    – ofer-a
    Mar 26 '21 at 19:40
  • $\begingroup$ Ah. Well, the model evidence $P(X)$ does depend on the model—implicitly! We tend not to write it, but it's there. The other asker decided to be explicit about it: $P(X \mid M)$. It's how much the observed data agrees with this class of model—that is, your assumptions about the world. It's not a 'reality probability', because it's entirely dependent on the assumptions. $\endgroup$ Mar 26 '21 at 20:33
  • 1
    $\begingroup$ I must say i like your question. $\endgroup$
    – Good Luck
    Mar 26 '21 at 22:57
1
$\begingroup$

In a more general format, $P(X)=\int_{\theta\in\Theta}f(X|\theta)P(\theta)\mathrm{d}\theta$.

Observe that $\int_\theta{}P(\theta)=1,$ so it is a proper probability distribution. That does not have to be true for $f(X|\theta)$ over the set of $\theta\in\Theta$.

That is why most people ignore the denominator as being anything except an annoying scalar. It makes the product in the numerator sum to unity. That view is that it is just a utilitarian necessity.

However, it does have an intuitive meaning. It is the subjective expectation of the likelihood of the observed data. It is the sum or generalized sum of all possible explanations for the observed data under your subjective prior beliefs.

It doesn't have much implied value because of two rules in Bayesian thinking.

First, your prior has to be fixed. Two people with differing priors will get two different values, but you don't get to go experimenting with priors once your experiment has started. Your beliefs are your beliefs. A good prior reflects that. A bad prior does not.

Second, in Bayesian thinking, the sample space is ignored, with the exception of the observed sample itself. Other than for things like model checking to prevent things like having negative calories in a model or other such nonsense, the sample space gets short shrift in day-to-day practice.

Nonetheless, consider the following model with a binomial likelihood. The parameter space is discrete, $\theta\in\{2/5,3/5,4/5\}$ and a sample space of $k=0\dots{5}$.

If your prior mass function was $\Pr(\theta=2/5)=1/3$, $\Pr(\theta=3/5)=1/2$, and $\Pr(\theta=4/5)=1/6$ then your denominator would be in the first chart over each possible realization.

denominator

If your friend's prior mass function was $\Pr(\theta=2/5)=8/10$, $\Pr(\theta=3/5)=1/10$, and $\Pr(\theta=4/5)=1/10$ then your denominator would be in the second chart.

second denominator

Nonetheless, it sort of begs a "so what." With a fixed prior and a known observation, it is the weight of the evidence, but we don't get to live in parallel universes or with parallel bodies in one universe.

It is the expected value of the likelihood of the observed data under your subjective beliefs. It doesn't have any inferential value. It is a scalar.

It is important. It has intuitive meaning, but it doesn't have much value other than the utilitarian purpose to cause the posterior to sum to unity.

Its most practical value is when it doesn't work. When the integral diverges, as it easily can with improper priors, then you should either be honest about your priors or use a Frequentist method.

A Bayesian model minimizes the average loss created by being unlucky and getting a bad sample. However, the prior acts much like a context for solving a problem in. Without an understanding of the prior probability of getting the actual observation, how could you possibly minimize a loss?

Consider a person raised on a desert island by parents that didn't tell the child about horses, racing, money or gambling. The parents die and the child is rescued as an adult. They look for a job and see one for a bookie.

How can a person, using only data, minimize a loss without proper data on what a horse is, what a race is, or the value of money?

Frequentist methods, in most general cases, minimize the maximum amount of risk you would be exposed to. Without a context to solve a problem in, which is really what a prior distribution is, that is a wonderful alternative.

The denominator can tell you when it is time to stop using a Bayesian method.

$\endgroup$
3
  • $\begingroup$ Thanks. so can we say it is equivalent to the prior predictive distribution? $\endgroup$
    – ofer-a
    Mar 28 '21 at 13:19
  • $\begingroup$ No, it's a unitless scalar. The prior predictive distribution is a distribution in $\tilde{X}$. $\endgroup$ Mar 28 '21 at 23:01
  • $\begingroup$ The integrals just look visually similar. $X$ is data and is a non-random set of constants. $\tilde{X}$ is a random variable. $\endgroup$ Mar 28 '21 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.